electronic configuration - The energy gap between a pi-conjugated system with (2 bonding and 1 anti-bonding orbital) and (1 bonding and 2 anti-bonding) orbitals

I asked a question previously about "why" it is the case the expanding the size of pi-conjugated systems decreases the required energy to excite an electron from a HOMO to a LUMO band: Why does the energy gap for π - π* transitions shrink with the size of the pi-conjugated system?. The user "Philipp" (and others) very patiently and kindly explained the reason for this by pointing out the expansion in the number of possible bonding and anti-bonding configurations for a polymethine-like pi-conjugated system.

However, there's something I'm missing that has to do with what is probably my lack of knowledge about chemistry. If we look at this image: http://i.stack.imgur.com/Ti3cx.png (provided by "Philipp") why is that that we have an energy gap between $\Psi_2$ and $\Psi_3$ smaller than the energy gap for a simpler, smaller pi-conjugated system where the HOMO corresponds to a single pair of p-orbitals in a bonding conformation and the LUMO corresponds to the same single pair of p-orbitals in an anti-bonding conformation (i.e. this system: http://i.stack.imgur.com/9cscv.png)?

My guess that this would be because the bonding interactions that exist in the larger pi-conjugated system mitigate the penalty for having the one anti-bonding interaction?

Update --- Could we justify the above guess using something like an overlap integral (if we know the shape of the two bonded p-orbitals forming a pi-bond)? If so, this would be quite intuitive.

Answer

If you want to really understand why energy levels of conjugated $\pi$ systems are the way they are I suggest you take a good look at the Hückel theory as @ssavec suggested - this method is only an approximation but because of its simplicity it is really good for helping you to understand the underlying concepts. Nevertheless, doing as you suggested and looking at the overall overlap between the $\text{p}$ orbitals in a given molecular orbital does indeed seem to work although I don't have a strict justification for that.

From Hückel theory it is known that the coefficients of the $\text{p}$ orbitals in a given molecular orbital can be calculated as

\begin{equation} c_{j,r} = \sqrt{\frac{2}{N + 1}} \sin\Bigl(\frac{r j \pi}{N + 1}\Bigr) \end{equation}

giving the coefficient $c_{jr}$ for atom $j$ in molecular orbital $r$ of a conjugated system of $N$ atoms (so that $j$ and $r = 1, 2, 3, \dots , n$).

Let's apply this to butadiene.

The $\pi$ molecular orbitals of butadiene are $\Psi_{1}$, $\Psi_{2}$, $\Psi_{3}$, and $\Psi_{4}$. They are built from a linear combination of 4 atomic $\text{p}$ orbitals ($\phi_{1}$, $\phi_{2}$, $\phi_{3}$, and $\phi_{4}$), so

\begin{equation} \Psi_{r} = c_{1,r} \phi_{1} + c_{2,r} \phi_{2} + c_{3,r} \phi_{3} + c_{4,r} \phi_{4} \end{equation}

When you calculate the atomic orbital coefficients for each molecular orbital $r$ then you get to the following picture:

where the numbers above the different $\text{p}$ orbitals are the coefficients.

Doing the same for ethene you get the following:

Now, you need to compare the overall overlap in the HOMOs of ethene and butadiene. Since it is already assumed in the Hückel theory only the overlap between neighboring $\text{p}$ orbitals is considered. Furthermore, it is useful to set the overlap integral of two "normal" $\text{p}$ orbitals (both with coefficient $c = 1$) to 1 (its exact value doesn't matter because only the relative amounts of overlap in different MOs are important for this comparison), so that you can calculate the overall overlap in a molecular orbital simply by multiplying the coefficients of neighboring $\text{p}$ orbitals and summing over these products. So, for the overlap in the ethene and butadiene HOMOs you get:

\begin{align} S_{\text{HOMO, ethene}} &= 0.707 \cdot 0.707 \\ &= 0.500 \\ \\ S_{\text{HOMO, butadiene}} &= 0.600 \cdot 0.371 + 0.371 \cdot (-0.371) + (-0.371) \cdot (-0.600) \\ &= 0.307 \end{align}

There is less overlap and thus less bonding in the HOMO of butadiene than in the HOMO of ethene which would imply that the HOMO of butadiene is higher in energy than that of ethene.

Now, what about the LUMOs:

\begin{align} S_{\text{LUMO, ethene}} &= 0.707 \cdot (-0.707) \\ &= -0.500 \\ \\ S_{\text{LUMO, butadiene}} &= 0.600 \cdot (-0.371) + (-0.371) \cdot (-0.371) + (-0.371) \cdot 0.600 \\ &= -0.307 \end{align}

There is more overlap and thus more bonding (or better to say less antibonding) in the LUMO of butadiene than in the LUMO of ethene which would imply that the LUMO of butadiene is lower in energy than that of ethene.

So, the HOMO of butadiene is higher in energy than that of ethene and the LUMO of butadiene is lower in energy than that of ethene. Thus, the HOMO-LUMO gap in butadiene is smaller than that in ethene.