laplace transform - How can a system be unstable if $L(jomega)$ is never exactly $-1$?

Say we have a linear system with unity feedback, with loop transfer function $L(j\omega)$. The closed-loop transfer function from reference to output is $T(j\omega) = \frac{Y(j\omega)}{R(\omega)}=\frac{L(j\omega)}{1+L(j\omega)}$.

At frequencies for which $L(j\omega)$ approaches $-1$, clearly $|T(j\omega)| \rightarrow \infty$, so the system is unstable - if excited at this frequency, the output is unbounded.

But systems can be unstable even if $L(j\omega)$ never equals $-1$. From the Nyquist plot, we can show that $T(j\omega)$ can have poles in the right half plane if the contour encircles $-1$, even if $L(j\omega)$ never exactly reaches it. (However, I have very little intuition for why this is true - I just see it as a theorem from complex analysis that happens to be useful here).

Alternatively, from the Bode plot, we say a system is unstable if there are any frequencies for which $|L(j\omega)|$ > 1 and $\angle L(j\omega)< -\pi$ (i.e. phase margin is negative, or gain margin is less than unity). However, I'm not sure why these two conditions result in instability, since these don't result in $|T(j\omega)|$ going to $\infty$.

Two questions:

(1) If $|T(j\omega)|$ never goes to infinity (which is the case when $L(j\omega)$ is never exactly $-1$), how can a system possibly be unstable? Is $|T(j\omega)| \rightarrow \infty$ not the right criterion for deciding whether a system is unstable?

(2) Intuitively, why is a system unstable if there are any frequencies for which $|L(j\omega)|$ > 1 and $\angle L(j\omega)< -\pi$? I understand that you can see it from the Nyquist plot because these two conditions tend to result in encirclements of $-1$ in the $L(s)$ plane, but I'm looking for the intuition.

Answer

You cannot make conclusions about the stability of a system by only considering its transfer function evaluated on the imaginary axis $s=j\omega$. Replacing $s$ by $j\omega$ in the transfer function only makes sense for a stable system, otherwise you get a function of $\omega$ that does not describe the system, but another (stable) system.

Let me explain this by an example. Assume a causal system with a transfer function

$$H(s)=\frac{1}{s+a}\tag{1}$$

If $a>0$, the system is stable and by substituting $s=j\omega$, we get its frequency response

$$H(j\omega)=\frac{1}{j\omega+a}\tag{2}$$

The corresponding impulse response is

$$h(t)=e^{-at}u(t)\tag{3}$$

where $u(t)$ is the unit step response. We have

$$H(s)=\int_{-\infty}^{\infty}h(t)e^{-st}dt,\qquad \text{Re}\{s\}>-a\tag{4}$$

and

$$H(j\omega)=\int_{-\infty}^{\infty}h(t)e^{-j\omega t}dt\tag{5}$$

Now imagine that we decrease the value of $a$. The system remains stable as long as $a>0$, but as soon as $a$ becomes $0$, $H(j\omega)\rightarrow\infty$ for $\omega=0$, and the system becomes unstable. If we further decrease $a$, the system remains unstable, but the expression $(2)$ does not go to infinity anywhere. However, the point is that the expression $(2)$ is not the frequency response of the system anymore if $a<0$. The reason is that the integral $(5)$ does not exist anymore. So it is pointless to try to deduce stability or instability from the frequency response of a system, because an unstable system has no frequency response.

What you get if you replace $s$ by $j\omega$ in the transfer function of an unstable system is the frequency response of another system with the same expression for the transfer function, but with a different region of convergence (ROC). The ROC is the region in the complex $s$-plane for which the integral $(4)$ converges. For the causal system in our example, the ROC was given by $\text{Re}\{s\}>-a$ (the ROC is always a right half-plane for causal systems). For an unstable system with $a<0$, the ROC does not include the $j\omega$-axis, and, consequently, $H(j\omega)$ does not exist. In that case, the expression $(2)$ is the frequency response of a different system with transfer function $H(s)$ given by $(1)$ but with a different ROC given by $\text{Re}\{s\}<-a$. That system is stable but anti-causal, and its impulse response is given by

$$h(t)=-e^{-at}u(-t)\tag{6}$$

which is clearly different from $(3)$.

Concerning your second question, note that $e^{\pm j\pi}=-1$, so a phase shift of $\pi$ for certain frequencies implies a sign change, which turns the negative feedback into a positive feedback. This can be compared to the positive feedback in an audio system where a microphone picks up the loudspeaker signal, which is fed back into the amplifier, sent to the loudspeaker, etc.