How to stabilize a filter

Suppose we have an unstable all-pole filter with transfer function $H(z) = A(z)^{-1}$. What is the best way to design a stable filter with frequency response as close to that of $H$ as possible. (I don't really know enough dsp to better define ''as close as possible'', but I am thinking in $L^2$ sense)

Answer

The magnitude of the frequency response will remain unchanged if you reflect any poles outside the unit circle - these are the ones causing instability - back inside the circle. I.e., a pole $p$ (with $|p|>1$) is replaced by the new pole $\tilde{p}=1/p^*$, where $*$ denotes complex conjugation. This will not change the magnitude of the frequency response (up to a scaling factor), but it will change the phase response.

The correct scaling is obtained if each factor $(1-p_iz^{-1})$ with $|p_i|>1$ of the denominator polynomial $A(z)$ is replaced by $|p_i|(1-z^{-1}/p_i^*)$.