filters - Convert a FIR to an equivalent IIR

Is there a way to convert a FIR to an IIR filter with the most similar behavior?

Answer

I would say that the answer to your question - if taken literally - is 'no', there is no general way to simply convert an FIR filter to an IIR filter.

I agree with RBJ that one way to approach the problem is to look at the FIR filter's impulse response and use a time domain method (such as Prony's method) to approximate that impulse response by an IIR filter.

If you start from the frequency response then you have lots of methods for designing IIR filters. Even though it was published about 25 years ago, I believe that the method by Chen and Parks is still one of the better ways to approach the design problem. Another very simple method for the frequency domain design of IIR filters is the equation error method, which is described in the book Digital Filter Design by Parks and Burrus. I've explained it in this answer.

If the phase response is of importance to you, then one problem you will be facing when designing IIR filters in the frequency domain is the exact choice of the desired phase response. If the overall shape of the desired phase is given you still have one degree of freedom, which is the delay. E.g., if the desired phase is $\phi_D(\omega)$, and the desired magnitude is $M_D(\omega)$ then your desired frequency response can be chosen as

$$H_D(\omega)=M_D(\omega)e^{j(\phi(\omega)-\omega\tau)}\tag{1}$$

where $\tau$ is an unknown delay parameter. Of course you can say that if $\phi_D(\omega)$ is given then you don't want to modify it with an additional (positive or negative) delay. But it turns out that in practice the average delay is not always important, and - more importantly - for certain values of $\tau$ your approximation will be much better for a given filter order than for others. So the delay $\tau$ can become an additional design parameter and should be chosen optimally or at least reasonably.

I've written a thesis on the design of digital filters with prescribed magnitude and phase responses. One chapter deals with the frequency domain design of IIR filters. That method can be used to design IIR filters with approximately linear phase in the pass-bands, or to approximate any other desired phase (and magnitude) response. The filters are not only guaranteed to be stable, but you can also prescribe a maximum pole radius, i.e., you can define a certain stability margin. You can also find this method in a paper published in the IEEE Transactions on Signal Processing.