fft - Remove a known wav file from recorded file

I looked at many similar questions but still cannot find an answer to my problem.

I'm playing an audio file N, while recording a man talking (b). The problem is that the microphone gets the played sound N together with the man talking, b. so actually I have y=N*+b in my recording, where N* is the played sound with a short time delay and a bit noise (due to the recording and the phone speaker).

My goal is to totally remove the N* from the recorded file. It seems like an easy problem cause I have the original played file N, but I find it more complicated.

What I tried:

1. finding the delay between the N and y, invert N and add to y. Seems like it's not working and the result is even worse probably because N* has different amplitudes and a phase shift.



2. Tried the same thing but subtracting in the frequency domain between y and N. Still no good.

Any suggestions?