fft - removal of blackman harris window results in signal higher than expected

We are capturing data using a PX14400 adc. The samples are 16bit unsigned integer, 200M samples/sec, I data only. 100MHz bandwith. We have a cw signal at 51MHz, -11.4dBm. The -11.4dBm is measured by a power meter so that includes all losses right before being digitized.

Question 1: when I apply a blackman harris 4 term window, the fft data calculates to -14dBm which sounds reasonable since bh loss is about 3dB, however when I dont apply the window the fft data calculates to -5.3dBm. I would have expected a 3dB jump, not almost 9. Why the large jump?

Question 2: Does anything jump out in the code below that might be off in my calculations. For brevity not all the code is shown (threads, board init, etc).

// Initialzied elsewhere in the class
const int fftSize = 8192;         // size of the fft
complex array[fftSize];   // array of complex (r+i)...used post fft
double dData[fftSize];            // array of doubles, used pre and post fft

fftPlan = fftw_plan_dft_r2c_1d(fftSize, dData, reinterpret_cast(array), FFTW_ESTIMATE);

void processNewBuff(uint16_t *rawdata) {

        // convert the samples to V.  This formula is provided by the vendor
        // dData is an array containing the converted to V data (double)
        // rawdata is an array containing the initial adc counts
        // since we only fft the first 8192 values, we effectively throw out the rest of the data
        // we skip the rest of the 10ms sample (200MS/s * .01 = 2MB sample, 8192 is processed the rest is discarded
        double dInputRange = .220;

        for (int x=0; x            dData[x] = (-dInputRange)/2 + ((rawdata[x]/65532) * dInputRange);
        }

        // apply blackmanHarris
//        double bhtmp;
//        for (int x=0; x//            bhtmp = (double)(2 * 3.1415 * x / fftSize);
//            dData[x] *=(double)(0.35875 - (0.48829*cos(bhtmp)) + (0.14128 * cos(2*bhtmp)) - (0.01168*cos(3*bhtmp)));
//        }


        // do the fft.  fftPlan is a "fftw_plan_dft_r2c_1d", discreet fourier transform, real 2 complex
        // see: http://www.fftw.org/doc/Real_002ddata-DFTs.html
        // after the fft, we throw away the upper half of the fft since we only started with 'I' data
        fftw_execute(fftPlan);

        // normalize the fft for power
        for (unsigned int x = 0; x <(fftSize/2); x++)
            array[x] = array[x] / complex(fftSize, 0);


        // convert the fft to power spectrum
        // I have this separated into 3 distinct lines for readability, final code will be all in a single line
        for (unsigned int x = 0; x <(fftSize/2); x++) {
                dData[x] = sqrt(real( array[x] * conj(array[x]) ));
                dData[x] = ((dData[x]) / 50) * 1000;     // convert to W, then mW
                dData[x] = (20 * log10(dData[x]));
        }
}

Answer

The reason for that is that you don't normalize the DFT samples properly. Dividing by number of samples in time domain is valid only for Rectangular Window. For simple case of DFT, you should divide your amplitude by sum of window samples:

$$S=\sum_{i=0}^{N-1}w_i $$

This for Rectangular Window will be obviously equal to $N$ ($8192$ in your case), whereas for Blackman-Harris Window it's about $2939$, therefore the Coherent Gain is about $0.36$ instead of $1$.

For reading more about this interesting topic I recommend you these papers (especially the first one):

G. Heinzel et al - Spectrum and spectral density estimation by the DFT, including a comprehensive list of window functions and some new flat-top windows - especially chapter 9

F. J. Harris - On the Use of Windows for Harmonic Analysis with the Discrete Fourier Transform