physical chemistry - Chemical bonding and dipole moment

We know that ammonia undergoes amine inversion. Why doesn't the dipole moment decrease in ammonia, since the direction of the dipole changes to the opposite direction every time there's inversion?

Shouldn't the effective dipole moment become zero?