This is also happening to me. When listening to music and I get a txt music drops out for a moment and comes back. Sometimes while listening to Pandora the program hangs and I have to close Pandora and re-start.
I don't think there's a way to stop it from happening. The audio codec is probably only capable of a single output, so when notifier sends a sound, it overrides the current audio to play the alert.