It's a bug in the newer Samsung phones. The notification system turns down the volume for all apps not just the notification noises, presumably to protect your hearing.
So the Note4, S5, all suffer from this bug.
There are loads of misleading bits of information out there like "reboot your phone" but all this are incorrect ways to get to the real issue: The error occurs when notifications happen, so you have to turn off all the things that would produce notifications. These are: WiFI, Bluetooth, GPS and possibly mobile data. Even then you can have notifications such as calendar reminders.
So what to do? First, complain to Samsung so that they fix this problem because it's annoying many people and chances are the bug still exists in the newest phones.
However, the best advice that I can give is what I do: If I am listening on a bluetooth device, then I kill everything other than bluetooth as less frequent drop-out is a lot less irritating than the "every few seconds" drop-out that can occur.
However, as I use my phone as podcast streamer in-car with sat nav as well as other items running, I have a plan B which is a hair tie combined with a wooden block that I strap across the phone such that the volume button is permanently depressed for "increase volume". The notifications come in, the volume drops then it is immediately restored to full volume again.
Oh, and don't fall for the "volume locking" apps... none of them work that I have tried and pretty much tried them all.