This issue started very recently for me and upgrading to 5.1.1 unfortunately didn't fix it. For most of the year or so I've owned my Verizon Galaxy Note 4, it's been free of skipping and stuttering until about the last month or so. I've tried all the solutions posited, I've tried uninstalling or disabling various apps, I've tried booting into Safe Mode (it still stutters in Safe Mode), I've tried keeping the screen on, messing with the developer options, everything and anything I could think of and none of it really works in a way that tests reliably over repeated trials. This leads me to believe that it's a hardware issue because my phone didn't ship this way, it became this way recently and no amount of tweaking, disabling, cache wiping, rebooting, or safe mode-ing has caused it to go away. The one thing that does strangely seem to help is when I use my fitness band (a Mi Band), my AptX headphones stop skipping. I have no idea why. My non-AptX devices, however, continue stuttering as normal (like my waterproof speaker which I use in the shower) at a rate of roughly once every 2 - 10 seconds (or not at all for brief periods), depending on its mood. Now part of the NFC capability of this phone is built into the battery, and I happen to have 2 (the original and new one I just got recently), so in the hopes that it was the battery, I tried swapping to the old one, but no dice.
I have no reasonable explanation as to why my fitness band stops the stuttering on my AptX bluetooth headphones (I have 3 different ones), but if I leave the fitness band behind or uninstall the Mi Fit app, they will stutter just like my waterproof speaker does. I had a Fitbit Zip for the longest time before the Mi Band also (but I no longer have it), but there was no stuttering present at that time. So my conclusion is hardware failure of the bluetooth radio in the phone. One of the other earlier posts stated they tested an external bluetooth radio plugged into the audio jack and that it was sending audio fine, without any skips, hesitation, or stuttering, so this might be a theory with some merit. Honestly after all my frustrations with Samsung in general and this phone specifically, I feel like this will be my last Samsung phone.