I have the same issue with my note 4. I keep my headset in my right ear. If I put my phone in my back left pocket I lose music streaming all together and voice is patchy. If I move it to my right back pocket voice clears up 90% and music is just spotty. I've tested the signal by leaving the phone sitting on a ledge roughly shoulder height and walked 5 feet away. If I keep my right side, where my headset is, facing the phone it is ok. If I turn 180 degrees and the headset is not in direct line of sight I totally lose music streaming. Otherwise It's an awesome phone. I had this same problem with my old Metro PCS LG Optimus M (MS690) but with my most recent HTC evoDesign 4G I could go around corners, even to a different floor and remain connected.
(GN4BLK, Sprint, Plantronics M50, M55, and M155)
Also This is my second Note4 (screen cracked in the cold weather on the first one) same issue.