I just left the Note 4 for the Nexus 6. I'll say that if you want more things readily setup for you and easy to use, the note 4 is a great phone. The stylus is great, which is something I'm missing, but knew I would. I'm picking up a regular stylus, which is nothing like the Note 4 stylus, just to write with on the Nexus 6. It's a pale comparison on that front.
Camera to me is okay on N6, definitely better on the Note 4. I have to try some camera apps to see if I need one or not. Honestly, I'm loving the bigger screen and the front facing speakers so much more than the rear speaker on the Note 4. I guess the reason I left the note 4 for me was, on my original, even with Noise Reduction turned off on phone calls and trying everything I read online about getting feedback when talking on the phone, still was an issue on my Warranty replacement for my original also. I would always hear my voice feedback when talking on certain calls, not all but a lot. To the point I couldn't have a conversation, because it was too distracting. I had the correct case for ATT, so that wasn't a problem.
I still have my Note 4 and will not sell yet, just because I can still use it and the stylus without calling service. I'm using project fi on my N6 and it's working well. If I ever decide to go back to my Note 4, it's an option. It's a great phone, hated to switch to something without the stylus. I love not being tied to the big 4 and waiting for their updates, plus the ability to root if I want to, is a big plus for the N6. So it's a tough call, but I'm happy so far with mine. No background feedback on the Nexus 6, so that problem was solved for me.