I LOVE my Note 4 and it is better for me. BUT, you can't say there's "no way" the Nexus 6 is better, because for some it is. If they prefer smooth, vanilla Android and rapid updates and a 6" screen, then that alone will make the Nexus better. I personally prefer the Spen, better battery, better camera, and better screen with the beautiful chamfered metal edges of the Note 4 so I like it better for me.
I loved my Nexus 5 but I forgot about the simple things that made me love Samsung. Like picking up my device when I have a message or voice mail and it gives a subtle reminder vibration, also being able to access all volume sliders from any screen simply by hitting the volume key and touching the gear. Since I loved my Nexus 5 so much I just downloaded the Google Now launcher and I truly believe I have the best of both worlds having a Note running the GN Launcher.