Re: Debating
You should just try it to see how you like stock Android, it will be a different experience. Just use it and make up your mind before the last day you have to return it. Coming from the Note series, stock Android may feel lacking to you since it's cleaner and lack some features, such as multiwindows (which Google is supposed to be working on). The Samsung's status bar quick toggles are much more useful than stock Android quick toggles as well because it have may more things you can toggle, however you can use third party software on the Nexus to compensate this. Also if the S-Pen is something vital that you use everyday, I recommend sticking with the Note 4.
One thing you will notice with stock Android though is it's going to be a lot smoother, even despite how powerful the Note 4 is.