Well, we can factor out resolution right off the bat since they share identical specs in that regard.
In terms of brightness, the Note 4 is noticeably brighter at peak brightness. The question there becomes how bright do you set it. I generally have my Nexus 6 set at 40%, so whether I could achieve the same brightness on the Note 4 at 30% or any other lower figure is basically a moot point. In direct sunlight, it is fine for me when set on full blast, though I wouldn't turn down the higher nits offered by the Note 4. If you expect to largely use it outside and similarly find yourself setting the display at peak brightness, you'll notice a difference. Black level is, of course, excellent on both (and on any AMOLED display).
Color accuracy also seemed to be very close on both. I might favor the Note 4 slightly, but the Nexus 6 is still excellent.
Basically, the overall screen difference is so slight that unless you're going to perpetually have a Note 4 sitting next to a Nexus 6, you won't notice any lacking on the Nexus 6. If for some reason you would, the difference isn't substantial, and then you can factor in whether the larger screen size creates enough of a better value proposition to balance things out or shift it one way or the other.
As to battery life, when you say the Note 4 "leaves a lot to be desired," do you mean it's not able to meet a day's needs? I'd desire a phone that would last me a month on a charge, but have absolutely no misgivings about it whatsoever so long as it will last a full day for me. If the Note 4 doesn't last a full day, and you don't have access to a means by which to recharge it during the day, the Nexus 6 (or any current phone, for that matter) may not be a sufficient improvement. If you can access a charger, either of these devices will take a boost from a Quick Charge compliant adapter extremely quickly; if not, the Note 4 does at least have the feature of a user-swappable battery - which the Nexus 6 does not.
The Nexus 6 doesn't have the bloat, but does burn more power driving the larger (5.96" vs 5.5") display. If your aim was for a large screen, the Nexus 6 is the largest 2k resolution out there right now.