What's the price difference between the two for you?
For me, even at the same price, I'd choose a Note 4. There are several features that make it worth the "older" build in my personal opinion.
1. IR Blaster - use it a lot.
2. Replaceable battery - I'm on my second one.
3. SD card slot - I keep one in there to make backups of data on local memory.
The Note 5 will get OS upgradesfor probably a year longer, of course, and has a few nice features like themes, note-taking with the screen off (available on the Note 4 with an aftermarket app), a better camera, and a few other niceties I'm likely forgetting.
Glass versus plastic/aluminum build is a matter of aesthetics. I actually like plastic. Not everyone does.