Well they are a legit comparison for me. I am looking at both. And as the OP said, price is a main concern for me. Samsung hasn't become as big as they are for no reason. The Nexus is still not a well recognized name and is little competition for the Note out in the General Public land. In here, sure it is a valid comparison.
I could make this a really long post by saying everything that is on my list for comparing both devices. But it really boils down to a couple things as they both have plus and minus. If I was going strictly on quality, it would Samsung. Give it 8 months and lets see how both devices are holding up. Samsungs will be cranking along two years from now and this chinese Nexus, I would bet they will start to have noticeable problems in 8 months and give it a year before many start seriously falling apart. But for me, I will be replacing it long before that and I accept it if I get a Nexus at a cheaper price. But for most this won't be acceptable and I agree.
For me, they are about the same size. Both have metal builds. Both have about the same screen. I would choose the Nexus. I prefer it unlocked and I prefer stock android. I could find a Note unlocked (ebay) and I could do a custom rom. I like the build quality and design better on the Note, I like the camera better on the Note. But I can't get Project Fi on the Note, which I really really want. Then when price comes into the picture, the Note just loses out in my situation. In short, I would absolutely love to have a Note 5 hands down, but it won't fit my situation.
But I am the 1% of the buyers that Google aimed at with the Nexus and they know it. The other 99% are going to buy a Note and they should. Hopefully Samsung joins HTC in releasing better unlocked devices near stock Android. The world is going in that direction and it is only a matter of time, a LONG time if carriers have their way.
So yeah, I am THAT guy. Project Fi, Unlocked, Near Stock Android, and Cheap "er". But I do recognize that the Samsung Note is a much better phone and if I could get it to fit my situation and my budget, I would do it.