Size.
I have a Nexus 5.
When the Nexus 6 first came out, I got it and used it for about 2 weeks. The size drove me crazy, too big. I could not one-hand it. Returned it to Sprint and went right back to my Nexus 5.
I'd still use my nexus 5, however I've already replaced the screen on it 3 times, and the motherboard once. Now the digitizer is starting to go and it probably can use a fresh battery, I don't want to keep dumping money into it anymore.
I will Very Much MISS not being able to use my Qi chargers. I have two in my house, one in my car (which was awesome for the N5 because of the magnet), and one in the office. I may still go back to the N5 yet if not having wireless charging gets on my nerves. Depends greatly on the N5x battery life