It really depends on what improvements come with the hardware. Having an LTE N7'13, my desire for latest and greatest AOSP is taken care of. I've really enjoyed going back to my old '05 MOTO RAZR for emergency voice calls, so I don't really have a need for an N5(4?). However, the one thing that could change my mind and make me pull the trigger is an exceptional camera experience and/or the ability to purchase unlocked and use on Verizon LTE (possible if they ever integrate Voice into Hangouts). I realize this is highly unlikely, so I imagine I'll just be congratulating the folks who the N5 works out for.
Now, show me a Motorola Nexus, and EVERYTHING changes....