I had the opportunity to try a Nexus 6 preloaded with Lollipop 5.0.2 and it was very nice - quick and responsive. I think the issue here is that the OTA 'upgrade' of Lollipop (5, 5.0.1, 5.0.2) is nowhere near as compatible to earlier Nexus devices as it is supposed to be. Upgrading to Lollipop seems to be almost universally disappointing on 'older' devices. My N7 works with 5.0.2, but its hardly the slick product it once was, 'even' after a Factory Reset - which no user should need to do to make a OS work.
IIABDFI (If It Ain't Broke...) seems to apply here. Stick with what you've got if it serves you well, I'd say.