I have moved from the S4 to the Nexus 5 recently and I think it was a great decision. As mentioned above, the S4 will give you (cons) bloatware, delayed updates, questionable extra features (I turned them all off anyway) and (pro's) removable storage slot, removable battery, better camera.
I bought a 32G N5 and its more than enough for me, the battery has not been an issue (between 8 hours (heavy use) and 16 hours (light use)) but I ensure i don't have battery draining apps (Facebook seemed to be a culprit). For me it is a delight to only have apps installed that I need/want. The default Google apps for me are more pleasing than the dated look of the Samsung ones (such as Keep v S Note, Email, SMS and dialer apps etc)