The only times I've noticed that the Gnex is as smooth as the N4 was during simple tasks. When you start doing multiple things at once (example: Installing app updates while steaming music and browsing the web), you really see how well the Nexus 4 can handle it all with ease. The Gnex can get choked up easily.
The difference in screen quality is huge. The Galaxy Nexus (or maybe just AMOLED?) had
some issues with it's screen. Besides the poor calibration of the N4 screen, which you can easily fix yourself, the screen is gorgeous. If you haven't noticed the poor quality of the Gnex screen, please don't try to look for it. Once you see it once, you'll see it every time you look at your phone.
I agree that specs don't mean everything, but with Android, more is always better. I'm not saying that the Gnex is a bad phone, but it's getting old... The Nexus 4 is a great, cheap upgrade.