Well, I can say that if you go to an iPhone forum, you are likely to see people singing the praises of iOS. And if you ask if they think you should switch, they will say yes. If you stay on an Android forum, people will sing the praises of Android and tell you to stay.
I am trying to be as unbiased as possible, though it is difficult as I am a huge Android nut. In regards to the GPU on the 4S, that is a legitimate point. But the question needs to be, when the Nexus comes out and you get time to play with it, what is it that it cannot do that the 4S can? And on the contrary, what can't the 4S do that the Nexus can? It is true; the iPhone works, and works well. And that is one of the biggest things that ICS is set out to fix; making the OS more accessible and having it "just work". I am confident the Nexus will deliver that experience, and I think you will be satisfied with it since you are already familiar with Android. In terms of GPU comparison, again, it comes down to what one device can do for you that another can't.
People talk about specs all day and night, comparing chips and GPU's and things of the sort. But if somebody gives you a device and doesn't tell you anything about its specs, and it does everything you would ever want it to, would you not like it anymore if you were later told it was lacking a certain spec that another phone has?
You sound like you have had some bad experiences with Android, and that is truly unfortunate. Like I mentioned, I had an Incredible, and I loved it. Your device issues have sounded mostly hardware related, though I do admit, I am not surprised to hear that you found the X or the Inc to be laggy, especially compared to the 4S. Do recall though that they are both on older operating systems and old hardware. I had a Bionic for a few weeks, and though overall I wasn't satisfied enough to keep it, I realized it was butter smooth, something I miss coming back to my Thunderbolt. Dual core does make a difference, even though 2.3 isn't optimized for it. I can only imagine that having the OS optimized for a dual core, and having a dual core phone, will give way to a totally smooth experience, unlike anything you have come across with Android thus far.
So here is my totally (attempted) unbiased recommendation. Play with your wife's 4S a little more. Tool around with the operating system, maybe even see if she will let you take it for a day to see how it functions under what will be your normal usage conditions. If it does everything you need to do, great! Make a pros and cons list of what your experience with the device was like that day. Keep it noted as a solid alternative. But as an Android fan, I say before you adopt the iPhone, just wait a few weeks (if you can) for the new wave of devices. Current leaks suggest that the RAZR, Nexus, and Rezound could all come out on the same day, or in close proximity. Seeing as how you like the fluidity of iOS and are shaking your fist at the experiences you have had with HTC and Motorola phones, I say look at the Nexus first. Play with it in the store, and, if it is a valid possibility, purchase it. Then, use it as you normally will be for a few days. Make a pros and cons list of how it functioned for you over the course of a few days, and compare it to what you noted about the 4S. If you find that one was obviously better than the other, then the choice should be easy! If they are very close, take your wife's 4S again, play with them side by side.
If you end up deciding the 4S is a better experience and better suits your needs, great! That is the most important thing; getting the device best suited to you.
If you decide you like the Nexus, even better! Keep on droiding!
Good luck, I know it isn't an easy choice to make. Hope you find something that you are satisfied with!