Pros of one are not necessarily cons of the other.
S4 pros: Larger screen, higher resolution, removable battery, removable storage, s-features (voice, etc), high build quality
X pros: Faster and smoother experience, closer to stock look/feel of software, better battery life (per charge, but not removable) more compact device (reviewed as feeling great), active notifications, touchless controls, customization options, high build quality, assembled in the US
Generally this list comes off to me as being mostly superficial versatility versus optimization. Many of the pros and cons are aesthetic and totally subjective, but the most obvious variances are the size and whether you want removable anything. If you want a bigger phone, the S4 is obvious. If you require removable battery or storage for your usage, then the Moto X will not work out well.
If you want the latest greatest of Android, in terms of optimization and design philosophy it's probably the Moto X. Specs-wise the G2 is probably the best thing coming out very near term... at least until a couple weeks when the Note 3 shed's it's curtain and a few weeks after that the optimization and design philosophy both get shaken up when we're talking Nexus reboot.
Of the S4 and Moto X I would probably prefer the Moto X because I appreciate directionally where the efficiency and optimization are going. They're using what many are mislabeling as lesser specs to achieve better performance and that's great for this OS. That being said, S4 owners are saying that their devices are running faster and smoother than ever with a recent update and it's probably one of the top 5 devices released so far this year.
For your case, being that you're already on one of the best, I'd say if it's going to cost your money it's not worth it. You're not going to see a drastically obvious performance improvement. The only way I would say go for it is if you're A: Unhappy with your current device and B: Really in love with a feature or aspect of the sought after device. Top 5 to top 5 device, even if it's 5th to 1st, smacks of side-grade generally.