I didn't go from the X but from the D4.
The only minus I could see was the hardware isn't as durable as the Moto phones. So I bought a case. I only used fancy cases on my D4 because they were fancy. And not even all the time. The screen is Gorilla Glass so I haven't put on my screen protector, yet. I bought one, but haven't used it yet.
I switched because my D4 was getting old.
I also wanted to keep unlimited data (at least) and since Verizon wasn't offering it, there were few options. The best and that saved me money was going to t-mobile and get their unlimited (everything--except hotspot, that's only 5gb).
One plus is my screen is better in color then my D4. I knew it would be as my dad has an S3 and that screen looked better.
Also a removable battery (does the X have one?).
Lighter as my D4 had a physical keyboard.
I hope this helps a little.
There's a lot of negativity on here, because most just post problems, and that doesn't help in deciding, so check out this thread:
http://forums.androidcentral.com/samsung-galaxy-s5/381967-official-i-love-my-galaxy-s5-thread-2.html#post3599561
Good luck in your decision.