"Worth it" is a personal decision that depends on what you value in a phone (and what you can afford). I use my phone every single day, so for me, I think it is worth every penny of the $80 to have the nicer device. That said, the Moto G is a great deal and the two phones share similar performance specs. Some of the big differences will be 32GB vs 8GB of storage, a better camera on the Moto X, and niceties like the Active Display and Touchless Control on the Moto X. A slightly larger screen in similar form-factor is also a plus in my book for the Moto X. Plus the ability to customize your own phone (colors, wood, etc) is pretty compelling for a lot of people.
So, worth an extra $80? Depends. For me I would say yes. But for someone who just wants the most phone for the dollar and doesn't care about the differences above, then the Moto G might be your pick...