I currently have both devices with me 100% of the time, and they are both active with AT&T SIM cards in them. I like them both, but my priorities may be different than yours. So here are my thoughts when comparing the two:
The Nexus 5 has a better display and camera than the X. This isn't even close, and there are quantifiable ways to prove the display is superior. It has better color accuracy, gets brighter, and is far more visible in daylight (which is extremely important to me). The X has a really good display for being AMOLED, but it's still AMOLED and has all the draw backs associated with that technology (over saturated colors, poor daylight visibility), but I haven't had it long enough to know if it suffers from the "black splotch syndrome" or will suffer burn-in (I suspect no burn-in since the display stays off far more often than any other phone I've used).
The camera on the X got much better with the updates, and so did the N5. The N5 camera is far more consistent for me, though, with focus being better, shutter speed faster, and low-light performance easily better. That being said, both camera are serviceable.
The software experience is going to be a preference. I love the voice actions on the X, and I actually use them in the car. The breathing for notifications is great. The N5 has GEL, but that's not a huge selling point in comparison to the X honestly. The X wins on software.
The X also wins in battery life. This isn't even close. I have franco kernel on the N5, with undervolting, and the X still stomps it in battery life. I can go a day and half without charging the X. The N5 will mostly make it through a day, but it also sits on my wireless charger while I'm at work. Which brings up something else, the N5 has wireless charging. For someone that sits at a desk this is a really nice feature. I can just pick it up and set it back down when I'm done using it. Honestly, because of this I don't worry about the battery life of the N5, but that's my situation. Someone else's may be different.
When it comes time to pick one up to use, I usually go for the N5. The two most important things for me are the display and camera, and I know the N5 won't disappoint in either of those when I go to use it. Daylight visibility alone (especially here in California) is almost enough by itself to choose the N5 for my particular use case.
All that being said, I wouldn't tell someone with an X that they should get the N5, but I would tell anyone they should give it a fair shot. If you've done that and still like the X better I can't fault your decision at all, especially if it does what you need it to do better than any other device available today.