The Nexus 6 was a pretty nimble phone, but things have moved on. Is it faster than the 5X? Yep, a little bit... in most tasks the big Moto has more grunt and the 805 happened to be one of the best thermally performing SoCs ever made (when it does throttle, it only does so slightly and you can run it forever at a very high load).
But the Nexus 5X wasn't supposed to replace the 6 (nor is the OP asking about that)... that's what the 6P was geared towards. The 5X has a less powerful GPU, less RAM, the speakers aren't nearly as good as the 6, etc... it was lower spec'd out when compared to the 6 and those differences are noticeable (including the price when new). Now, the 5X does absolutely wipe the floor with the 6 when it comes to taking pictures, it's not even close. Well, not 100% true, in good lighting, the 6 takes better video due to it having OIS where the 5X has nothing (though you can add EIS with root access).
The OP isn't looking to hear which phone has the best performance. He wants to go with a smaller phone and the Pixel, even though it would be ideal choice, is too expensive. So that leaves the 5X if he wants to stick with Google's offerings. What would he miss? I would say the main thing would be that the Nexus 6 was a joy to watch stuff on... big screen, excellent stereo front speakers, etc. And if you hammer on the 5X, you'll most likely start to see that it is limited by having a little less under the hood. BUT.... the 5X is still an excellent phone in its own right now that LG seems to have got their stuff squared away (7.1.1 runs really well on it).
If you can live without the multimedia prowess of the 6, the 5X should do you well, OP.