Overall, the 5X is a nice phone. It's very light which combined with the smaller screen make it easy to hold and pocket and operate with one hand. I started with Android in ~2009, this is the smoothest operating handset of the group (Nexus 6P is also a solid performer).
However, my 5X camera does struggle. HDR and panorama processing put a strain on the processor (battery consumption, heat, etc.).
After a few weeks I sometimes notice a bit of lag or stutter launching or switching apps. Now I restart the handset every few days and wipe the cache partition every two weeks or so. I've had little to no issues since.
+1 for rebooting and wiping the cache partition.
I've had Android phones since Froyo (OS 2.2)--you know, back in the day when one had to do a lot [manually] to keep your phone running smoothly because the OS was pretty much in its infancy. Over time I have acquired a host of habits like manually turning off my screen before putting the phone into my pocket, powering down at least once a day, plugging in (topping off) whenever possible, charging while I sleep, wiping my cache partition every 7-10 days or if I've uninstalled a bunch of apps recently, FDR-ing if I got a bad update (only had to do that once, phew!), and swipe away apps when done. I'm sure I have other OCD things that I do without even thinking about them.... Anyway, my phones run smoothly and reliably, so while I can't cite causality, there does seem to be a correlation between what I do and how well the phone runs.
Lots of people say "you shouldn't have to do [reboots, wiping cache, etc]" but for me, if I barely have to think about [reboots, wiping cache, etc] and it helps to keep my device continually productive, then I just shrug and continue doing what simply seems to
WORK.