I've had the Pixel 5 for two days now, after having the Pixel 4Xl for a year and I'm very happy with the switch.
The size is the best difference. I don't have small hands, nor very big hands, I wear medium to large gloves, and the Pixel 5 is so much easier to use one handed and just fits in pockets nicer. You don't lose much actual screen real estate since you mostly just lose bezels - I read a lot of stuff on my phone and haven't missed the Pixel 4XL at all. But yeah, just walking my dog or around the city and checking my phone, so much better now, plus checking my grocery list while shopping as well.
FPS is real nice when having a mask on, you just gotta rebuild that muscle memory that has your finger on the sensor as it comes out of your pocket, it's already coming back for me.
Haven't noticed any difference in speed when it comes to what I do with the phone, which is everything but gaming. I did take a few test photos and it does take that little bit longer to process the photo, which I don't like.
The hole punch camera is a bit distracting but I suspect it will turn into background noise as I use the phone more.
No issue with the front speaker sound while on phone calls or navigating or watching videos (don't watch many videos and they are not music videos)