The only differences of which I am aware besides physical size are the screen resolution, battery size, and charger (that being a recent revelation).
Screen resolution difference isn't something you'll likely notice unless you plan on using Daydream VR. 1440P will have less of a 'window screen' effect than 1080P.
Battery size is smaller, but as it's driving a smaller, lower resolution display, should be similar. Actual difference may depend on your personal usage profile, but it won't be a dramatic amount.
Charger is slightly lower output, but should charge the device in similar time to what the XL manages.
They come with the same exact 18W capable charger and 6ft USB 2.0 Type C cable (60W capable so it can even be used to charge most MBP, need a 100W capable cable to use Apple's fastest charger but I digress). AFAIK the difference isn't the charger, which is a very good one (and very much worth $35 with that quality cord if anyone needs a spare, not gonna do much better/cheaper going aftermarket)... The smaller Pixel itself is limited to 15W charging, internally, presumably over thermal concerns.
The larger Pixel XL can decide between charging at 3A - 5V (15W) or 2A - 9V (18W) when the battery is really low, whereas the smaller Pixel will only select the former. It's all still part of the USB PD spec, and once past 50% charge they both slow way down anyway. In the end it doesn't make much of a difference, but it's worth being accurate about what's really going on. At best it makes it slightly easier to find an optimal car charger for the smaller phone (there's a few cheap but well vetted ones that do 15W max, Nekteck's particularly).
Either way both phones still charge way faster than non-QC phones (which I guess are rare at this point), those would top out at 7.5-10W, and they both support USB PD fully and come with spec compliant chargers (unlike the Nexus 5X and 6P which featured some oddities and could be finicky about charger, I wouldn't use theirs with a Pixel btw).