I wish I had your self-restraint! By that time, it's likely you'll see better incentive pricing on the new Pixels as they'll be halfway through their cycle, and facing newer competition from other OEMs, and Google will have had the time to resolve any software bugs present at launch. And more importantly, there will be a much better understanding of just how good and how reliable the new SOC is.
I'm not sure there's really been a flagship in a long time that's actually had something for which people had a "need" instead of a strong "want" but you're probably mostly safe there. 5G can be a pretty solid selling point depending on your carrier's local coverage (for me it actually checked that 'need' box not because the speeds are faster (which they certainly are) but rather because it has provided me much better in-building and rural coverage). Beyond that, you'll continue to enjoy most new features introduced in 'Pixel drops' that aren't Tensor-dependent - though I expect Google to invest heavily in leveraging value out of that chip to provide unique differentiating features to the Pixels which competitors won't easily, if at all, be able to match.
I had upgraded generally annually from the Nexus 4 all the way through the Pixel 3 XL, but with the ever-escalating pricing and not finding anything compelling enough about the 4 XL (don't get me wrong - I still wanted it, but the 3 XL was running like a champ and there weren't enough practical differences to justify my budgeting for it), I held off a year. When the 5 came out as a good, but decidedly upper-mid-range phone, and with Samsung's S20 FE being so broadly praised, I decided to give it a run. Bloat is a thing, but not as bad as I had expected, the UI wasn't as tough to get used to either. Updates, while not quick, have rolled in (I got August's update about 2 weeks ago and then September's last week). Insane battery life, gorgeous screen, plenty of storage (I bought a micro-SD card for it, but candidly have yet to have needed it), and a bit faster, but outside of that, I felt in most ways it was, despite having better parts nearly all the way around, at best a match and frequently lagging behind my then-two-year-old 3 XL. 95% of my photos are with the normal camera, and I have yet to capture a shot I think betters what the 3 XL would have done, and definitely miss the unflappable reliability with which Pixels take a great shot. Every time. Now that's my experience - I'm sure a lot of people have moved away and been delighted.
If you do decide to switch over to something different than a Pixel, I would encourage you consider either keeping the 4 XL as a 'backup' or selling it directly rather than trading it in, as with having done the latter with my 3 XL I found myself in the unenjoyable position that I couldn't simply return the Samsung and go back to where I was.