I've also found adaptive brightness wonky in most of my devices over the years. I only use it because I drive with my phone mounted a lot and like not having to fiddle with brightness mid-drive to adjust for varying sun/cloud/night situations outside. However I did notice that with it turned off the screen is much brighter on a lower slider setting so there's probably even more battery savings to be had that way.
My biggest issue with adaptive brightness on Android is that the variances are too subtle. What I mean by that is if even if I go from a dark room to sunny daylight the increase in brightness isn't large enough for my liking and I end up having to pull down the quick menu and raise it anyway, defeating the purpose of 'adaptive'. If there's was also a scaler option for how adaptive it behaves besides just the regular brightness slider that would be a great idea. For example there could be a submenu under adaptive brightness settings asking you if you would like 10% increments or 20% increments. For simplicity's sake I believe they intend that to be handled automatically, but in my experience it doesn't do that well enough.