Update: so yesterday I did my 4th factory reset, except after I did a normal reset I booted into recovery mode, cleared the cache, and did another wipe. Not sure if that's what did it, but it has improved greatly. Before Android System was always ahead of screen usage, by the end of the day screen was at 20-something percent and android system was around 30. Right now, even though android system is still second, it's more online with everything else and far below screen. It also really helped as last night I let my phone idle and battery only went down 1% after an hour, before it would go down 5-10%.
*knocks on wood*