Welp, restarted my phone and the fix broke, had to delete and re-add. I've tested virtually everything, It's seemingly random when it wants to work.
Google, you can do so much, but you can't get a simple location service to function. :I
— Only1canoli
Worked for me, seems to suggest something about the accuracy of the location tracking.
I'm on a Moto X 2nd. Time to see how long this fix lasts.
Update: I just removed my neighbors, left with just "Home" from Google Maps, and it still works. Weird.