So, turns out it works (at least when on airplane mode, haven't tested it when there's just no signal) if I connect it to my secondary WiFi network. Same router. Still a bit of a mystery, but I guess I'll just leave my phone on that one.
More info: It's a Samsung Galaxy s10e. Verizon network but the phone was purchased as an unlocked refurbished from another seller. Restarting the router does not help.