I was also thinking along these lines, or perhaps a tablet (even if used to meet budget) if you have access to private Wi-Fi. If this is primarily to pay bills, the only other option I could think of would be to stick with some of the traditional ways of paying by phone or mail. What you are asking for is convenience, and unfortunately convenience comes with added costs by default.
Though this also comes with some other complexities. Do you already have online accounts to make those bill payments if you had a device? If not, you'll have to set those up, which means keeping track of login information. That in itself can become daunting without a password manager. You'll also start to become at greater risk at things like identity theft and scammers. Maybe you have already thought about these things, but I don't know. What I do know is the scams are always being changed and improved on, making them harder to catch until it's too late. They also target the elderly when they can. This isn't meant to scare you, but just the reality of the situation that you need to know if you didn't already.