You don't need a wallet case, you need self control. Put the phone where you keep it, and only "look at it" when you need to do something with it. Every time you reach for the phone, stop and ask yourself, "What am I going to do with it?" If it's "Just look at it", put that hand somewhere else. (I stopped smoking that way, and nicotine is harder to go cold turkey from than cocaine, so "breaking the cellphone habit" should be a breeze.) After a week of having to stop yourself, you'll find that you're not reaching for it.
(I've had cellphones longer than you have, probably - since the early 90s. At one time, repping 4 carriers and an MVNO, I had 5 flip phones on my belt. I've never considered a phone, or a watch to be "part of my body". (Of course, I grew up, most of my life, with "if you're in public and need to make a phone call, look for a phone booth or a Bell sign on a store [that said that they had a payphone inside for public use].)
My daily driver, whatever it's been for years now, is in an Otterbox Defender belt clip - and unless I need to do something with it, about the only time I look at it is when I'm here, online (sometimes there are questions about cellphones on another site I frequent too) to see how to answer someone's question. I might glance at my phone once or twice a week to see the time (I don't wear a watch, I have a phone with the time on it if I need to know the time).
A little self control is good for you. Exercising it once makes it easier to exercise it the next time you need it. The more you exercise it, just like a muscle, the stronger it gets. (Giving up smoking? I just decided to, handed my pack to my wife and said, "Here - I'm not going to smoke any more." After decades of a 4 pack a day habit. And a few years of deliberately exercising self control.)