I would doubt that you do any real damage. But having said that, why keep installing and removing the case that you don't like? I'd suggest that you keep it in the case while you look for something better for your needs. At least the phone is protected while you search. I've shopped several cases for my now week-old Edge, and all of them tend to fit differently than the cases (both light and heavy) for my other devices. It may have something to do with the shape of the corners and the case having to grab at least a piece of the curved edges to stay on the phone.
What case you you have? And how does it "annoy the crap out of" you?