After spending a few days dealing with this problem, I have a pretty solid theory, as well as a solution.
My theory is that everyone has this problem has a nice, tight case, like an otterbox. This is causing SOMETHING that is creating this problem. My guess is either that dirt is getting stuck...