I am having a problem on my Nexus 4 that affects almost every app I run on it.
The screen will just suddenly pause or refresh itself.

Here is how it acts in different apps:
Chrome scrolls back to the top of whatever page I'm currently reading and resets to the default zoom level.
The browser in RedditIsFun does the same thing.
Gmail just flashes white for a moment.
Youtube flashes black for a split second and pauses the video.
Netflix exits the video and goes back to the video information.
Mxplayer flashes black for a split second and then continues playing a video.

I was originally on JellyBean. I tried upgrading to kitkat and that did nothing.
Someone suggested I do a factory reset, so I tried that. This fixed the problem for about a week.
Then I noticed the problem start again. I rebooted the phone, and then the problem stopped again for a couple of days.
Now the problem wont go away. It does this every few minutes and can be incredibly annoying if I am trying to watch a movie or read a long webpage.
What could be wrong?