I had the same problem with my Nexus 4. For me this is what fixed it:
On your phone go to Settings>Storage>USB computer Connection (the USB Computer Connection option is available by clicking on the three dots on the top right corner of the screen (at least on my Nexus 4 with Android 4.4)
When...