The N4 seems to be more sensitive than most phones regarding positioning against the ear. I find that moving it even a few millimeters in any direction can have a huge impact on the perceived sound level.
As to the volume of the ringer, try different ringtones. If I recall correctly, the Pyxis ringtone is by far the loudest of the stock ringtones and will break through any other sound in the room. I typically use the Hangout ringtone however and find that it is not unusual for me to miss a call while watching TV or even if the phone is in my front pocket while I am in any environment with background noise - the grocery store, for example. But, the Pyxis ringtone will wake the dead.
Also, when the phone is sitting on my desk or another flat surface, I always place it screen down/speaker up. Of course, I use a TPU case (
Amazon.com: i-Blason Slim Fit S Line Flexible TPU Case for Google Nexus 4 Smart Phone E960 LG 8gb 16gb (Black): Cell Phones & Accessories ) which has a raised rim around the screen thus the screen never actually comes into contact with the desk/table/etc.