I've inadvertently played handball with my Note 3 (caught the charging cable with my elbow) and just about half an hour ago knocked it face down from the desk to a hard plastic chair shield. It's never going to be outside the Otterbox Defender again, unless I need to change the battery, SD card or something like that. That case saved me 2 phone replacements.
The screen? I can notice just a slight drag once in a while with the S-Pen (but I'm still using the original tip - 17 months old - maybe I should put a new one in?) but only once in a while, like once a week. Not enough to bother me. With my finger, I don't notice any problem. The screen stays clean and I can pop the case off and use a cleaner on it (or even wash it if I had to) without getting anything wet near the phone). If I ever do buy a phone again (and it's going to have to be a lot more than the Note 4 to entice me), it's going to have a Defender case for it or I'm not buying it. (No, I'm not trying to sell Otterbox cases, but I'm VERY impressed by the pounding it can take and still protect the phone. It's the Timex of phone cases - takes a licking and still keeps ticking.) And with the 3 (I don't know about the 4), it still fits into my shirt pocket (just barely). But it usually sits in the holster (which is also practically indestructible).