Other than on women's jeans (they have tiny pockets), I haven't noticed anyone with issues carrying this in their front pockets. I have the Nexus 6, which is larger than the Note 4, and it easily fits in jeans, pants and jacket pockets.
I never kept a phone in my pocket opting instead for the defender cases because it is less likely to get damaged in my opinion. I have small hands and the phone is not a problem, you get used to it quickly.
Carry my Note 4 in my front pockets. I'm on the heavy side and thus am not a skinny jeans kind of guy so the larger size of the Note 4 isn't an issue for me (except when wearing board shorts with smaller pockets).
As for pocket carrying in general I've never had an issue carrying my phones in my front pockets, even without a case or screen protector. All it takes is a conscious effort to keep one pocket dedicated to the phone and nothing else to ensure the phone is never scratched.
I carry mine in my front jeans pocket no problem . I have a verus Thor case on too. I have medium hands and can use the phone easily one handed. The phone is perfect for me!
I keep my Note 4 in my front pocket all the time. Jeans, shorts, never have a problem. Sometimes it's hard to get out, because I'm thin and sometimes the pockets are tighter on the jeans. I have a casemate case on mine. I do find it hard for one handed use, but that's just because I never used any phone that way.
I carry mine in my front pocket. Dress slacks, no problem. Jeans, a bit tight with a seat belt and driving. Any drive 15 minutes and the phone goes in the console.
I carry my Note 4 (with the Samsung Protective Cover) all the time in front pockets and very carefully in my back pockets. The case is very study, looks good, and most importantly, keeps a slim profile.