I feel you. Coming from a 2013 MOTO X, note 4 is huge! I dropped it at least 3 times in my garage when working on the car. No case on it back then, thankfully, the screen is fine just some scuffs on the edges.
Then I decided to buy a case just in case. Got some metal bumper but they intervene with the signal and dropped 2 bars; Then I got Spigen Neo hybrid it is a great case but rather big and heavy to my taste. Finally I am now settled with Cruzerlite TPU case. Does not add any bulk and provides more than enough protection. Maybe after a few months I will go naked again tho, note 4 is too sexy to be put a case on~