In my recent experience, Black Note 4's have loose volume and power buttons, but a very firm home button. The white note 4's have firm volume and power buttons with a looser home button. This is purely my own experience. I've had four note 4's over the course of 2 months.
All in all, I'd say its a pretty normal thing. I wouldn't be concerned about it.