Just got my Note 4 last Friday and when I launch the camera it won't focus right away even when selecting an object for it to focus on. It takes about a good 10 seconds for it to finally start focusing and is then fine. I just uninstalled Instagram and launched the camera again and it focused right away, will have to keep an eye on it. I upgraded from an S5 as I wanted the larger screen and the focus on that phone was immediate and very responsive. I'm on AT&T.
This seems to be a common problem. I had to have my Note 4 exchanged.