I was having a similar problem with my note 4. The rear camera would take 5 to 6 seconds to focus. The image would appear blurry but if I waved the camera about a bit, eventually it would focus on my subject. Upon closer inspection I discovered that when I launched the camera app it was focusing at it's shortest depth, so things very close to the lens would be sharp. I could see the focus icon trying to find the subject, but it wouldn't focus at a distance until 5 or 6 seconds had passed. Once it found an object at depth, it would continue to focus fine with no problems, until I restarted the camera app and the whole thing started over again. Interestingly, this problem was across all apps that use the camera.
I was able to resolve the issue by resetting all of my camera settings. Someone in this thread mentioned the camera focus mode being the problem. I actually couldn't find that setting, but resetting them all seemed to resolve the issue. Now, the camera will focus on an object at any depth as soon as I launch the app.