In the course of handling any phone, you're liable to touch the camera lens at some point, especially when taking your phone out of your pocket. Just a simple wipe of the lens and you're good to go, so I don't think this will be an issue.
I don't even clean the lens on the S3 and 4S that I use. It's always covered in smudges, but hasn't really affected photos or anything in real world use.
I have a Note 2 and I get my lens dirty all the time and there's even no buttons around it.. It'll depend on how you used to hold your phone. Buttons won't matter much.