Probably. The sensor is typically at the top, near the earpiece. Your body (hand, head, arm, shoulder) might be interfering. None of us sit perfectly still, especially reading on a smartphone.
The only way i can think of to truly test it is in a room with no light, leavig the screen timer on for a few minutes, and waking as far away as possible. If it continues to self-adjust, it may be time to start trying cache clearing/hard reset to see if it's hardware or software.
It could also be reflection, sunlight variation, etc. The best thing to do since it's bothering you is to just turn it off. If you're leaving your screen on for extended periods of time, the auto-adjustment features won't help battery life much.