I have had the same problem with every auto-brightness capable phone I have ever owned. Have it with the Nexus 5, had it with the Nexus 4, had it with an HTC Evo 4G. Etc.
The reality, in my opinion, is that auto-brightness is a good idea which just doesn't work well in real life. I have found that the slightest change in ambient lighting can trigger a response to the point of major annoyance. For example, if the room lighting is behind me, simply raising the phone from a table top so that I can look at it (or, alternatively, leaning over to look at the display) is enough to throw a slight shadow across the phone's sensor and thus initiate a brightness change. Or, in a dimly lit room while watching TV, the variation in light output from bright and not-so-bright scenes will also trigger the sensor. Heck, when lit from the front or above, even the reflection off my forehead seems to be able to trigger the sensor.
To sum up, do what I do: turn auto-brightness off and be done with it.