When I first received my N5 (AT&T) it had some lag on wakening - for all modes.
After I turned off the AT&T stuff (pretty much all of it), disabled the TW items I didn't want, and then disabled ANT, the lag pretty much disappeared.
It seems to be triggered by functions in the background that awaken when you wake up your device. I suspect it is a specific one but did not spend any time trying to identify it.
I'm all good now.