In the settings it says it only works when the screen is off...
The only time I've left my pen out was at home... was doing something on the phone in the living room, then a call came in, so I just set the pen down and took the call.
Walked around a bit during the call, throughout the house, outside, back inside, finishing up in the kitchen. I ended the call and turned off the screen. Took a few steps back toward the living room and got the notification.
So I assumed it was a combination of:
1. Screen is off.
2. Pen is out.
3. Phone detects a certain number of "steps" and issues an alert.
So at least in my case, it did not alert until the screen was off (completely off, not just off due to the proximity sensor of the phone to my ear during a call) and I'd taken a few steps.
ETA: just tested this here at work.
1. Removed pen and set it on my mousepad.
2. Leaving the screen on, I left my desk and walked to the men's restroom, which was 125 steps away.
3. (Details omitted...)
4. I turned the screen off and began walking back to my desk.
5. After approximately 10 steps, I got the notification.