Yeah, you need the app as the foreground app. There's a battery saving mode where if you have your phone upside down it blacks out the display and it'll vibrate and audio prompt you. But if you go back to your homescreen or another app, you are out of luck.
Which is good... this thing would DESTROY your phone's battery if it didn't hibernate itself when you dropped out of it.