A system overlay is an app that has the ability to write screen updates on top of other apps. Typical examples are screen dimmers (Lux, Twilight), Facebook Bubbles, menus or trays that slide in from the sides of the screen.
Because these kind of apps have the ability to write/hide/change everything you see on the screen, Android flags them all as a security problem. If you try to do something "important" with an overlay running, Android will prompt you to first turn off the overlay app. Unfortunately, Android will not tell you which app is acting as the overlay.