It happens if you long press the app to remove it and you don't actually bring the app up to the remove option. When the X sticks just long press the app again and let go, without removing the app and the X will disapear. This occurs on Nova Launcher for me.