Yeah, the way Android firmware updates work is by replacing the system files, so the "factory data reset" is kind of a misnomer: you'll erase your user data, but the OS version will never go backwards. The only way to downgrade is to unlock the device bootloader and manually install an old system image.