what's the worst thing that can happen if you flash the wrong recovery with fastboot ?

You simply won't have a working recovery right ?

Assuming that you can get back to the bootloader, just wasted time. But if the recovery you flash goes "blue sky", it can wipe out the bootloader partition, and Android, and you have a paperweight.