The card becoming more prone to data corruption is the biggest problem. SD cards are suited for write once/read many uses, such as storing media and other independent files. They can't stand up to the kind of repetitive write cycles that Android performs for system and app uses.
It also encrypts and ties the card to that specific phone. If you remove it, the card would need to be reformatted to use in anything else, even another phone of the same make and model. That also means if the card corrupts like mentioned above, the data on it is gone. You should be keeping backups anyway, but it's especially important in this case.
Google also no longer officially supports this function, and Motorola has been known to not implement it well. If you do decide to revert the card to external storage, make sure to backup any data you want to keep to an external source before you do so. It'll reformat the card in the process, and no telling what is saved on it that you'll lose.