It shouldn't slow down the phone if everything is working correctly. I have a note 4 with a 128GB microSDXC Sandisk 48MB/s card and no issues.If it just happened it might be indexing, I would let it run for a day. If it still happens after a day, One way to test would be to put a blank file labeled ".Nomedia" in the root directory of the sd card, and see if you still have problems. If that fixes your problem then you have an indexing problem potentially with a corrupt file somewhere. You can remove the nomedia file from the root directory and put the nomedia file in each directory individually one at a time to test to see which directory has the corrupt file.
You would also see indexing service use a lot of battery resources and force wake lock.
If you put the nomedia file in the root directory and still have problems then probably a bad sd card
It is also possible that you have a bad sd card, so you can try to exchange it first, if you don't want to troubleshoot.
Sent from my Verizon Samsung Galaxy Note 4