Normally you don't have to take the SD card out when you're performing a Factory Reset, but since you're on such a roll
it's probably safer to do so.
You may also want to make a backup of the contents of the SD card to your computer or the cloud - weird behavior involving the SD card is often the first sign that the card is going to take a dump.
Regarding apps - unless you're concerned with keeping your Candy Crush Saga game progress (and you're going to use an app like Helium to backup your app data) I wouldn't bother making a backup. I prefer the fresh start experience. YMMV.
Sent from my rooted, debloated, deodexed Sinclair ZX-80 running CM -0.001 using Tapatalk