It made a major difference. Probably 30 seconds max boot time. Why would it boot so slow with the SD inserted?
Dude! There's your problem right there.... it sounds like it's trying to reboot through the Autonooter every time... all you really need to do is reformat that card and slap it back in there, should fix it.