Do I have to copy the contents of my SD Card to my computer by unmounting the SD Card and removing it from my phone and then copying it to my PC? And vise versa when I copy the contents back on the new SD Card? Can I leave my phone on and copy the new contents via the phone USB?
You can, but, eventually you will have to unmount to do the switch.
Usually though, turning the phone off, then removing the sd, swapping sd's after copying is really the most sensible. Phones can get a little wonky if they're currently running an app from the sd, and then you unmount it.
I'll assume you don't have an sd reader on your pc?