I am able to swipe the notification when I do NOT unmount the USB drive before removing it.
If you unmount it then you're stuck with the persistent notification. There is no need to unmount.. just like in windows, unless there is some data transfer going on you won't have any ill effects.
Peace!