It's unfortunate, but it must be either a) your Flip unit, or b) your SD card.
I say this because the first Flip I got from Amazon wouldn't even let the SD card be inserted fully; opposite of your problem, I guess. It was so intensely springy that it would literally SHOOT out from the slot if I breathed the wrong way. I had to push on it about 10 or 20 times before it would finally accept it inside without spitting it out, but then eventually spit it out after about a minute.
I returned the Flip for another from Amazon, got it 2nd-day, and the same SD card slides right on in and stays in just fine.
Now, the trackpad on this unit is super clicky compared to my first one...guess Asus has some quality control issues on these things?
If you can, I would try to swap it with another. If you can't, at least you can still get it out if you push on it but use something to push the card down a little while releasing, right?