Sorry for the simple-sounding question, but are you sure you're swiping up completely? Sometimes, if you don't swipe up enough, it isn't enough to register.
Wiping the cache is a very routine process that doesn't affect user data, so I wouldn't be worried about it. But if you can tolerate the problem, then I guess it's not a big deal.