Welcome to the forums. What phone do you have? With most phones you have to slide the call icon to the right or the left, where there should be a green icon. The red icon is to dismiss the call.
In addition to what Javier said, some newer versions of the Google phone app now have you swiping UP to answer calls, and down (though this is tricky) to decline.