I just got my Nexus 5 yesterday and I was having the same problem. However, if I touch the phone icon and drag right, it works. It is just particular about "where" you swipe when to answer the call. Not a big problem for me because the phone speed and graphics are amazing so far. :)