Might be a dumb question, but did you verify that the card has been added to the phone?
And yes, as noted above, Google Wallet is problematic on the N4 for some reason. Some have no problems, others can't even get cards activated in the app. You may just have to put it aside until the Android team figures out what's going on.