It appears there's no way to remove Android Pay. However, I found a way to circumvent this issue and default to another payment method. This solved my issue
Go to 'Settings' in your Uber app. Select your 'Personal' profile.
If you don't see a 'Personal' profile (I didn't), just click to set...