Not sure about the "open app..." option. I'd try setting to "Still use Android Pay" and see if that works. That's what I did with the Note 5, and had no problems using Android Pay, although I don't use PayPal.
Some terminals may also have a slightly different location where the NFC reader lives, so you may have to experiment a bit with where you hover/place the Note 8 so it can be in the terminal' NFC reader "zone".
How to set up NFC-enabled payment feature | Samsung HK_EN has some pictures that show where the NFC zone lives on the Note 8. That may help with matching the Note 8's NFC zone with the terminal's reader.