I've been struggling with this for some time now because sometimes it works flawlessly and sometimes not at all, even after I pick up the phone again within minutes. This is what I found.
Smart unlock only works when pressing the virtual start button. That's the one at the bottom, in the screen, which requires a hard press. All other means of waking up the phone will require the selected unlock actions.
I suspect there's a relation between the Google Maps home location and the location you entered for safe unlocking. This has come to mind when hard press didn't unlock while at home. Since the unlock address states it's the "Google Home address", I checked the address where Google Maps Timeline thinks I am. That was unknown and had to be updated. After I entered I am at my home location, unlocking worked fine!
I hope you can find your way using this information.
Kind regards,
Gerard