Update...
Took the phone to my nearest authorised Samsung repairer (45m drive away), who said they could only take it and look at it in a few days...
One of their techs had a brief look at it, confirmed that he couldn't get it to rotate, but said he didnt know why or a solution.
Went in to one of my carrier's storefronts, where again they looked at the phone and verified they couldn't get it to rotate.
The storefront refused to replace the phone, also wouldn't take the phone for it to be sent back to their online ordering headquarters, because I had ordered it online.
Spoke to the carrier's call centre. They are sending me a courier bag for it to be returned and evaluated.
They won't be sending me a new one until after it is evaluated.
Time frame is 3-5 business days for me to get the bag (!?) and then unspecified amount of time for it to be looked at.
So now I'm waiting what looks like 2-3 weeks to use the phone. I asked them multiple times just to replace it to no avail.
Back to the old S4 for now ...