No, I am not using headphones. I am using the "earspeaker" (the normal phone-against-the-ear mode).
Any case, looks like this has already been reported by someone in the google issuetracker for the beta program (yes, I am using Android 8.1.0 beta). Here is the link to the issues in the issue...