I tried 3 different headphones and all had the same issue. I then went forward with doing a factory reset and the click is gone now. Hopefully this helps if anyone else is having this issue.
EDIT: Found out it was Screen Notifications causing issue. No need for factory reset.