I know with larger cameras and lenses like on DSLR's, you can damage the lens and sensor components without ever snapping a single photo. All of the glass elements in those lenses are designed to focus light on to one point, and just like using a magnifying glass to burn ants, things heat up. This blog from a rental company shows just how bad it can get, after people ignored the warnings of not shooting the recent solar eclipse without special filters.
https://www.lensrentals.com/blog/2017/09/rental-camera-gear-destroyed-by-the-solar-eclipse-of-2017/
I tried to look up some info about smartphones, but I couldn't find anything that wasn't also related to the eclipse. So here's my best guess...
I would think that pointing your phone at the sun will carry some risk of damage. If it's just a quick panning motion, that should be fine. The longer you point at the sun, the more heat can be generated. It's like how you can safely pass your hand through a candle flame without feeling it, but it'll burn you if you leave it.
On the flip side, very little light is passing through the lens relative to a DSLR lens simply due to its small size. They are also lower quality optics. Going back to the magnifying glass example, it would be like comparing a fine crafted high end laboratory magnifying glass to a toy one. Both magnify the subject, but the toy will not be good enough to focus a point of light. Any heat that is generated in the smartphone lens would likely take a lot longer to build up. Depending on the time of day, location relative to the sun, and ambient temps, this may even be a non-issue.
I still wouldn't suggest pointing at the sun, though. As is demonstrated in the article I linked above, heat damage can happen even without taking the shot. On top of that, the sensor is exposed at all times on a smartphone. The only difference between taking a photo and just carrying it around is whether the sensor is powered on or not. Even with the phone powered off, the risk is still present if you aren't careful.