Welcome to Android Central! What "My Activity" are you referring to? Do you mean you're seeing the Phone app appear in your Google Activity? Keep in mind that many apps will ping your Google account on a regular basis to refresh data, even if you're not actively using the app.