Your worries are not well founded. I can assure you it takes more than just having your number to get your location, unless you're concerned about law enforcement.
Please consider creating an account so we can communicate further and better assist you. As a guest you are not able to reply any threads.
I have provided the link below so that you can register.
Welcome to Android Central! They could only do that if they're in law enforcement (and have a legitimate reason to track your phone, like if you dialed 911), or if you're actively sharing your location with them.