Anyone can "access" any address on the internet. But you can't do anything with the server you access unless you know the password. (If it's badly written, almost anyone can access it - if it's well-written, your ip address goes into a "block this address" list and your provider is notified.)
If you're worried about your phone, run a firewall on it.