1. The speed of the internet that you're accessing.
2. The strength of the mobile data or wi-fi signal.
3. Sometimes the connection speed of the router. If you're using a router that uses an older protocol (like wireless B or G), then even if the ISP is giving you 100 Mbps download speeds, you won't be able to take advantage of it, because you'll be limited by the maximum router connection speed (11 Mbps for B, 54 Mbps for G).
4. Sometimes the DNS server. The Ultimate Guide to Changing Your DNS Server