You can't turn off just that sound without being rooted afaik. You can however lower the volume slider for System Sounds down to zero, that does it but then you won't get system sounds. Never needed them myself on my S3 since notification are a separate slider and I have all the touch sounds turned off anyway.
*EDIT - just tested on my S5. I have the System Sounds setting down to zero too and when I reboot I get a fraction of the AT&T sound before the volume slider loads and kills it it seems. So not a 100% fix on the S5 but it was on my S3. Just seems to be the order in which the OS components load or something.
Are you talking about the startup sound? If the phone is on mute or vibrate, the startup sound does not play. I'm on Verizon if that makes a difference.