Unfortunately there's currently no way to change the emoticons in the gTalk app.
It's really quite surprising how little love the gTalk app receives from Google. IMHO, if Google gave even a little effort into improving the app, google talk could/would become the predominate method of text communication and alleviate the plethora of new chat tools (whatsapp, kik, pingchat etc..) that seem to be trying to reinvent the wheel.
However; There are a lot of other 3rd party apps for gtalk, yahoo, AIM etc.. that have many more features. Some free, some paid. I use Trillian myself.