If you hit Applications to go into the app drawer, then you can long-press on the dock icons to drag them off the dock back up into applications; and then you can long-press on the new application you want and drag into the dock.
Also, you can hit Applications > Menu > Edit and it's more clear that you can move dock items around.
Incidentally, TouchWiz seems not to allow you to remove the Applications/Home dock item, but it will let you remove or replace the others.