You using the Gmail app? Just go into settings, choose the account, then at "Inbox sound & vibrate" for the sound choose "Silent" and make sure the vibrate box is checked. If you've done all that and it still isn't working, I'm not sure what else could be the issue.