I have had this issue if one of two conditions occur:
1. I'm actively involved in the conversation using the desktop client (via gmail web interface). I have found that once I've responded to a conversation via the web client, my phone will stop notifying me of new message for a period time. Closing the web browser session *usually* fixes this, but sometimes I need to actually send a message from my phone to the conversation before it starts updating again.
2. If I'm signed into Google Hangouts on more than 1 mobile device. I used to use talk with both my Nexus tablet and S4 phone all time with no issues, but I have found that if I'm signed into Hangouts on both devices, often only 1 device will update with notifications (though sometimes they both will). Same as above, I found that I need to sign out of Hangouts on my tablet then send a message from Hangouts on my phone to get it to start working again.
It's an annoying bug, and hopefully one that gets fixed. For now, I make sure to leave Hangouts signed out on my Nexus tablet, which seems to make the phone client much more reliable (which is most important to me). Not sure if this is what happening to you, but figured I'd share my findings.