When I'm on a Hangouts video call with someone my Nexus 5 tends to get hot on the back of it. After the call is done the phone will cool off and go back to normal temperature.
Is this a normal behaviour?
Yes.... There's hot and there's HOT. When you're in a video call, you're using a lot of resources. Your phone's camera is recording, CPU is transcoding the data and compressing it in real time, and then Hangouts is transmitting all of that... While also receiving video in return.
As long as you don't get a thermal alarm and have the phone power off on you, you should be OK.