I messed around with today on my Nexus 4 (GSM obviously) and found the solution.
1. If you are on a call and another comes in and you swap to it, then hang up with the second caller and can't get audio back to the initial call, that means the second caller hung up before you can switch. All you have to do is get rid of the blue line underneath the two vertical lines which is the universal symbol for PAUSE. Once you tap the pause icon, which is located at the bottom of call screen, directly next to the right of the mute icon, the horizontal blue line will go away and you can then hear your original caller.
2. If the second caller has not hung up yet, then all you have to do is tap the circular icon that has 2 arrows and that will swap back to the initial call.
3. Now, as far as the merge calls problem when you have two calls going at the same time and you want to swap to just one call, tap the MANAGE CONFERENCE icon which is right above the end calls icon, and that will the bring up another screen asking which call you want to swap to.
I hope this helps. I'm not sure about CDMA phones and the layout on your screens. Also, this is the Nexus 4 I'm describing. So, let me know what's up with this description please. Thanks.