How do I downgrade my g4 to 5.1

You'd have to root the phone and flash the 5.1 ROM. Check over at xda-developers.com if you can't find any help here.
 
I asked at a Verizon store if they could do it, and not surprising they said no. Im sure they could if they wanted to, but they are all about the latest updates.