Definitely wait until 5.1. Or 5.0.2 or whatever it is. I got an unbranded phone from Samsung (as replacement for my burnt out S4) and so never had any time with kitkat to make a true comparison.
If I compare my lollipop S5 with my kitkat S4, then the replacement is slow, laggy, terrible...