Older versions of android have fewer security patches.
Of course that matters, but how much that matters is heavily dependent on the bad security habits of the human user.
That said, I would upgrade my phone for the better hardware and more features, assuming my "old" phone was not making me happy. However, keep in mind that as time goes by fewer older versions of android will be supported by new apps or app updates.