The Nexus 6 is still a decent phone, but it is getting a little long in the tooth. It is expected to be updated to the next version of Android (Nougat), but that will probably be it's last update.
You said you tried factory reset, did you trying using it before re-installing all your apps? I'm wondering if some app is slowing things down.
Whether or not you should upgrade to a new phone is kind of a personal decision. It depends on if you're happy with the way your current phone is running (it sounds like you're not), and on your budget, what you can afford, and what you're willing to spend. The last couple of generations of Samsung phones (Galaxy s6 and newer) have all been really great. The reviews of the OnePlus3 are really good. Honestly, all of the flagship phones are good; it just depends on what you want.
Keep in mind, if you own your Nexus 6 (and don't have to turn it in when you upgrade), you can sell it for almost $200 right now. That's about what they are going for on Swappa. That would help to offset some of your cost. That market price will probably fall a lot when the new Nexus phones come out in September/October. (it's always falling a little).
https://swappa.com/buy/nexus-6-att?ref=mobilenations&data1=UUacUvbUpU5252245