When it can no longer run the current apps. A phone designed to run Cupcake can't run much these days. I expect my Note 3 to last at least another 3 or 4 years. (I still keep my V551 running, though. If I need a phone, but there's a chance it might get broken or stolen, that's the one I take. Replacing it from Craig's List for $20 is a ripoff. Replacing a Note 3 isn't cheap.)
Why would I change, and to what? An Ara phone, if I wanted to change and had the money to buy one with decent specs. Then upgrading becomes a lot cheaper. Otherwise, I'm happy with what I have for the foreseeable future (and I have great "future vision").