This has all got me worried reading all this. Is it worth downloading lollipop then?
It depends. If you're happy with how your phone works now, there is no need to update. Despite the numbering change from 4.4.4 to 5.0.2, Lollipop seems more like an incremental update, not a major update.
Lollipop has new features, some good, some not so useful (I like being able to tell my phone to be quiet when I'm sleeping, but I miss being able to add new widgets to the lock screen). If you like trying out new stuff, give it a try.
Will I constantly get promoted to download lollipop if I keep ignoring the notification to download it?
I don't know.