Might be time to replace the battery... But before you do, make sure that the watch IS connected to the phone (when it's not, actively looking for a phone drains battery like crazy). Also uninstall/re-install Android Wear on your phone (sometimes updates to Android Wear make the watch go nuts; that solved my latest battery drain a few weeks ago).
If that doesn't help, then reset your watch and don't pair it to your phone. Just leave it out of the cradle. If in that 'blank' mode the battery doesn't last more than a few hours, then yeah, the battery needs replacement.