Don't quote me on this... but I think you need to have permissions set properly in Android Wear. I am just running with the default permissions activated. At some point when I am more comfortable with what AW is doing, I may throttle it back down a bit.
Do you have any other bluetooth accessories / apps running at the same time? I have found Android Wear doesn't play nicely with the other apps I need to run my other smart watches. Because all these watch related apps have so many similar background running processes... they crash into each other for lack of a better term.
So I had to un-install those apps and I now am looking to activate those non-AW watches as stand-alone devices on SIM cards.