Dan Moren has a great write up on why slowness is killing the Apple Watch over on sixcolors.com and I couldn't agree with it more.
This part sums up my experience and frustration completely:
The problem with the Apple Watch is that we’re being asked to strap something to our wrist—to attach it to our very body—without it delivering on the corresponding promise that it will be much faster to use than our phones. The stale data and the lack of speed means that either you have to stare at your Watch for several seconds and hope the data updates; or tap on the complication to load the Watch app, which as we all know takes a good long while as well; or simply give up and pull out your phone.
It’s not just that the Apple Watch is slow; it’s that it’s slow while promising to be faster. That the Watch is a remarkable piece of engineering I won’t dispute, and as a fashion statement, well, it’s as personal as fashion always is. But as a consumer electronics product, it doesn’t really deliver on its primary reason for existing.