It is the Location Service with is slowing down wifi in some cases, you can try to disable it in the settings or set it to "GPS only" or something
The original G+ thread is this: productforums_google_com/forum/#!topic/nexus/kdix_8YPyq8 (replace the underscores before the first "/" with ".")...