i had the exact same issue, wifi router is a tp-link.
first seconds everythign ok then deadly slow. i figured out the problem,in my case it is network encryption problem because when i disabled the wifi security everything went ok, btw you can still have it working with wpa usinf fixed ip instead dhcp, my procedure:
-access the router config page take note of the SSID, ad give your note4 mac address (fromt the dhcp table) a fixed ip and take note.
-turn off the router
-delete the network on the phone
-add manually the network by inserting the ssid, the wpa password then check advanced settings and insert the ip manually as well as the gateway (the router ip) and save
-turn on the router again
it worked for me nevertheless its a shame that ppl have these issues for what it costs.