AVOlio wroteUpdate :
I tried downloading with VPN on, it didn't have a huge impact on download speed, maybe even it didn't have any impact.
But, I tired downloading on another device than my note 4, on a Huawei phone, the download speed was much much better, the lowest speed I saw was at 1MB/s and it reached until 2.9MB/s, whereas 1MB/s was the highest I was seeing on my note 4 and rarely.
It's really weird that on my device the download speed is slow and on another is acceptable.
Especially considering that my device is able to fully use my internet download speed everywhere else except Netflix.
Samsung has some great phones and their hardware is topnotch, however, their touchwiz (now experience) skin has created some problems which show up in random places. I guess you found on of them. Even stranger, is that you could probably find another Note 4 somewhere, clone yours and it may not have the same issue.
Go figure.
Glad you found an answer to the slower download speed though--probably not what you wanted to hear.