Update :
I tried downloading with VPN on, it didn't have a huge impact on download speed, maybe even it didn't have any impact.

But, I tired downloading on another device than my note 4, on a Huawei phone, the download speed was much much better, the lowest speed I saw was at 1MB/s and it reached until 2.9MB/s, whereas 1MB/s was the highest I was seeing on my note 4 and rarely.


It's really weird that on my device the download speed is slow and on another is acceptable.
Especially considering that my device is able to fully use my internet download speed everywhere else except Netflix.
AVOlio wroteUpdate :
I tried downloading with VPN on, it didn't have a huge impact on download speed, maybe even it didn't have any impact.

But, I tired downloading on another device than my note 4, on a Huawei phone, the download speed was much much better, the lowest speed I saw was at 1MB/s and it reached until 2.9MB/s, whereas 1MB/s was the highest I was seeing on my note 4 and rarely.


It's really weird that on my device the download speed is slow and on another is acceptable.
Especially considering that my device is able to fully use my internet download speed everywhere else except Netflix.
Samsung has some great phones and their hardware is topnotch, however, their touchwiz (now experience) skin has created some problems which show up in random places. I guess you found on of them. Even stranger, is that you could probably find another Note 4 somewhere, clone yours and it may not have the same issue.

Go figure.

Glad you found an answer to the slower download speed though--probably not what you wanted to hear.
25 days later
Update #2
I just discovered that when starting the downloads on Netflix and the pressing back repeatedly to go out of the application, somehow kills the app' permission to use full speed of your wifi.
Another proof of this, when after closing the Netflix app by tapping back to exit, and after a very long time of slow download speed, when a download finishes, I always see a notification of "a wifi connection is required to download xxxx", xxxx being the next episode in the download list.

On the other hand, if you start the download and stay in the application (specifically the downloads tab) the download speed used is significantly higher, as well as pressing directly the home button to go to the home screen (this will take you to the home screen without closing the Netflix task I presume).
7 days later
Are you limiting background permissions by any chance?
I don't think so,
I can only restrict data usage in background for mobile data only in the application settings, it says wifi will be used if available.