LebGeeks

RandomMemory

Member

openmptcprouter setup and ping overhead

For openmptcprouter which region is the best to rent a vps in and which provider gives the lowest ping, if someone has set this up already what overhead did you end up with when it comes to increased ping and did you face any other issues, does it work well with separate providers?

Last edited by RandomMemory (November 27 2020)

samer

Admin

Re: openmptcprouter setup and ping overhead

Regarding VPS providers, your best bet is to run some latency tests yourself to figure out which region provides the lowest latency. I would imagine that European servers will yield better results than US or Asia.
Try this out: https://cloudpingtest.com/

nefe_lpmk

Member

Re: openmptcprouter setup and ping overhead

Usually Datacenters located in Western Europe have the best delay to Lebanon. Since we are international-fiber connected to Marseille.
You should expect a delay of around 50-80ms from a good local ISP to WE. That's the best you are going to get.

Beta0

Member

Re: openmptcprouter setup and ping overhead

My quick and dirty review:

Ping during torrent, the two spikes in the graph are only 20ms with 182 connections, TCP low latency mode is checked, MPTCP over VPN is mandatory for Ogero (filtered). Terranet is the master MPTCP interface as there is no VPN being used to enable MPTCP, but it works vice versa as well, just slower to connect cold start.
Terranet WDSL + Ogero ADSL. BLEST algorithm is selected for heterogenous links, the default is better for 2 lossy links but it degrades speed while the latency remains the same. Ogero is not lossy in packets, it just keeps disconnecting 60 times per day, while Terranet around 3 times a month.

Terranet MPTCP eligibility tracebox example:

I've went through 8 providers in the past 2 years, so far only OVH and Vultr have high uptime longer than 10 months, e.g: outage is 15 mins every 10 months. They did remind me 1 week before it happened.
I've yet to see any provider other than Vultr offering high frequency (3.8ghz) 6 dollar VPS without shared utilization. It offers the best interface as well.
Vultr high frequency (6$) vs OVH ($6):

I'm getting 7 ms lower with Vultr HF Paris compared to OVH.

One culprit with OMR is the status page and applied changes delays, you should wait around 1 minute before the real status / change appears or happens due to the 10+ scripts running in the background, this can be really confusing for starters . Another less concerning culprit is the author's childish elitism attitude towards people running or using Windows on github issues/IRC, as someone who worked on the Glorytun part it wasn't a great experience contributing to the project, including OpenWRT community.
This was running fine over HyperV in my homelab, though I prefer a dedicated device for months long uptimes, Raspberry Pi 3 is more than enough for <500mbit of total aggregated link speed. My Ryzen 4500u laptop could do 4.2 gigabit with the VPS part in local VM.
Plugging 4g modem, Android, iPhone in tethering to the RPi automatically adds the connection and aggregates without interruption or any packet loss, I often use that to boost internet speed for the entire house with no interruption, from 15mbit (Terra+Ogero) to 50mbit with Alfa unlimited, very useful while uploading very large files without pausing/restarting. Was also useful during 1080p wide angle TrueConf conference call, not a single frame dropped or stuttered and the image improved immediately without restarting the call. This also allows iOS devices to transition between mobile data and home network during video/voice calls without any signs of switching, as Apple is an early adopter of MPTCP, facetime and even apple.com website has MPTCP enabled. iOS switches to redundancy mode first, to aggregation, then disables mobile data once it verifies WiFi is good, tested and working perfectly with OMR, Google Duo implementation only works with ProjectFi, I hope they open it up. Alfa and Touch do not filter MPTCP thankfully. MPTCP enabled websites somehow behave the same as HTTP3 (udp) even if they are still running HTTP1.1, latest Linux kernels 5.4+ have it enabled by default. You may also notice that single stream downloads are boosted due to having subflows (starts at 4), so even using OMR for a single connection "as a VPS" improves internet connection speeds for high latency sources, e.g a Californian website with >10 sources loading, that include ads, fonts, bootstrap and other non static frameworks etc will be improved as HTTP1.1 in browsers is single threaded. Some software downloaders such as Epic, Android studio are single threaded. Secure unique links are usually single, encountered that with Uni's website, researchgate and paid download sites as well. This is where multi thread download (accelerators) don't work such as IDM, but MPTCP does. Keep in mind that, OMR does not use MPTCP to do aggregation only, it also exposes/enables MPTCP for clients with the VPS's public IP. iOS so far utilizes that, technically bypassing OMR aggregation proxy and VPN for Apple servers.

Netflix bans datacenter IPs, not just VPNs, you can enable bypass to each interface based on DPI protocol filtering:

You can also use this for load balancing. About load balancing, if the VPS is down, OMR will switch to load balancing mode, it even has a mode that fallbacks to the lowest latency link, or the faster one as a choice. (dynamic - balancing). OMR has forked and improved VNstat interface for LuCI compared to default OpenWRT, it saves the data as well.

Edit1: Another note, do not enable SQM for buffer bloat, it is not needed, and you will get lower speeds with wireless providers if they are behind ordered buffer queue such as Terranet's. BBR is enabled by default, since both the VPS and OMR are using BBR, it works effectively removing buffer bloat from the Shadowsocks TCP connection (this is the pipe to mptcp for non mptcp connections, another VPN runs over this proxy/pipe for UDP, ICMP and everything else, MPTCP is bypassed), assuming that the VPS to the outside world doesnt have bufferbloat of coarse. Usually TCP congestions algos are useless if the other server does not use the same algo, and even if does that is TCP only, and mostly useful for uploads.

For a beginner you should be up and running in less than 30 minutes, no need to tune anything, this detailed post may make this sound too complicated ;)

I may shoot a video physically ripping one of the ISP's modem cable while showing CSGO's jitter net graph if it sparks anyone's interest. 0% dropped frames in Twitch (OBS) as well.

About locations: Paris is defiantly the way to go, I'm getting only 5-10ms extra to European and American servers compared to barebone (interleaving disabled) Ogero. On a sidenote, Terranet WDSL has lower pings compared to Ogero. Pinging to 1.1.1.1 or 8.8.8.8 does not go through Europe on Ogero/Wireless, that's why you get 40-55ms, it goes through IXP from a near datacenter, Cloudflare is near Terranet's building, near Berytech, they use microwave. I recommend to always ping to a server outside Lebanon when doing comparisions, that's why the ping graph in the top post is 65ms, pinging to Cloudflare Paris is around 62ms, Lebanon 48ms, OMR or any VPN from Paris will only affect latency to Lebanese servers, which are almost non existent, you can still bypass. (e.g Lebanese csgo server)

Edit2: To save the trouble for others, Speedify no longer aggregates packets, except iPad with multiple USB ethernet, and it does it at userspace level with bad latency and 100% cpu usage. It's a fancy load balancer with a single IP endpoint (the vpn part) to avoid session dropouts, there is still a 5 second packet loss each time a connection is down, udp based services will drop, only downloads and tcp services such as ssh will remain but with a 5 second of inactivity/zero speed. They use nDPI to detect video calls and certain games and do FIFO packet duplication from each interface to the VPN, known as redundant mode, also used for "Streaming mode" not channel bonding. Their IP addresses are on the watch list of many captcha/verification security services, expect tons of suspicious activity captcha from a simple google search, and their nDPI does way too many false positives on torrent detection, switching you to a very congested Amsterdam server, dropping calls easily. Only the team edition with private server implements packet aggregation, and only on Linux/iPad, costs more than $200 a month with evaluation discount, very PITA to setup as a router on Debian, heavily relies on outdated Network Manager, requires Linux knowledge. Peplink services are optimized for TDD links such as LTE, else you have to manually tune it if one of the non-TDD ISP's speed go down, affecting latency, ~200ms on load, pricey as well, suitable for non-satellite media coverage since their streaming service does dynamic artificial latency 500-1500ms for time skew buffer, same way Youtube "Ultra low latency" work.

Edit3: I recommend using CUBIC congestion control only when using BLEST, the default BBR is too aggressive with MPTCP and may cause large buffer bloat, MP-BBR is coming soon.

Last edited by Beta0 (January 12 2021)

Beta0

Member

Re: openmptcprouter setup and ping overhead

Regarding VPS providers, your best bet is to run some latency tests yourself to figure out which region provides the lowest latency. I would imagine that European servers will yield better results than US or Asia.
Try this out: https://cloudpingtest.com/

Great site! Unfortunately Vultr is scoring consistently 30ms higher than my Vultr server in Paris, seems to be using a large HTTP ping, not ICMP, or my browser/pc is busy.
Most providers offer looking glass service open, they have test servers and IPs that you can ping to as well.

Last edited by Beta0 (November 29 2020)

RandomMemory

Member

Re: openmptcprouter setup and ping overhead

First of all thanks a lot for the detailed explanation, I was trying with one connection today since my terranet installation was pushed back to this week so doing preparations I guess. I noticed that after enabling the OpenMPTCPRouter server my speed dropped in half. I am using an dell optilex (With an intel pentium) for this since I was using the same machine for pfsense load balancing. I noticed that the fans started kicking in really loud after this option got enabled and I tried doing a couple of speed tests. If I disable the server and connect to a normal vpn I get a solid 12-14mbps. After enabling the server The speed dips to around 5mbps-6mbps. The server is rented from OVH and it's located in france. Do I need a stronger machine for this or should I try some other configurations.

Last edited by RandomMemory (November 30 2020)

Beta0

Member

Re: openmptcprouter setup and ping overhead

First of all thanks a lot for the detailed explanation, I was trying with one connection today since my terranet installation was pushed back to this week so doing preparations I guess. I noticed that after enabling the OpenMPTCPRouter server my speed dropped in half. I am using an dell optilex (With an intel pentium) for this since I was using the same machine for pfsense load balancing. I noticed that the fans started kicking in really loud after this option got enabled and I tried doing a couple of speed tests. If I disable the server and connect to a normal vpn I get a solid 12-14mbps. After enabling the server The speed dips to around 5mbps-6mbps. The server is rented from OVH and it's located in france. Do I need a stronger machine for this or should I try some other configurations.

No problem, that's odd, how old is the Pentium? you can try running OMR in a VM on a Linux desktop on that machine, openwrt kernel is incomplete for very old systems in order to reduce size. Is this with MPTCP over VPN enabled?

Last edited by Beta0 (November 30 2020)

RandomMemory

Member

Re: openmptcprouter setup and ping overhead

Intel(R) Pentium(R) 4 HT CPU 3.00GHz
2 CPUs: 1 package(s) x 2 hardware threads
AES-NI CPU Crypto: No

MPTCP over VPN is enabled

Last edited by RandomMemory (November 30 2020)

Beta0

Member

Re: openmptcprouter setup and ping overhead

Intel(R) Pentium(R) 4 HT CPU 3.00GHz
2 CPUs: 1 package(s) x 2 hardware threads
AES-NI CPU Crypto: No

MPTCP over VPN is enabled

Ah that's too old, OMR auto selects encryption based on what VPS supports, OVH and Vultr KVMs do have AES enabled, it assumes that the client (your desktop) is always faster than the VPS and it probably is doing AES. In the wizard page, what is selected as encryption under advanced settings? I guess chacha20 may be a bit faster with Pentium 4, not to mention it favors 4 cores. My orange pi zero ($12) had better performance than the first gen atom based celeron (2c) from 2012, though I do not recommend any Rockchip SoCs as they're unstable with the latest kernels.

Edit1: Even with a recent Celeron quadcore setup (no HT), with peg causing 10-20 ms extra latency on full load with tether:

Last edited by Beta0 (December 11 2020)

RandomMemory

Member

Re: openmptcprouter setup and ping overhead

Selected encryption is chacha20, I have a rock64 board that I use for Android TV app development, but it's not on the supported list and I'm guessing I'd need multiple USB to ethernet adapters (Are those even stable?) I could use my alternative PC with an I7 but I cannot, for the life of me, find a network card with 4 ethernet ports in lebanon.

Beta0

Member

Re: openmptcprouter setup and ping overhead

Selected encryption is chacha20, I have a rock64 board that I use for Android TV app development, but it's not on the supported list and I'm guessing I'd need multiple USB to ethernet adapters (Are those even stable?) I could use my alternative PC with an I7 but I cannot, for the life of me, find a network card with 4 ethernet ports in lebanon.

I see, I simply use a gigabit switch, and assign IP, only one port is needed for OMR board, if you want to go with the USB route, ASIX based USB ethernets are very reliable, not to mention they're the only ones to go near 930 MBit, I've had my experience with them for a temporary ESXi server during maintenance. But OMR supports MACVLAN (you don't even need VLAN), so that even works with unmanaged switch (what I'm currently using). MACVLAN is simply MAC address spoofing. MACVLAN is limited to 8 separate internet connections in OMR out of one gigabit port, while VLAN is open ended.

RandomMemory

Member

Re: openmptcprouter setup and ping overhead

So Instead of getting multiple network cards and adapters I can get a solid switch and do VLANs on 1 port?

Last edited by RandomMemory (November 30 2020)

Beta0

Member

Re: openmptcprouter setup and ping overhead

So Instead of getting multiple network cards and adapters I can get a solid switch and do VLANs on 1 port?

Yes possible, MACVLAN may be simpler as OMR use it by default. If you want to assign VLAN, you have to write the interface name in "Add new interface @network page", followed by dot then vlan number. Ex: eth0 is the port, eth0.55 = VLAN tag 55. VLAN may affect MTU, though OMR constantly adjusts the MTU every 30 seconds per interface, I haven't tested the performance with VLAN, though I doubt it will affect it, I think it's only 4 bytes, performance is only a concern with VLAN due to it being software VLAN, unlike dedicated switches which have 0 impact. RPi3 ethernet switch hardware accelerated VLAN mode is quirky for example, even Intel based PCI ethernet suck. My setup: RPi3 + 10 dollar 5 port unmanaged gigabit network switch, you can also use any scrapped router as unmanaged switch. I recommend for each ISP to have it's own modem/router, e.g don't use TerraNet PPPoE connection in OMR. TerraNet WDSL gave me the credentials on paper btw, they asked if I needed a router. OMR doesn't like network adapters going down, PPPoE may go down due to provider issues, thus interrupting all connections (other ISPs), static IP only is advised, and let a cheapo router do the PPPoE, just like any ADSL modem.

TPL:
Terranet dish (ethernet)---> router --->
                                                                  switch ---> OMR RPi3
Ogero phone line      ----> modem --->         ^
                                                                      |
Dumb bridged WiFi APs --------------------------
                                                                      |
Ethernet devices ----------- switch ---------------
                                              |
            (optional depending on main switch no. of ports)

Total cost excluding WiFi APs was around 90 USD back then. (incl. extra Gb switch)

Last edited by Beta0 (December 1 2020)

RandomMemory

Member

Re: openmptcprouter setup and ping overhead

Okay, so I was being scammed I guess by the reseller, Ended up calling terranet and getting a direct subscription with them, the installation is scheduled for the 10th so I'll have to test all this stuff out when the connection is ready. How is Terranet's non parallels download speeds. I suffer with large github repos downloading at 40KB/s because github doesn't allow multiple connections while downloading a single zip.

Last edited by RandomMemory (December 2 2020)

Beta0

Member

Re: openmptcprouter setup and ping overhead

Okay, so I was being scammed I guess by the reseller, Ended up calling terranet and getting a direct subscription with them, the installation is scheduled for the 10th so I'll have to test all this stuff out when the connection is ready. How is Terranet's non parallels download speeds. I suffer with large github repos downloading at 40KB/s because github doesn't allow multiple connections while downloading a single zip.

2Mbit plan, you can see the bit burst effect, overshooting 2.2Mbit due to their PCQ limiter (to keep latency down during congestion)
It also tricks dumb speed testing services like the famous Ookla one during congestion to show 2Mbit, while in reality it's around 1.8Mbit average during peak hours, I had the same effect with 10Mbit plan. I recommend using DSLReports for testing.

(LuCI does smoothing to the graph displaying wrong average unless I run it for 1 hour +)

This is to all Europe servers, doing so with far west American server gets it down to around 200, you can use Cloudflare WARP+ (argo) Wireguard profile as these act like a middle box and tailor TCP settings per client, most VPN providers don't. This issue is universal to any ISP around the world, and it's exaggerated on 100mbit+ connections.

Last edited by Beta0 (December 2 2020)

RandomMemory

Member

Re: openmptcprouter setup and ping overhead

Okay, so I was being scammed I guess by the reseller, Ended up calling terranet and getting a direct subscription with them, the installation is scheduled for the 10th so I'll have to test all this stuff out when the connection is ready. How is Terranet's non parallels download speeds. I suffer with large github repos downloading at 40KB/s because github doesn't allow multiple connections while downloading a single zip.

2Mbit plan, you can see the bit burst effect, overshooting 2.2Mbit due to their PCQ limiter (to keep latency down during congestion)
It also tricks dumb speed testing services like the famous Ookla one during congestion to show 2Mbit, while in reality it's around 1.8Mbit average during peak hours, I had the same effect with 10Mbit plan. I recommend using DSLReports for testing.

https://i.ibb.co/bBmxnLy/image.png
(LuCI does smoothing to the graph displaying wrong average unless I run it for 1 hour +)

This is to all Europe servers, doing so with far west American server gets it down to around 200, you can use Cloudflare WARP+ (argo) Wireguard profile as these act like a middle box and tailor TCP settings per client, most VPN providers don't. This issue is universal to any ISP around the world, and it's exaggerated on 100mbit+ connections.

Thank you for your detailed responses. Much appreciated.

RandomMemory

Member

Re: openmptcprouter setup and ping overhead

My final solution was to buy an extra 1 port network card for the terranet incoming connection. I installed proxmox on an i7 3770 computer. Then Installed pfsense and openmptcrouter in proxmox, pfsense handles the 2 incoming WANs, then made 2 rules in the firewall to make 2 certain IPs use WAN1 and WAN2 accordingly, Then I made 2 virtual interfaces in openmptcp and used those are incoming WAN connections and passed the bonded connection back to Pfsense as a third WAN to distribute it over the house with DHCP. Was very fun. (I still didn't get the terranet installation, but when I do it will be plug and play)

Last edited by RandomMemory (December 6 2020)

Beta0

Member

Re: openmptcprouter setup and ping overhead

My final solution was to buy an extra 1 port network card for the terranet incoming connection. I installed proxmox on an i7 3770 computer. Then Installed pfsense and openmptcrouter in proxmox, pfsense handles the 2 incoming WANs, then made 2 rules in the firewall to make 2 certain IPs use WAN1 and WAN2 accordingly, Then I made 2 virtual interfaces in openmptcp and used those are incoming WAN connections and passed the bonded connection back to Pfsense as a third WAN to distribute it over the house with DHCP. Was very fun. (I still didn't get the terranet installation, but when I do it will be plug and play)

That's nice! Did you use static routing or double nat (port forwarding all ports)?

RandomMemory

Member

Re: openmptcprouter setup and ping overhead

I ended up doing double nat and made a DMZ interface on pfsense

RandomMemory

Member

Re: openmptcprouter setup and ping overhead

Terranet installed today for me, this is the current speed with another bonded WAN that is 4mbps at this time of day

Beta0

Member

Re: openmptcprouter setup and ping overhead

Terranet installed today for me, this is the current speed with another bonded WAN that is 4mbps at this time of day https://www.speedtest.net/result/10627567849.png

Interesting, was this done with "Single" mode?
What is the monthly cost for this setup? I'm paying 120k LBP for Ogero openspeed(200gb) + Terranet 2mbit. Getting aggregated 15 download and 3 up.,

RandomMemory

Member

Re: openmptcprouter setup and ping overhead

Well for some reason the VPN tunnel always shows as down but it's still bonding if that's what you mean by single mode. This setup is expensive for someone who is not working remotely or getting paid in USD. It's 150k LBP for local wireless guy where the speed goes from 20mbit at 5 am to 7 at 12 pm to 3.5 after 4 pm. Terranet is 12mbps at 270K LBP. so I guess around 420 a month. For me, this lets me do my work extremely efficiently and removes every headache I've had for the past 5 years, so 100% worth it. Also local cable resellers seem to always have high upload speeds for some reason and ogero says my max line speed is 1mbps (imagine 1mbps in 2020)

Last edited by RandomMemory (December 21 2020)

Beta0

Member

Re: openmptcprouter setup and ping overhead

Well for some reason the VPN tunnel always shows as down but it's still bonding if that's what you mean by single mode. This setup is expensive for someone who is not working remotely or getting paid in USD. It's 150k LBP for local wireless guy where the speed goes from 20mbit at 5 am to 7 at 12 pm to 3.5 after 4 pm. Terranet is 12mbps at 270K LBP. so I guess around 420 a month. For me, this lets me do my work extremely efficiently and removes every headache I've had for the past 5 years, so 100% worth it. Also local cable resellers seem to always have high upload speeds for some reason and ogero says my max line speed is 1mbps (imagine 1mbps in 2020)

Wow that's pretty bad, 420K is very high, VPN down but UDP is still aggregating? Just an info for others on how this works:

               | WAN 1____________________
               |                                                   |
   VPS __|  TCP to MPTCP by the kernel  |__Shadowsocks TCP proxy__Glorytun TCP VPN
               | WAN 2___________________|                   |                                          |
                                                                                     |______________________|
                                                                                       \       Firewall splitting       /
                                                                                          |__________________|
                                                                                       TCP                         Everything else minus TCP
                                                                                              \                         /
                                                                                                       LAN

TCP through VPN:
--------------------|\    VPN   /-------        MPTCP in this case is congested and
--------------------| }----------{--------       doing paths on one TCP connection
--------------------|/              \-------        via VPN, lowering performance.
TCP through Proxy:
--------------------|\ ---Proxy--/-------Side note:Glorytun TCP VPN which is
--------------------| }-----------{-------- responsible for (UDP and others) is the only path
--------------------|/ ------------\------- that is congested.(one TCP connection for possibly hundred UDP)

Shadowsocks is used to implement "TCP through Proxy"

UDP based clients are smart enough to be not affected by this, they implement their own packet flow algo, unlike TCP which is a very old standard.
OMR is configured to prioritize Glorytun by marking it with DSCP, then the prioritization is done and detected at Shorewall on the VPS.

Last edited by Beta0 (December 22 2020)

RandomMemory

Member

Re: openmptcprouter setup and ping overhead

what do you recommend to fix it. How is my IP showing up from OVH, is that the proxy's work?

Beta0

Member

Re: openmptcprouter setup and ping overhead

what do you recommend to fix it. How is my IP showing up from OVH, is that the proxy's work?

Sorry, late edit. See above, I recommend hitting Save and apply in the wizard again, see if that fix it, else look in the System log to see if there is any error.
And yeah, most websites use HTTP1/2 which is TCP, what shadowsocks serve. Youtube, google and few others with the latest Chrome (Windows) are using QUIC/HTTP3 which is UDP, but it fallbacks to HTTP2 if UDP/QUIC is blocked or unsupported. OMR actually blocks QUIC and HTTP3 since TCP over MPTCP performs better in this setup instead of being tunneled via the UDP VPN (glorytun). QUIC/HTTP3 is better on UDP only VPN or direct connection.

Last edited by Beta0 (December 21 2020)