openmptcprouter setup and ping overhead

My quick and dirty review:

Ping during torrent, the two spikes in the graph are only 20ms with 182 connections, TCP low latency mode is checked, MPTCP over VPN is mandatory for Ogero (filtered). Terranet is the master MPTCP interface as there is no VPN being used to enable MPTCP, but it works vice versa as well, just slower to connect cold start.
Terranet WDSL + Ogero ADSL. BLEST algorithm is selected for heterogenous links, the default is better for 2 lossy links but it degrades speed while the latency remains the same. Ogero is not lossy in packets, it just keeps disconnecting 60 times per day, while Terranet around 3 times a month.

Terranet MPTCP eligibility tracebox example:


I've went through 8 providers in the past 2 years, so far only OVH and Vultr have high uptime longer than 10 months, e.g: outage is 15 mins every 10 months. They did remind me 1 week before it happened.
I've yet to see any provider other than Vultr offering high frequency (3.8ghz) 6 dollar VPS without shared utilization. It offers the best interface as well.
Vultr high frequency (6$) vs OVH ($6):


I'm getting 7 ms lower with Vultr HF Paris compared to OVH.

One culprit with OMR is the status page and applied changes delays, you should wait around 1 minute before the real status / change appears or happens due to the 10+ scripts running in the background, this can be really confusing for starters . Another less concerning culprit is the author's childish elitism attitude towards people running or using Windows on github issues/IRC, as someone who worked on the Glorytun part it wasn't a great experience contributing to the project, including OpenWRT community.
This was running fine over HyperV in my homelab, though I prefer a dedicated device for months long uptimes, Raspberry Pi 3 is more than enough for <500mbit of total aggregated link speed. My Ryzen 4500u laptop could do 4.2 gigabit with the VPS part in local VM.
Plugging 4g modem, Android, iPhone in tethering to the RPi automatically adds the connection and aggregates without interruption or any packet loss, I often use that to boost internet speed for the entire house with no interruption, from 15mbit (Terra+Ogero) to 50mbit with Alfa unlimited, very useful while uploading very large files without pausing/restarting. Was also useful during 1080p wide angle TrueConf conference call, not a single frame dropped or stuttered and the image improved immediately without restarting the call. This also allows iOS devices to transition between mobile data and home network during video/voice calls without any signs of switching, as Apple is an early adopter of MPTCP, facetime and even apple.com website has MPTCP enabled. iOS switches to redundancy mode first, to aggregation, then disables mobile data once it verifies WiFi is good, tested and working perfectly with OMR, Google Duo implementation only works with ProjectFi, I hope they open it up. Alfa and Touch do not filter MPTCP thankfully. MPTCP enabled websites somehow behave the same as HTTP3 (udp) even if they are still running HTTP1.1, latest Linux kernels 5.4+ have it enabled by default. You may also notice that single stream downloads are boosted due to having subflows (starts at 4), so even using OMR for a single connection "as a VPS" improves internet connection speeds for high latency sources, e.g a Californian website with >10 sources loading, that include ads, fonts, bootstrap and other non static frameworks etc will be improved as HTTP1.1 in browsers is single threaded. Some software downloaders such as Epic, Android studio are single threaded. Secure unique links are usually single, encountered that with Uni's website, researchgate and paid download sites as well. This is where multi thread download (accelerators) don't work such as IDM, but MPTCP does. Keep in mind that, OMR does not use MPTCP to do aggregation only, it also exposes/enables MPTCP for clients with the VPS's public IP. iOS so far utilizes that, technically bypassing OMR aggregation proxy and VPN for Apple servers.

Netflix bans datacenter IPs, not just VPNs, you can enable bypass to each interface based on DPI protocol filtering:

You can also use this for load balancing. About load balancing, if the VPS is down, OMR will switch to load balancing mode, it even has a mode that fallbacks to the lowest latency link, or the faster one as a choice. (dynamic - balancing). OMR has forked and improved VNstat interface for LuCI compared to default OpenWRT, it saves the data as well.


Edit1: Another note, do not enable SQM for buffer bloat, it is not needed, and you will get lower speeds with wireless providers if they are behind ordered buffer queue such as Terranet's. BBR is enabled by default, since both the VPS and OMR are using BBR, it works effectively removing buffer bloat from the Shadowsocks TCP connection (this is the pipe to mptcp for non mptcp connections, another VPN runs over this proxy/pipe for UDP, ICMP and everything else, MPTCP is bypassed), assuming that the VPS to the outside world doesnt have bufferbloat of coarse. Usually TCP congestions algos are useless if the other server does not use the same algo, and even if does that is TCP only, and mostly useful for uploads.

For a beginner you should be up and running in less than 30 minutes, no need to tune anything, this detailed post may make this sound too complicated ;)

I may shoot a video physically ripping one of the ISP's modem cable while showing CSGO's jitter net graph if it sparks anyone's interest. 0% dropped frames in Twitch (OBS) as well.

About locations: Paris is defiantly the way to go, I'm getting only 5-10ms extra to European and American servers compared to barebone (interleaving disabled) Ogero. On a sidenote, Terranet WDSL has lower pings compared to Ogero. Pinging to 1.1.1.1 or 8.8.8.8 does not go through Europe on Ogero/Wireless, that's why you get 40-55ms, it goes through IXP from a near datacenter, Cloudflare is near Terranet's building, near Berytech, they use microwave. I recommend to always ping to a server outside Lebanon when doing comparisions, that's why the ping graph in the top post is 65ms, pinging to Cloudflare Paris is around 62ms, Lebanon 48ms, OMR or any VPN from Paris will only affect latency to Lebanese servers, which are almost non existent, you can still bypass. (e.g Lebanese csgo server)

Edit2: To save the trouble for others, Speedify no longer aggregates packets, except iPad with multiple USB ethernet, and it does it at userspace level with bad latency and 100% cpu usage. It's a fancy load balancer with a single IP endpoint (the vpn part) to avoid session dropouts, there is still a 5 second packet loss each time a connection is down, udp based services will drop, only downloads and tcp services such as ssh will remain but with a 5 second of inactivity/zero speed. They use nDPI to detect video calls and certain games and do FIFO packet duplication from each interface to the VPN, known as redundant mode, also used for "Streaming mode" not channel bonding. Their IP addresses are on the watch list of many captcha/verification security services, expect tons of suspicious activity captcha from a simple google search, and their nDPI does way too many false positives on torrent detection, switching you to a very congested Amsterdam server, dropping calls easily. Only the team edition with private server implements packet aggregation, and only on Linux/iPad, costs more than $200 a month with evaluation discount, very PITA to setup as a router on Debian, heavily relies on outdated Network Manager, requires Linux knowledge. Peplink services are optimized for TDD links such as LTE, else you have to manually tune it if one of the non-TDD ISP's speed go down, affecting latency, ~200ms on load, pricey as well, suitable for non-satellite media coverage since their streaming service does dynamic artificial latency 500-1500ms for time skew buffer, same way Youtube "Ultra low latency" work.

Edit3: I recommend using CUBIC congestion control only when using BLEST, the default BBR is too aggressive with MPTCP and may cause large buffer bloat, MP-BBR is coming soon.

samer wroteRegarding VPS providers, your best bet is to run some latency tests yourself to figure out which region provides the lowest latency. I would imagine that European servers will yield better results than US or Asia.
Try this out: https://cloudpingtest.com/

Great site! Unfortunately Vultr is scoring consistently 30ms higher than my Vultr server in Paris, seems to be using a large HTTP ping, not ICMP, or my browser/pc is busy.
Most providers offer looking glass service open, they have test servers and IPs that you can ping to as well.

RandomMemory wroteFirst of all thanks a lot for the detailed explanation, I was trying with one connection today since my terranet installation was pushed back to this week so doing preparations I guess. I noticed that after enabling the OpenMPTCPRouter server my speed dropped in half. I am using an dell optilex (With an intel pentium) for this since I was using the same machine for pfsense load balancing. I noticed that the fans started kicking in really loud after this option got enabled and I tried doing a couple of speed tests. If I disable the server and connect to a normal vpn I get a solid 12-14mbps. After enabling the server The speed dips to around 5mbps-6mbps. The server is rented from OVH and it's located in france. Do I need a stronger machine for this or should I try some other configurations.

No problem, that's odd, how old is the Pentium? you can try running OMR in a VM on a Linux desktop on that machine, openwrt kernel is incomplete for very old systems in order to reduce size. Is this with MPTCP over VPN enabled?

RandomMemory wroteIntel(R) Pentium(R) 4 HT CPU 3.00GHz
2 CPUs: 1 package(s) x 2 hardware threads
AES-NI CPU Crypto: No

MPTCP over VPN is enabled

Ah that's too old, OMR auto selects encryption based on what VPS supports, OVH and Vultr KVMs do have AES enabled, it assumes that the client (your desktop) is always faster than the VPS and it probably is doing AES. In the wizard page, what is selected as encryption under advanced settings? I guess chacha20 may be a bit faster with Pentium 4, not to mention it favors 4 cores. My orange pi zero ($12) had better performance than the first gen atom based celeron (2c) from 2012, though I do not recommend any Rockchip SoCs as they're unstable with the latest kernels.

Edit1: Even with a recent Celeron quadcore setup (no HT), with peg causing 10-20 ms extra latency on full load with tether:

RandomMemory wroteSo Instead of getting multiple network cards and adapters I can get a solid switch and do VLANs on 1 port?

Yes possible, MACVLAN may be simpler as OMR use it by default. If you want to assign VLAN, you have to write the interface name in "Add new interface @network page", followed by dot then vlan number. Ex: eth0 is the port, eth0.55 = VLAN tag 55. VLAN may affect MTU, though OMR constantly adjusts the MTU every 30 seconds per interface, I haven't tested the performance with VLAN, though I doubt it will affect it, I think it's only 4 bytes, performance is only a concern with VLAN due to it being software VLAN, unlike dedicated switches which have 0 impact. RPi3 ethernet switch hardware accelerated VLAN mode is quirky for example, even Intel based PCI ethernet suck. My setup: RPi3 + 10 dollar 5 port unmanaged gigabit network switch, you can also use any scrapped router as unmanaged switch. I recommend for each ISP to have it's own modem/router, e.g don't use TerraNet PPPoE connection in OMR. TerraNet WDSL gave me the credentials on paper btw, they asked if I needed a router. OMR doesn't like network adapters going down, PPPoE may go down due to provider issues, thus interrupting all connections (other ISPs), static IP only is advised, and let a cheapo router do the PPPoE, just like any ADSL modem.

TPL:
Terranet dish (ethernet)---> router --->
switch ---> OMR RPi3
Ogero phone line ----> modem ---> ^
|
Dumb bridged WiFi APs --------------------------
|
Ethernet devices ----------- switch ---------------
|
(optional depending on main switch no. of ports)

Total cost excluding WiFi APs was around 90 USD back then. (incl. extra Gb switch)

RandomMemory wroteOkay, so I was being scammed I guess by the reseller, Ended up calling terranet and getting a direct subscription with them, the installation is scheduled for the 10th so I'll have to test all this stuff out when the connection is ready. How is Terranet's non parallels download speeds. I suffer with large github repos downloading at 40KB/s because github doesn't allow multiple connections while downloading a single zip.

2Mbit plan, you can see the bit burst effect, overshooting 2.2Mbit due to their PCQ limiter (to keep latency down during congestion)
It also tricks dumb speed testing services like the famous Ookla one during congestion to show 2Mbit, while in reality it's around 1.8Mbit average during peak hours, I had the same effect with 10Mbit plan. I recommend using DSLReports for testing.


(LuCI does smoothing to the graph displaying wrong average unless I run it for 1 hour +)

This is to all Europe servers, doing so with far west American server gets it down to around 200, you can use Cloudflare WARP+ (argo) Wireguard profile as these act like a middle box and tailor TCP settings per client, most VPN providers don't. This issue is universal to any ISP around the world, and it's exaggerated on 100mbit+ connections.