Selected encryption is chacha20, I have a rock64 board that I use for Android TV app development, but it's not on the supported list and I'm guessing I'd need multiple USB to ethernet adapters (Are those even stable?) I could use my alternative PC with an I7 but I cannot, for the life of me, find a network card with 4 ethernet ports in lebanon.
openmptcprouter setup and ping overhead
I see, I simply use a gigabit switch, and assign IP, only one port is needed for OMR board, if you want to go with the USB route, ASIX based USB ethernets are very reliable, not to mention they're the only ones to go near 930 MBit, I've had my experience with them for a temporary ESXi server during maintenance. But OMR supports MACVLAN (you don't even need VLAN), so that even works with unmanaged switch (what I'm currently using). MACVLAN is simply MAC address spoofing. MACVLAN is limited to 8 separate internet connections in OMR out of one gigabit port, while VLAN is open ended.RandomMemory wroteSelected encryption is chacha20, I have a rock64 board that I use for Android TV app development, but it's not on the supported list and I'm guessing I'd need multiple USB to ethernet adapters (Are those even stable?) I could use my alternative PC with an I7 but I cannot, for the life of me, find a network card with 4 ethernet ports in lebanon.
- Edited
So Instead of getting multiple network cards and adapters I can get a solid switch and do VLANs on 1 port?
- Edited
Yes possible, MACVLAN may be simpler as OMR use it by default. If you want to assign VLAN, you have to write the interface name in "Add new interface @network page", followed by dot then vlan number. Ex: eth0 is the port, eth0.55 = VLAN tag 55. VLAN may affect MTU, though OMR constantly adjusts the MTU every 30 seconds per interface, I haven't tested the performance with VLAN, though I doubt it will affect it, I think it's only 4 bytes, performance is only a concern with VLAN due to it being software VLAN, unlike dedicated switches which have 0 impact. RPi3 ethernet switch hardware accelerated VLAN mode is quirky for example, even Intel based PCI ethernet suck. My setup: RPi3 + 10 dollar 5 port unmanaged gigabit network switch, you can also use any scrapped router as unmanaged switch. I recommend for each ISP to have it's own modem/router, e.g don't use TerraNet PPPoE connection in OMR. TerraNet WDSL gave me the credentials on paper btw, they asked if I needed a router. OMR doesn't like network adapters going down, PPPoE may go down due to provider issues, thus interrupting all connections (other ISPs), static IP only is advised, and let a cheapo router do the PPPoE, just like any ADSL modem.RandomMemory wroteSo Instead of getting multiple network cards and adapters I can get a solid switch and do VLANs on 1 port?
TPL:
Terranet dish (ethernet)---> router --->
switch ---> OMR RPi3
Ogero phone line ----> modem ---> ^
|
Dumb bridged WiFi APs --------------------------
|
Ethernet devices ----------- switch ---------------
|
(optional depending on main switch no. of ports)
Total cost excluding WiFi APs was around 90 USD back then. (incl. extra Gb switch)
- Edited
Okay, so I was being scammed I guess by the reseller, Ended up calling terranet and getting a direct subscription with them, the installation is scheduled for the 10th so I'll have to test all this stuff out when the connection is ready. How is Terranet's non parallels download speeds. I suffer with large github repos downloading at 40KB/s because github doesn't allow multiple connections while downloading a single zip.
- Edited
2Mbit plan, you can see the bit burst effect, overshooting 2.2Mbit due to their PCQ limiter (to keep latency down during congestion)RandomMemory wroteOkay, so I was being scammed I guess by the reseller, Ended up calling terranet and getting a direct subscription with them, the installation is scheduled for the 10th so I'll have to test all this stuff out when the connection is ready. How is Terranet's non parallels download speeds. I suffer with large github repos downloading at 40KB/s because github doesn't allow multiple connections while downloading a single zip.
It also tricks dumb speed testing services like the famous Ookla one during congestion to show 2Mbit, while in reality it's around 1.8Mbit average during peak hours, I had the same effect with 10Mbit plan. I recommend using DSLReports for testing.

(LuCI does smoothing to the graph displaying wrong average unless I run it for 1 hour +)
This is to all Europe servers, doing so with far west American server gets it down to around 200, you can use Cloudflare WARP+ (argo) Wireguard profile as these act like a middle box and tailor TCP settings per client, most VPN providers don't. This issue is universal to any ISP around the world, and it's exaggerated on 100mbit+ connections.
Thank you for your detailed responses. Much appreciated.Beta0 wrote2Mbit plan, you can see the bit burst effect, overshooting 2.2Mbit due to their PCQ limiter (to keep latency down during congestion)RandomMemory wroteOkay, so I was being scammed I guess by the reseller, Ended up calling terranet and getting a direct subscription with them, the installation is scheduled for the 10th so I'll have to test all this stuff out when the connection is ready. How is Terranet's non parallels download speeds. I suffer with large github repos downloading at 40KB/s because github doesn't allow multiple connections while downloading a single zip.
It also tricks dumb speed testing services like the famous Ookla one during congestion to show 2Mbit, while in reality it's around 1.8Mbit average during peak hours, I had the same effect with 10Mbit plan. I recommend using DSLReports for testing.
https://i.ibb.co/bBmxnLy/image.png
(LuCI does smoothing to the graph displaying wrong average unless I run it for 1 hour +)
This is to all Europe servers, doing so with far west American server gets it down to around 200, you can use Cloudflare WARP+ (argo) Wireguard profile as these act like a middle box and tailor TCP settings per client, most VPN providers don't. This issue is universal to any ISP around the world, and it's exaggerated on 100mbit+ connections.
- Edited
My final solution was to buy an extra 1 port network card for the terranet incoming connection. I installed proxmox on an i7 3770 computer. Then Installed pfsense and openmptcrouter in proxmox, pfsense handles the 2 incoming WANs, then made 2 rules in the firewall to make 2 certain IPs use WAN1 and WAN2 accordingly, Then I made 2 virtual interfaces in openmptcp and used those are incoming WAN connections and passed the bonded connection back to Pfsense as a third WAN to distribute it over the house with DHCP. Was very fun. (I still didn't get the terranet installation, but when I do it will be plug and play)
That's nice! Did you use static routing or double nat (port forwarding all ports)?RandomMemory wroteMy final solution was to buy an extra 1 port network card for the terranet incoming connection. I installed proxmox on an i7 3770 computer. Then Installed pfsense and openmptcrouter in proxmox, pfsense handles the 2 incoming WANs, then made 2 rules in the firewall to make 2 certain IPs use WAN1 and WAN2 accordingly, Then I made 2 virtual interfaces in openmptcp and used those are incoming WAN connections and passed the bonded connection back to Pfsense as a third WAN to distribute it over the house with DHCP. Was very fun. (I still didn't get the terranet installation, but when I do it will be plug and play)
I ended up doing double nat and made a DMZ interface on pfsense
14 days later
Terranet installed today for me, this is the current speed with another bonded WAN that is 4mbps at this time of day 

Interesting, was this done with "Single" mode?RandomMemory wroteTerranet installed today for me, this is the current speed with another bonded WAN that is 4mbps at this time of day https://www.speedtest.net/result/10627567849.png
What is the monthly cost for this setup? I'm paying 120k LBP for Ogero openspeed(200gb) + Terranet 2mbit. Getting aggregated 15 download and 3 up.,
- Edited
Well for some reason the VPN tunnel always shows as down but it's still bonding if that's what you mean by single mode. This setup is expensive for someone who is not working remotely or getting paid in USD. It's 150k LBP for local wireless guy where the speed goes from 20mbit at 5 am to 7 at 12 pm to 3.5 after 4 pm. Terranet is 12mbps at 270K LBP. so I guess around 420 a month. For me, this lets me do my work extremely efficiently and removes every headache I've had for the past 5 years, so 100% worth it. Also local cable resellers seem to always have high upload speeds for some reason and ogero says my max line speed is 1mbps (imagine 1mbps in 2020)
- Edited
Wow that's pretty bad, 420K is very high, VPN down but UDP is still aggregating? Just an info for others on how this works:RandomMemory wroteWell for some reason the VPN tunnel always shows as down but it's still bonding if that's what you mean by single mode. This setup is expensive for someone who is not working remotely or getting paid in USD. It's 150k LBP for local wireless guy where the speed goes from 20mbit at 5 am to 7 at 12 pm to 3.5 after 4 pm. Terranet is 12mbps at 270K LBP. so I guess around 420 a month. For me, this lets me do my work extremely efficiently and removes every headache I've had for the past 5 years, so 100% worth it. Also local cable resellers seem to always have high upload speeds for some reason and ogero says my max line speed is 1mbps (imagine 1mbps in 2020)
| WAN 1____________________
| |
VPS __| TCP to MPTCP by the kernel |__Shadowsocks TCP proxy__Glorytun TCP VPN
| WAN 2___________________| | |
|______________________|
\ Firewall splitting /
|__________________|
TCP Everything else minus TCP
\ /
LAN
TCP through VPN:
--------------------|\ VPN /------- MPTCP in this case is congested and
--------------------| }----------{-------- doing paths on one TCP connection
--------------------|/ ------- via VPN, lowering performance.
TCP through Proxy:
--------------------|\ ---Proxy--/-------Side note:Glorytun TCP VPN which is
--------------------| }-----------{-------- responsible for (UDP and others) is the only path
--------------------|/ ------------------- that is congested.(one TCP connection for possibly hundred UDP)
Shadowsocks is used to implement "TCP through Proxy"
UDP based clients are smart enough to be not affected by this, they implement their own packet flow algo, unlike TCP which is a very old standard.
OMR is configured to prioritize Glorytun by marking it with DSCP, then the prioritization is done and detected at Shorewall on the VPS.
what do you recommend to fix it. How is my IP showing up from OVH, is that the proxy's work?
- Edited
Sorry, late edit. See above, I recommend hitting Save and apply in the wizard again, see if that fix it, else look in the System log to see if there is any error.RandomMemory wrotewhat do you recommend to fix it. How is my IP showing up from OVH, is that the proxy's work?
And yeah, most websites use HTTP1/2 which is TCP, what shadowsocks serve. Youtube, google and few others with the latest Chrome (Windows) are using QUIC/HTTP3 which is UDP, but it fallbacks to HTTP2 if UDP/QUIC is blocked or unsupported. OMR actually blocks QUIC and HTTP3 since TCP over MPTCP performs better in this setup instead of being tunneled via the UDP VPN (glorytun). QUIC/HTTP3 is better on UDP only VPN or direct connection.
Terranet is 12Mbps all day? or does it fluctuate? what is its upload speed alone?RandomMemory wroteWell for some reason the VPN tunnel always shows as down but it's still bonding if that's what you mean by single mode. This setup is expensive for someone who is not working remotely or getting paid in USD. It's 150k LBP for local wireless guy where the speed goes from 20mbit at 5 am to 7 at 12 pm to 3.5 after 4 pm. Terranet is 12mbps at 270K LBP. so I guess around 420 a month. For me, this lets me do my work extremely efficiently and removes every headache I've had for the past 5 years, so 100% worth it. Also local cable resellers seem to always have high upload speeds for some reason and ogero says my max line speed is 1mbps (imagine 1mbps in 2020)
- Edited
I'll update the image at night if vnstat does not clip few hours, this is from yesterday 12AM till now, wan2 is terranet 2mbit plan. I would be interested to see if the 12mbit plan is the same, I had 10mbit for 2 months with no issues but that was almost 2 years ago.wollyka wroteTerranet is 12Mbps all day? or does it fluctuate? what is its upload speed alone?RandomMemory wroteWell for some reason the VPN tunnel always shows as down but it's still bonding if that's what you mean by single mode. This setup is expensive for someone who is not working remotely or getting paid in USD. It's 150k LBP for local wireless guy where the speed goes from 20mbit at 5 am to 7 at 12 pm to 3.5 after 4 pm. Terranet is 12mbps at 270K LBP. so I guess around 420 a month. For me, this lets me do my work extremely efficiently and removes every headache I've had for the past 5 years, so 100% worth it. Also local cable resellers seem to always have high upload speeds for some reason and ogero says my max line speed is 1mbps (imagine 1mbps in 2020)
Downloading MSFS whopping 150GB :)

I expect direct usage will maintain 2mbit average, this is proxied over MPTCP, single flow.
- Edited
Thanks for the great read... I tried changing all settings and for some reason none of them were able to put the VPN tunnel up again. This morning there was an update. I did an update on the server and it's working now (for some reason it's saying terranet has multipath blocked.Beta0 wroteSorry, late edit. See above, I recommend hitting Save and apply in the wizard again, see if that fix it, else look in the System log to see if there is any error.RandomMemory wrotewhat do you recommend to fix it. How is my IP showing up from OVH, is that the proxy's work?
And yeah, most websites use HTTP1/2 which is TCP, what shadowsocks serve. Youtube, google and few others with the latest Chrome (Windows) are using QUIC/HTTP3 which is UDP, but it fallbacks to HTTP2 if UDP/QUIC is blocked or unsupported. OMR actually blocks QUIC and HTTP3 since TCP over MPTCP performs better in this setup instead of being tunneled via the UDP VPN (glorytun). QUIC/HTTP3 is better on UDP only VPN or direct connection.

this is my speedtest right now with terra being at 12mbps and the other ISP almost dying...

I am only keeping the other ISP because they have 20mbps post 2 am and 15mbps cache on all popular services all day (netflix, youtube)
- Edited
It's a symmetrical up and down. I have the connection for 2 days and freaking amazing. it's a consistent 11-11.5mbps all day and it's unlimited. I also have their IPTV subscription running on 2 TVs which takes 6mbps on each TV but the route is different from your internet so it will use up intranet bandwidth instead of internet bandwidth.wollyka wroteTerranet is 12Mbps all day? or does it fluctuate? what is its upload speed alone?RandomMemory wroteWell for some reason the VPN tunnel always shows as down but it's still bonding if that's what you mean by single mode. This setup is expensive for someone who is not working remotely or getting paid in USD. It's 150k LBP for local wireless guy where the speed goes from 20mbit at 5 am to 7 at 12 pm to 3.5 after 4 pm. Terranet is 12mbps at 270K LBP. so I guess around 420 a month. For me, this lets me do my work extremely efficiently and removes every headache I've had for the past 5 years, so 100% worth it. Also local cable resellers seem to always have high upload speeds for some reason and ogero says my max line speed is 1mbps (imagine 1mbps in 2020)