Hi everyone,

I have just finished getting this solution working, and thought I'd post it in the hope that Google will find it and save some people some time.

Also, thank you very much to the ClearOS developers - it's an awesome system!

---------------------------------

How to Configure a Load Balancing 3G Multi-WAN Router + Wireless AP

Objectives

[ol]

We wish to run open source events at locations that may have bad internet, or not any internet at all. So we need to be able to provide internet to 30 - 50 laptops. The 3G keys of all 3 network providers in our country give between 2mbps and 8mbps downstream, and 0.5 to 2mbps upstream, depending on the location and time of day. Say on average, 3mbps downstream, is not enough for 30 laptops. So we need to load balance WAN those laptops across as many 3G USB keys as possible.

At many of these events, we will be doing Linux (Ubuntu) installs, and need a transparent way to cache repository .deb files. 40 downloads of large package updates (e.g KDE/Gnome) apt-get upgrades etc over 3G is simply not possible, even if we have 5 keys to load balance across). So after a deb is downloaded once, it should be fetched instantaneously over the local network, without going out over a WAN.

In addition to running free open source events, the internet where our office is located is unreliable. Sometimes the ADSL is the best. Other times a 3G USB key is better. We need an easy way to switch between networks, keeping our network intact when one connection goes down or becomes low performing.

[/ol]

Discussion

There are a number of Open Source firewalls available with Multi-WAN capabilities. From my reviews, the two best seem to be pfSense and ClearOS.
I first tried pfSense as many blog and forum posts I read say its Multi-WAN features are much better than ClearOS, and pfSense is also 100% free, where as ClearOS has a community and pro version which is paid. It's important to note though, that although there is a paid version, the ClearOS community version is extremely generous, and provides a huge number of features for free.
The problem with pfSense is that it's based on an old version of FreeBSD, and is very locked down. That makes doing anything out of the ordinary very difficult.
So in our situation, we needed to make the entire solution portable, which means we need to use a laptop. Any laptop within the last couple of years will have hardware without drivers in the pfSense distribution, and our biggest driver issue was because the entire solution needs to be easily portable, we need to use USB Ethernet adapters. The only ones available are cheap ones based that use the dm9601 driver, and this didn't appear to be available on the old version of FreeBSD that pfSense is based on. I couldn't find any easy instructions about how to backport or compile a driver in FreeBSD. I have very little BSD experience, so this may be possible and could be easy, but I couldn't figure out how to do it in a reasonable amount of time. In addition, not having apt repository makes other tasks difficult. It looks like pfSense blocks all this stuff intentionally in order to make the distribution more secure, which it achieves. But with our needs I'm happy to trade off some security in order to get greater flexibility and actually achieve our objective.

Final solution

The final solution we chose was as follows below. Please note that I'm well aware that USB Ethernet adapters are not highly performing and completely reliable, however they are the best fit for our needs for a cheap, portable solution.

[ul]

1x Laptop: Dell Inspiron 15 3521 with 4 USB ports, 1 RJ45 10/100mbps adapter

4x USB Ethernet Adapters:

-- 3x Kontron (Industrial Computer Source / ICS Advent) DM9601 Fast Ethernet Adapter

-- 1x Davicom Semiconductor, Inc. DM9601 Fast Ethernet Adapter

1x Wireless AP: TP-LINK TL-WA801ND 300Mbps Wireless N Access Point

3x 3G USB Key AP: TP-LINK TL-MR3020 3G/4G USB Key Access Point

3x 3G USB Keys (the system can support up to 5, if I can get the Atheros WiFi card working, 4 if I can't)

-- 2x Orange Tunisie 3G USB keys

-- 1x Tunisie Telecom 3G USB key

Also:

--1x 1.5m USB Extension cable to allow us to fit all the fat USB ethernet adapters in the closely spaced 3 USB ports on the Dell laptop.

--2x 4 plug multiplugs daisy chained to provide power.

[/ul]

Install ClearOS 6.5

[ol]
[li]Download ClearOS 6.5 beta2 from www.clearfoundation.com[/li]

In our case due to a hardware/driver issue, boot with kernel option biosdevname=0 or you will get anaconda error "keyerror: eth0" and the installer will crash and halt.

During the install process, leave 210GB free for Ubuntu (not compulsory, see below)

[/ol]

Install Ubuntu 13.10

This step is not compulsory. Of course you can dedicate the entire machine to ClearOS, making the install process a lot simpler. But in our case we wanted to have it Dual Boot between Ubuntu and ClearOS, defaulting to ClearOS. This could be useful if we need an emergency backup laptop for some reason.

[ol]

Install Ubuntu with basic options.

[/ol]

Fix Ubuntu/Grub

It doesn't appear that Ubuntu/ClearOS work very well together with their approach to bootloading (in default/simple install). So installing one next to the other wipes the other's bootloader config. Thus we need to fix Grub in order to be able to Dual Boot easily.

[ol]

Boot from Ubuntu

[li]Follow instructions at https://help.ubuntu.com/community/Boot-Repair[/li]
[li]Follow instructions at http://askubuntu.com/questions/100232/how-do-i-change-the-grub-boot-order[/li]

Change the boot order and set ClearOS to default

[/ol]

ClearOS Modifications/Hacks

Installing ClearOS is quick and easy, but depending on your hardware, you may have driver issues, especially if you use a laptop, and really especially if you use a new laptop. Of course these can usually be fixed by installing driver backports and fixing old drivers, as we had to do with our Dell Inspiron 3521 laptop and our Kontron and Davicom USB Ethernet adapters.

There were three problems with the USB Ethernet adapters.

[ol]

The dm9601 driver wasn't enabled. Solution: Add the kernel module to /etc/rc.modules

Some of the adapters weren't recognized by the driver, and came up as "UNCLAIMED". Solution: Hack driver and recompile

Some of the adapters had the same mac address, either a hardware or driver issue and so were mounted as "rename7", "rename8" etc. Solution: Buy different adapter variants to get different mac addresses.

[/ol]

1. Enable the dm9601 Driver

# nano /etc/rc.modules
------
modprobe usbnet
modprobe dm9601
------
# chmod +x /etc/rc.modules
# shutdown -r now

2. Hack dm9601 Driver

The driver provided in ClearOS doesn't recognize the newer 9700 variants of the dm9601 and thus the adapter will remain as "UNCLAIMED".

Upgrade Kernel, Install Kernel Headers and GCC

Log in via the console or via ssh

# yum --enablerepo=clearos-updates-testing install kernel-devel kernel-headers
# yum --enablerepo=clearos-updates-testing update kernel
# yum --enablerepo=clearos-updates-testing install gcc
# shutdown -r now

If you have installed Dual Boot Ubuntu, reboot into ubuntu and run grub-customizer. Move new ClearOS kernel to the top of the list.
Reboot back into new ClearOS Kernel. If you haven't installed this, Grub will automatically be updated with the latest kernel version becoming the default, so you won't need to do this step.

3. Download revision of dm9601.c for your kernel and compile

This section is based on http://tech.firdooze.com/2011/11/16/how-to-instal-davicom-9601-drivers-dm9601-on-linux/ - see there for more info.

# mkdir working && cd working
# yum install wget
# wget --no-check-certificate https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/plain/drivers/net/usb/dm9601.c?id=refs/tags/v2.6.32.61
# nano dm9601.c
Ctrl+w to search for manufacturer ID#: 0x0fe6
Copy and paste the USB Device lines so you have a new definition
In your new definition change 0x8101 to 0x9700
Save and exit
Save makefile from http://tech.firdooze.com/2011/11/16/how-to-instal-davicom-9601-drivers-dm9601-on-linux/ as Makefile
Fix the errors in the Makefile by adding tabs to the targets

# make

Copy the new driver dm9601.ko and overwrite the old one

# cp dm9601.ko /lib/modules/2.6.32-358.23.2.v6.i686/kernel/drivers/net/usb/
# shutdown -r now
# lshw -C network should now show USB Ethernet card as DISABLED

Now plug all three usb adapters into the computer, in order of port. Ordering is not required, it's just for clarity. Then lock their mac addresses to clearly named eth20, eth21, eth22 like so:

# nano /etc/udev/rules.d/70-persistent-net.rules
--------------------------------------------------------------------------------------------------------------
# PCI device 0x10ec:0x8136 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="74:86:7a:26:8e:a3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# USB device 0x0fe6:0x8101 (usb)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:10:13:50:01:58", ATTR{type}=="1", KERNEL=="eth*", NAME="eth20"

# USB device 0x0a46:0x9601 (usb)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:06:06:e0:39:ec", ATTR{type}=="1", KERNEL=="eth*", NAME="eth21"

# USB device 0x0fe6:0x9700 (usb)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:4c:53:44:58", ATTR{type}=="1", KERNEL=="eth*", NAME="eth22"

# USB device 0x0fe6:0x8101 (usb)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:10:13:50:a3:43", ATTR{type}=="1", KERNEL=="eth*", NAME="eth23"
--------------------------------------------------------------------------------------------------------------

# shutdown -r now

Now, all your adapters should be recognized by the OS and should work.

3. Buy different adapter variants to get different mac addresses

Finally, some of the Kontron cards appear to the OS as having identical mac addresses, which basically makes it impossible to use more than one at the same time. If you insert more than one, the first gets locked to an ethernet name like eth1 properly, and then the others get named things like "rename7" and "rename8" etc.
The most obvious solution was to look for adapters from completely different manufacturers, but there was only one manufacturer available in the entire country so that was not possible.
So I went to many shops, looking for slightly different versions of the adapters. The adapters have semi-transparent plastic, so it's possible to look at the circuit board and see which ones are slightly different (and are thus likely to be slightly different variants that will be likely appear with a different mac address).
Also note that a couple of variants I bought were recognized by the OS/driver, worked enough to be given an IP address etc, but did not function - no traffic would travel over them no matter what I did. Throw out and try another variant was the solution.

Prepare your WANs

Configuring in this exact way is not required, but it helps. I found that trying to configure 4 WAN connections got very confusing - knowing which USB key is which ethernet card etc etc.
So to make things clear I set things up according to the following rules:

[ol]

Physical ethernet assigned eth0. When I get the Atheros wifi card working, it will become eth1.

All USB adapters locked by mac address to eth20, eth21, eth22 etc.

If a TL-MR3020 3G AP is plugged into eth20, then it gets IP config of: 192.168.20.1, if plugged into eth21: 192.168.21.1 etc etc.

Then I put a sticker on each 3G AP with its IP address so I know which one is which.

The TL-WA801ND 300Mbps Wireless N Access Point gets given 192.168.99.1. ClearOS router gets 192.168.99.2

When adding each 3G AP as an external net, I give the the ClearOS router a static address of 192.168.x.2. If eth0, 192.168.20.2, eth21, 192.168.21.2 etc etc.

So ClearOS laptop is always accessible at .2 whatever net you are on

[/ol]

Configure Your ClearOS powered Network

[ol]

Complete the install using only a single external interface. Don't try to add all the interfaces before the install wizard has been completed. Things got really confusing and appeared to break.

Add a LAN interface on hardward card eth0 for the configured WIFI AP. The Wireless AP line will have the most traffic travelling over it, so it should get the best port.

Set manual DNS servers. Do NOT set auto update DNS servers, this causes problems with multiple WANs

After finishing the install wizard, then configure the 3 - 4 external WANs. Give each of them static addresses or Multi-WAN won't work - ports will constantly be pushed into "offline" mode and you won't be able to make them online again.

Install Web Proxy, Filter and Proxy Report, Bandwidth Viewer, Multi-WAN

Configure Multi-WAN. Set ports and ratios

Configure Web proxy. Large cache of 200gb, large object size 1gb. This will cache all debs for installing/upgrading linux saving huge amounts of traffic.

[/ol]

Notes

[ol]

If you get locked out by: "Ooops the installer hasn't completed" go to the URL /app/base/wizard/stop

[/ol]