Hello, No, I have had a set back. I live in a very rural area and I am having problems with one of my links (i have a wifi bridge and 3g) So, I have put the project on hold until it all gets sorted out. I will report when I get back up and running.

To answer your question, I have Either 1 static (wifi) and 1 DHCP (3g) or 2 dhcp or 2 static!... depending on how I finally set it up.

In reality, I only have 1 public IP that is currently static. But, things may change soon if the wisp cannot fix the problem :/

Glad to hear this is working for your network @Alex Vojacek!

Are your "Current private ips from my ISP coming from the same datacenter (same DNS for both works)" handed to you by static or DHCP? I am only having problems when I use DHCP.

@cltech - have you been successful? again if so are you implementing multi-wan with DHCP or static addresses?

I hope to hear how you've made this work as I need a solution using DHCP!

I can confirm this also works for my network.


Current private ips from my ISP coming from the same datacenter (same DNS for both works)

Previous to this fix, it wasn't working, when one uplink is down, the whole network is down, no more internet.

I took the liberty do modify failed_interval to 5 seconds and the link is down detection is a lot faster.

This is a success, previously it wasn't with my dual wan configuration

Now if only I could change the time it takes to switch link, since load balancing is not working when you download a file in multiple blocks (it is working on PFSense).

I hope you will get success when you will follow the the same steps. Waiting to hear your success :-)

I hope you will get success when you will follow the the same steps. Waiting to hear your success :-)

Tony Ellis wrote:

Happy to read of your success - hopefully anyone else who has the same WAN configuration can apply the same solution...

Thumbs up!

I just went multi-wan with private ip's (no other choice) on a full server install and I am also having these problems

I am going to implement these suggestions and see If I get the same success

Thank you very much.

I am having same issues with new 6.5. I am currently running 5.2 on XenServer with no issues using 6 visualized physical interfaces supplied to 5.2 using 4 as wans (set as DHCP from cable/SOHO router) in gateway mode with no issues. When I duplicate the setup or try to utilize vlans within 6.5 I'm faced with zero stability. Once I add the second external interface (DHCP) the interfaces are no longer stable. I have tried following the forums regarding syswatch trouble with no luck. 6.5 would be extremely valuable but without stability in the current state I'm completely out of luck.

Happy to read of your success - hopefully anyone else who has the same WAN configuration can apply the same solution...

@ tony, i have tried it, and now its working fine.

sorry for late reply, thanks for all your efforts and help.

@ Nick thank you as well.

you both guys are very good in this field.

Once again thanks all of you.

faizan - I took the time to set up MultiWAN on 6.5 (note five) with two private addresses for WANs to simulate as best I could your set-up.

Just like you found out, and a previous poster had indicated for 5.2, it doesn't work with two private addresses. I got EXACTLY the same problems as you did with the same errors in /var/log/syswatch. However, I did get it working for a short series of tests - try it and see if it works for you.

First :- change the two WAN interfaces back to static - this is important - make sure BOTH are static as this is the only testing I have done when the changes below were made.

Second :- edit /usr/sbin/syswatch - look for about line 140 - since you say 6.4 - I don't know if yours is exactly the same - I have indicted the line to be changed by using #<=== at the end. You will be adding a new line underneath this... Find the same line as I have indicated with #<== (line 5 in my little extract)

 

$config{debug} = 0;  # debug 0 = no debug, 1 - some debug

$config{retries} = 5;  # number of ping retries before taking action

$config{interval} = 60;  # ping interval (in seconds)

$config{heartbeat} = 10;  # number of intervals between writing a "heartbeat" in the log file

$config{use_get_ip} = $STATE_ENABLED;    # use remote server to determine IP #<=== 

$config{failed_interval} = 10;  # ping interval (in seconds) when 'net is down

$config{ping_servers_autodetect} = $STATE_ENABLED;

$config{try_pinging_gateway} = $STATUS_YES;

$config{is_multiwan} = $STATE_DISABLED;

Now put a # in the first position of that line and add the line as per mine underneath

$config{debug} = 0;  # debug 0 = no debug, 1 - some debug

$config{retries} = 5;  # number of ping retries before taking action

$config{interval} = 60;  # ping interval (in seconds)

$config{heartbeat} = 10;  # number of intervals between writing a "heartbeat" in the log file

#$config{use_get_ip} = $STATE_ENABLED;     # use remote server to determine IP #<===

$config{use_get_ip} = $STATE_DISABLED;     # NO remote server to determine IP   

$config{failed_interval} = 10;  # ping interval (in seconds) when 'net is down

$config{ping_servers_autodetect} = $STATE_ENABLED;

$config{try_pinging_gateway} = $STATUS_YES;

$config{is_multiwan} = $STATE_DISABLED;

Having made these changes restart the network and syswatch(service network restart && service syswatch restart) and test...
It all worked correctly for me - hopefully it will for you as well. Note that my testing was not extensive - so no warranty provided :-)

One last point - why have you not configured weights to your Multi-WAN interfaces.

Good Luck

Oh! - one last thing (two actually). If you are not 100% comfortable editing that file make a copy first - then you can always restore to the original state.

Note also that future ClearOS updates might wipe this change - in which case you will need to keep a copy of these instructions (if they worked) and apply again.

@tony and nick

Well now i am using ClearOS 6.4,

Please check the output of following commands attached here,

ifconfig
cat /etc/resolv.conf
cat /etc/clearos/mutiwan.conf
cat /etc/sysconfig/network-scripts/ifcfg-eth*
lspci -k | grep Eth -A4

and specially i am showing the result of tail -f /var/log/syswatch

Now please let me know where i am doing some thing wrong ....? and how i can resolve it ?? [file name=ClearOS_6.txt size=8789]http://www.clearfoundation.com/media/kunena/attachments/legacy/files/ClearOS_6.txt[/file]

Yeah, since the thread was about ClearOS 6 Multi-WAN, I too had assumed he was running 6.x. Annoying as it appears we have been wasting out time to some extent :-( My test was performed using 6.5 on purpose.

Nick re. /etc/resolv-peerdns.conf - On my 6.5 system I have that file, but it I can see that it is definitely being ignored and the contents of /etc/resolv.conf is what is controlling my dns lookups... but that is OT so I'll not pursue that further here...

faizan - the /var/log/syswatch extract doesn't show a problem - I can only assume you have shown us an extract when both interfaces are on-line - which if that is the case, is not helpful - we need to see what is recorded when one goes down and then should have come on-line as per my example posted above. If this is what you see when one interface is off-line - then you have a strange problem - we should be seeing pings to check if the interface that went off-line has come up again...

@faizan,
Have you forgotten to tell us you are running 5.x and not 6.x as the thread title? If so look at Dave Burkholder's comment on p.3

Hi Tony and Nick,

Please check the output of following commands attached here,

ifconfig
cat /etc/resolv.conf
cat /etc/clearos/mutiwan.conf :. No such file or directory exist
cat /etc/sysconfig/network-scripts/ifcfg-eth*
lspci -k | grep Eth -A4 :. -k ( nothing)


Waiting for your response, how issue will resolve ?

thanks

@faizan, link now fixed. From memory I use server.howitts.lan and have called my domain howitts.lan but I can't check until I'm at home.

Nick i have used the public DNS server as well ( 8.8.8.8 and 8.8.4.4) but in that case the issue is same, and one more thing in that case i can't be able to ping any external server such as yahoo, hotmail etc ( I can't check internet connectivity as well in that scenario )

And the link which you have given;

http://www.clearcenter.com/support/documentation/user_guide/ip_settings]this%20document[/code]

not working showing Error "the topic does not exist yet"

what should be the host name ...? the hostname should be same which clearos system have??, which image i have attached in previous post..... and in this also?




And for Both DHCP , hostname should be same...?

Just a quickie, /etc/resolv.conf is now more more or less obsolete and replaced by /etc/resolv-peerdns so I suggest that is posted as well.

Have a look at /var/log/syswatch and see if you can spot a problem when an interface should be coming back on-line. You can watch it dynamically by using "tail -f /var/log/syswatch" if you want, or just view the log file after the event. You should see ping checks on each interface as the connection is checked. The next pings to an interface after the cable is re-connected should pass... (look at the output I provided of Multi-WAN working and see where yours differs).

As has already been suggested - do not use your ISP DNS servers - use public DNS servers that are available from either WAN interface. eg 8.8.8.8 and 8.8.4.4

Then provide the following so we can look to see if anything appears not quite right..

Outputs from :-

ifconfig
cat /etc/resolv.conf
cat /etc/clearos/mutiwan.conf
cat /etc/sysconfig/network-scripts/ifcfg-eth*
lspci -k | grep Eth -A4

As I posted earlier, I really suggest you ditch your ISP's DNS servers and use a public service like OpenDNS (208.67.222.222 and 208.67.220.220) or GoogleDNS (8.8.8.8 and 8.8.4.4). There are others.

Also why is your host name an IP address? Have a look at this document

attaching the DHCP hostname setting in image as well.




Please suggest what to do...

Hi Tony,

I hope you would be fine,

I have been assigned IP's to Both WAN by their DHCP (WifI Router),

And in Hostname of Both Wan i have given their gateway's IP's 192.168.9.1 and 192.168.2.1 ( Imp thing: I haven't check mark on Automatic DNS Servers their)

I am using Primary DNS IP of Both ISP's in DNS Server # 1 and Server #2




Now my one wan link is going to restore , but other link is not restoring, i have tried a lot. I have changed the setting of eth0 by eth2 as well. But that particular one wan is not restoring.

Sorry about the length of this. As you can see below - it passed the test. Setup :-
eth0 - WAN ADSL via typical 4 port modem (using NAT)
eth1 - LAN
eth2 - WAN cable modem
totally different ISP providers - ClearOS 6.5 completely up to date
pulled ethernet cable for each WAN in turn - significant evens have had ################# added
(you will need to pull the bottom scroll bar to the right to see them all because of line length))

Sun Jan 26 10:54:09 2014  info:  system - current WANs in use - eth0 eth2         #################

Sun Jan 26 10:55:58 2014  info:    eth0 - ping check on server #1 failed - 69.90.141.72

Sun Jan 26 10:56:03 2014  info:    eth0 - ping check on server #2 failed - 67.18.166.2 

Sun Jan 26 10:56:03 2014  warn:    eth0 - connection warning

Sun Jan 26 10:56:16 2014  info:    eth0 - ping check on gateway failed - 192.168.4.1

Sun Jan 26 10:56:18 2014  info:    eth0 - ping check on server #1 failed - 69.90.141.72

Sun Jan 26 10:56:23 2014  info:    eth0 - ping check on server #2 failed - 67.18.166.2 

Sun Jan 26 10:56:23 2014  warn:    eth0 - connection warning

Sun Jan 26 10:56:35 2014  info:    eth0 - ping check on gateway failed - 192.168.4.1

Sun Jan 26 10:56:37 2014  info:    eth0 - ping check on server #1 failed - 69.90.141.72

Sun Jan 26 10:56:42 2014  info:    eth0 - ping check on server #2 failed - 67.18.166.2 

Sun Jan 26 10:56:42 2014  warn:    eth0 - connection is down         #################

Sun Jan 26 10:56:44 2014  info:    eth0 - waiting for static IP reconnect

Sun Jan 26 10:56:44 2014  info:  system - changing active WAN list - eth2 (was eth0 eth2)

Sun Jan 26 10:56:44 2014  info:  system - current WANs in use - eth2         #################

Sun Jan 26 10:56:44 2014  info:  system - restarting firewall

Sun Jan 26 10:56:44 2014  info:  system - updating intrusion prevention whitelist

Sun Jan 26 10:56:44 2014  info:  system - adding ping server 69.90.141.72

Sun Jan 26 10:56:44 2014  info:  system - adding ping server 67.18.166.2 

Sun Jan 26 10:56:44 2014  info:  system - adding DNS server 192.168.4.1  

Sun Jan 26 10:56:57 2014  info:    eth0 - ping check on gateway failed - 192.168.4.1

Sun Jan 26 10:56:59 2014  info:    eth0 - ping check on server #1 failed - 69.90.141.72

Sun Jan 26 10:57:04 2014  info:    eth0 - ping check on server #2 failed - 67.18.166.2 

Sun Jan 26 10:57:04 2014  warn:    eth0 - connection is down         #################

Sun Jan 26 10:57:06 2014  info:    eth0 - waiting for static IP reconnect

Sun Jan 26 10:57:18 2014  info:    eth0 - ping check on gateway failed - 192.168.4.1

Sun Jan 26 10:57:20 2014  info:    eth0 - ping check on server #1 failed - 69.90.141.72

Sun Jan 26 10:57:25 2014  info:    eth0 - ping check on server #2 failed - 67.18.166.2

Sun Jan 26 10:57:25 2014  warn:    eth0 - connection is down

Sun Jan 26 10:57:27 2014  info:    eth0 - waiting for static IP reconnect

Sun Jan 26 10:57:39 2014  info:    eth0 - ping check on gateway failed - 192.168.4.1   

Sun Jan 26 10:57:41 2014  info:    eth0 - ping check on server #1 failed - 69.90.141.72

Sun Jan 26 10:57:45 2014  info:    eth0 - ping check on server #2 passed         #################

Sun Jan 26 10:57:45 2014  info:  system - changing active WAN list - eth0 eth2 (was eth2)

Sun Jan 26 10:57:45 2014  info:  system - current WANs in use - eth0 eth2         #################

Sun Jan 26 10:57:45 2014  info:  system - restarting firewall

Sun Jan 26 10:57:45 2014  info:  system - updating intrusion prevention whitelist

Sun Jan 26 10:57:45 2014  info:  system - adding ping server 69.90.141.72

Sun Jan 26 10:57:45 2014  info:  system - adding ping server 67.18.166.2 

Sun Jan 26 10:57:45 2014  info:  system - adding DNS server 192.168.4.1

Sun Jan 26 10:58:06 2014  info:    eth0 - ping check on server #1 passed - 69.90.141.72

Sun Jan 26 10:58:26 2014  info:  system - heartbeat...

Sun Jan 26 11:01:33 2014  info:    eth2 - ping check on server #1 failed - 69.90.141.72

Sun Jan 26 11:01:38 2014  info:    eth2 - ping check on server #2 failed - 67.18.166.2

Sun Jan 26 11:01:38 2014  warn:    eth2 - connection warning

Sun Jan 26 11:01:50 2014  info:    eth2 - ping check on gateway failed - 144.136.112.1 

Sun Jan 26 11:01:52 2014  info:    eth2 - ping check on server #1 failed - 69.90.141.72

Sun Jan 26 11:01:57 2014  info:    eth2 - ping check on server #2 failed - 67.18.166.2

Sun Jan 26 11:01:57 2014  warn:    eth2 - connection warning         #################

Sun Jan 26 11:02:10 2014  info:    eth2 - ping check on gateway failed - 144.136.112.1 

Sun Jan 26 11:02:12 2014  info:    eth2 - ping check on server #1 failed - 69.90.141.72

Sun Jan 26 11:02:17 2014  info:    eth2 - ping check on server #2 failed - 67.18.166.2

Sun Jan 26 11:02:17 2014  warn:    eth2 - connection is down         #################

Sun Jan 26 11:02:19 2014  info:    eth2 - restarting DHCP connection

Sun Jan 26 11:02:26 2014  info:    eth2 - setting IP address cache to unknown

Sun Jan 26 11:02:26 2014  info:    eth2 - ping check - no IP available

Sun Jan 26 11:02:26 2014  info:  system - changing active WAN list - eth0 (was eth0 eth2)

Sun Jan 26 11:02:26 2014  info:  system - current WANs in use - eth0         #################

Sun Jan 26 11:02:26 2014  info:  system - default route vanished - using eth0 

Sun Jan 26 11:02:26 2014  info:  system - setting default route to 192.168.4.1

Sun Jan 26 11:02:26 2014  info:  system - restarting firewall

Sun Jan 26 11:02:27 2014  info:  system - restarting firewall

Sun Jan 26 11:02:27 2014  info:  system - updating intrusion prevention whitelist

Sun Jan 26 11:02:27 2014  info:  system - adding ping server 69.90.141.72

Sun Jan 26 11:02:27 2014  info:  system - adding ping server 67.18.166.2 

Sun Jan 26 11:02:27 2014  info:  system - adding DNS server 192.168.4.1

Sun Jan 26 11:02:27 2014  info:  system - reloading intrusion prevention system

Sun Jan 26 11:02:40 2014  info:    eth2 - ping check - no IP available

Sun Jan 26 11:02:41 2014  info:    eth2 - restarting DHCP connection  

Sun Jan 26 11:02:57 2014  info:    eth2 - ping check - no IP available

Sun Jan 26 11:02:57 2014  info:    eth2 - restarting DHCP connection  

Sun Jan 26 11:03:13 2014  info:    eth2 - ping check - no IP available

Sun Jan 26 11:03:13 2014  info:    eth2 - restarting DHCP connection

Sun Jan 26 11:03:29 2014  info:  system - heartbeat...

Sun Jan 26 11:03:29 2014  info:    eth2 - ping check - no IP available

Sun Jan 26 11:03:29 2014  info:    eth2 - restarting DHCP connection

Sun Jan 26 11:03:36 2014  info:    eth2 - new IP address detected - 144.136.112.120         #################

Sun Jan 26 11:03:36 2014  info:    eth2 - ping check on server #1 passed - 69.90.141.72

Sun Jan 26 11:03:36 2014  info:  system - changing active WAN list - eth0 eth2 (was eth0)

Sun Jan 26 11:03:36 2014  info:  system - current WANs in use - eth0 eth2         #################

faizan - my DNS setup is not ClearOS 'standard'. As dnsmasq lacks the features I require, I run bind for DNS and the isc dhcp server. Thus my /etc/resolv.conf is set to 127.0.0.1 for the first DNS server and my live backup server for the second. However, the first two DNS in my /etc/named.conf 'forwarders' section are public (8.8.8.8 and 8.8.4.4) effectively doing what Nick indicates above.

Dave. I haven't had problems on servers using private addresses for external interfaces with Mulit-WAN - but again my experience is limited to one site only and my success may not be typical.

faizan - I will do a test similar to what you describe on my 6.X system and report the results..

Just curious, about something here. As I recall, in the COS 5.2 days, supposedly MultiWAN didn't work when both WANs were local IPs. Is that no longer the case, or is my memory faulty?

You can assign DNS servers manually and if you use different ISP's for each WAN generally you need to use public DNS servers such as OpenDNS or GoogleDNS.

Hi Tony,

I will try it in next working day, and then i will let you know the situation,

But i remember something that if i will select WAN links as DHCP, so DNS will automatically take primary and secondary DNS IP's of one WAN link only....?

Have you given Static DNS IP of Both Wan Links ??

anyway i will try ....let see what happened !

thanks Tony

faizan - are you able to configure one or both of your WAN interfaces as DHCP rather than static just for a test?

In my limited experience, Multi-WAN works better with a least one of my two interfaces configured as DHCP - but that could just be something unique to my systems...

Hai All,

Hope you would be fine,

I have the following scenario, and i am facing the following issue in ClearOS 5.2, please suggest

I am using two Wan having two different isp connections, load balancing and Auto failover configure;

eth0 192.168.9.X ( Wan1 )
255.255.255.0
192.168.9.1

eth2 192.168.2.X ( Wan2 )
255.255.255.0
192.168.2.1

eth1 192.168.1.X (Lan)

System providing load balancing according to weight defined, alternatively traffic of users going from both Wan1 & Wan2.

And the Status of Multi-wan interface is like that .Both Wan are connected and in use.

When Wan1 link fails or unplug ( Wan2 provide failover and all remaining traffic divert toward is well) & its show "Wan1 Connection offline" in Multi-wan interface and Wan2 connection "in use".

But when Wan1 link restore or plug after some time. ( Its doesn't up the Wan1 link in Multi-wan interface) until the system has restarted.

Similary,
When Wan2 link fails or unplug (Wan1 provide failover and all remaining traffic divert toward is well) & its show "Wan2 Connection offline" in Multi-wan interface and Wan1 connection "in use".

But when Wan2 link restore or plug after some time. ( Its doesn't up the Wan2 link in Multi-wan interface ) until the system has restarted.

AND...
.
.
.

I am unplugging the cable which are coming from link sys router to ClearOS (But its not coming directly to ClearOS there is some IO ports which are bridging wire from Router to ClearOS, And i am unplugging the last end of cable which is directly going in Clear OS ) . And I think modem/ router are not adsl. (Because both Wan are dedicated CIR link, and after NAT i am using it) .

I am using Static IP's on Both Wan which i have got After NAT.
192.168.2.x and using DNS of ISP
192.168.9.x and using DNS of ISP
and i have selected Type Static ,

Please help me to resolve this issue and let know what is the reason behind it ?? tell me how much time clearos takes to up the link after its restore..?

Waiting for prompt response thanks in advance

DaveBurkholder wrote:

This module has improved the situation signficantly. Now when a WAN fails, internet connectivity is restored after a minute or so. Formerly we had to reboot. So it's getting better. Hopefully 6.5 eliminates even the momentary connection drop.

That's just a matter of tuning the failover configuration in /etc/syswatch to be more aggressive. By default, the multiWAN system:

- Marks a connection down when a ping fails 3 times in a row ("retries" parameter in /etc/syswatch)
- Performs a ping check every 20 seconds ("interval" parameter) when the network is happy
- Performs a ping check every 10 seconds ("failed interval") when ping checks start to fail

So it will take about a minute for the WAN connection to be marked as "down" by default. Feel free to change this behavior by editing the /etc/syswatch configuration and restarting syswatch ("/sbin/service syswatch restart").

It's common to have occasional ISP hiccups for 10 to 30 seconds and connections will recover without interruption. Keeping connections up for things like RDP, VPNs, SSH, etc. is a bonus. If you are too aggressive with marking WAN connections offline, you may end up interrupting sessions that would have normally been automatically restored.

Peter Baldwin wrote:

In a couple of hours, the "app-firewall-core-1.5.6-1" version will appear in "clearos-test" and it includes the fix to skip over any network interfaces without an IP configured. To install:

yum --enablerepo=clearos-test upgrade app-firewall-core

This module has improved the situation signficantly. Now when a WAN fails, internet connectivity is restored after a minute or so. Formerly we had to reboot. So it's getting better. Hopefully 6.5 eliminates even the momentary connection drop.

I apply upgrade firewall...and I sent logs.
When I disconnected PPoE connection, internet connection is on-off, on-off ...
I connected PPoE and I disconnected WAN DHCP, and internet connection is ok, just as the network status in the gateway status dashboard shows offline
Multiwan is not installed.

2xWAN
1xPPPoE
1xDHCP

DaveBurkholder wrote:

I have a machine with a PPPoE WAN and a static IP WAN and all you need to do to replicate the issue is unplug the PPPoE line. Would that be of any assistance?

That would be great! A test before and after applying the update would be especially useful.

I have a machine with a PPPoE WAN and a static IP WAN and all you need to do to replicate the issue is unplug the PPPoE line. Would that be of any assistance?

Many thanks for the logs. I really want to put this one to bed -- it's beyond frustrating and annoying :angry:

We can kind of duplicate the issue, but it requires manually removing IP settings from a static network interface. IPs should *never* disappear from a static IP, but that's exactly what we're seeing in the logs that were sent. In a couple of hours, the "app-firewall-core-1.5.6-1" version will appear in "clearos-test" and it includes the fix to skip over any network interfaces without an IP configured. To install:

yum --enablerepo=clearos-test upgrade app-firewall-core

I can confirm the problem on 6.5 beta.
I'm sorry to say this , but installed pfsense everything worked fine...sorry because this project is very good...

Logs likewise sent. This is 100% replicable behavior - 1 WAN fails and DNS just falls on its nose. Tried 6.5 beta before reading this thread and can confirm it's still broken in 6.5.

No news yet... sorry.

No news? I can't believe this bug has no solution ?

Is there any bug fix for this in 6.5.0 beta1 ?

I can confirm the issue with only 2 WAN too .

My configuration is :

Globnal DNS configured ( OPENDNS )

1 LAN (DHCP SERVER)
1 WAN PPPoE (ADSL MODEM in bridge mode)
1 WAN DHCP (ADSL MODEM in bridge mode)

I've made 3 installation in two different hardware configurations with the same network topology and providers with the same result:

When one Internet connection goes offline , LAN cannot connect to internet anymore.

I tried also with a fresh install in gateway mode and the multiwan module ( I usually install bandwidth manager and web proxy and other "viewer" staff ) with the same results.
Clearos PC seems to ping and trace outside when one line is offline, but no clients can.

I will send logs asap to developer.

Simon

Duly sent!

I can recreate the issue by simply pulling the cable from one of my External ports, causes one additional other port to go offline, leaving 1 out of 3 online, but no internet access.

TIA
Andy