Can you post last 50 lines or so of /var/log/syswatch.

Thx

Please see below.



ue Apr 3 13:01:57 2018 info: enp3s0f0 - ping check on server #1 failed - 8.8.8.8
Tue Apr 3 13:02:02 2018 info: enp3s0f0 - ping check on server #2 passed - 54.152.208.245
Tue Apr 3 13:02:22 2018 info: system - heartbeat...
Tue Apr 3 13:02:22 2018 info: enp3s0f0 - ping check on server #1 passed - 8.8.8.8
Tue Apr 3 13:03:25 2018 debug: enp3s0f0 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 13:03:27 2018 info: enp3s0f0 - ping check on server #1 failed - 8.8.8.8
Tue Apr 3 13:03:34 2018 info: enp3s0f0 - ping check on server #2 failed - 54.152.208.245
Tue Apr 3 13:03:34 2018 warn: enp3s0f0 - connection warning
Tue Apr 3 13:03:44 2018 info: enp3s0f0 - ping check on server #1 passed - 8.8.8.8
Tue Apr 3 13:04:06 2018 debug: enp3s0f0 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 13:04:08 2018 info: enp3s0f0 - ping check on server #1 failed - 8.8.8.8
Tue Apr 3 13:04:15 2018 info: enp3s0f0 - ping check on server #2 failed - 54.152.208.245
Tue Apr 3 13:04:15 2018 warn: enp3s0f0 - connection warning
Tue Apr 3 13:04:29 2018 info: enp3s0f0 - ping check on gateway failed - 192.168.0.1
Tue Apr 3 13:04:31 2018 debug: enp3s0f0 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 13:04:33 2018 info: enp3s0f0 - ping check on server #1 failed - 8.8.8.8
Tue Apr 3 13:04:40 2018 info: enp3s0f0 - ping check on server #2 failed - 54.152.208.245
Tue Apr 3 13:04:40 2018 warn: enp3s0f0 - connection warning
Tue Apr 3 13:04:54 2018 info: enp3s0f0 - ping check on gateway failed - 192.168.0.1
Tue Apr 3 13:04:56 2018 debug: enp3s0f0 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 13:04:56 2018 info: enp3s0f0 - ping check on server #1 passed - 8.8.8.8
Tue Apr 3 13:06:19 2018 debug: enp3s0f0 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 13:06:21 2018 info: enp3s0f0 - ping check on server #1 failed - 8.8.8.8
Tue Apr 3 13:06:28 2018 info: enp3s0f0 - ping check on server #2 failed - 54.152.208.245
Tue Apr 3 13:06:28 2018 warn: enp3s0f0 - connection warning
Tue Apr 3 13:06:42 2018 info: enp3s0f0 - ping check on gateway failed - 192.168.0.1
Tue Apr 3 13:06:42 2018 info: enp3s0f0 - ping check on server #1 passed - 8.8.8.8
Tue Apr 3 13:07:42 2018 info: system - heartbeat...
Tue Apr 3 13:08:04 2018 debug: enp3s0f0 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 13:08:06 2018 info: enp3s0f0 - ping check on server #1 failed - 8.8.8.8
Tue Apr 3 13:08:13 2018 info: enp3s0f0 - ping check on server #2 failed - 54.152.208.245
Tue Apr 3 13:08:13 2018 warn: enp3s0f0 - connection warning
Tue Apr 3 13:08:15 2018 debug: enp3s0f1 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 13:08:29 2018 info: enp3s0f0 - ping check on gateway failed - 192.168.0.1
Tue Apr 3 13:08:31 2018 debug: enp3s0f0 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 13:08:33 2018 info: enp3s0f0 - ping check on server #1 failed - 8.8.8.8

Can you give the output to:

lspci -k | grep Eth -A 3

ls /etc/sysconfig/network-scripts/ifcfg-e*

Please see below:


lspci -k | grep Eth -A 3
02:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ether net (rev 20)
Subsystem: Hewlett-Packard Company NC382i Integrated Multi-port PCI Expr ess Gigabit Server Adapter
Kernel driver in use: bnx2
Kernel modules: bnx2
02:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ether net (rev 20)
Subsystem: Hewlett-Packard Company NC382i Integrated Multi-port PCI Expr ess Gigabit Server Adapter
Kernel driver in use: bnx2
Kernel modules: bnx2
03:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ether net (rev 20)
Subsystem: Hewlett-Packard Company NC382i Integrated Multi-port PCI Expr ess Gigabit Server Adapter
Kernel driver in use: bnx2
Kernel modules: bnx2
03:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ether net (rev 20)
Subsystem: Hewlett-Packard Company NC382i Integrated Multi-port PCI Expr ess Gigabit Server Adapter
Kernel driver in use: bnx2
Kernel modules: bnx2
[root@gateway ~]# ls /etc/sysconfig/network-scripts/ifcfg-e*
/etc/sysconfig/network-scripts/ifcfg-enp2s0f0
/etc/sysconfig/network-scripts/ifcfg-enp2s0f1
/etc/sysconfig/network-scripts/ifcfg-enp3s0f0
/etc/sysconfig/network-scripts/ifcfg-enp3s0f1

Drivers are OK. What is in /etc/sysconfig/network-scripts/ifcfg-enp2s0f1? Can you hide it by putting a "." in front of the file name then restart networking?

I did that but still the same.


Tue Apr 3 18:30:20 2018 info: enp3s0f1 - ping check on server #2 failed - 54.$
Tue Apr 3 18:30:20 2018 warn: enp3s0f1 - connection warning
Tue Apr 3 18:30:34 2018 info: enp3s0f1 - ping check on gateway failed - 192.1$
Tue Apr 3 18:30:34 2018 info: enp3s0f1 - ping check on server #1 passed - 8.8$
Tue Apr 3 18:32:55 2018 info: system - heartbeat...
Tue Apr 3 18:34:17 2018 debug: enp3s0f1 - ping check on server #1 failed - 8.8$
Tue Apr 3 18:37:58 2018 info: system - heartbeat...

Tue Apr 3 18:07:41 2018 info: system - restarting firewall
Tue Apr 3 18:12:43 2018 info: system - heartbeat...
Tue Apr 3 18:17:44 2018 info: system - heartbeat...
Tue Apr 3 18:20:27 2018 debug: enp3s0f1 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 18:20:29 2018 info: enp3s0f1 - ping check on server #1 failed - 8.8.8.8
Tue Apr 3 18:20:36 2018 info: enp3s0f1 - ping check on server #2 failed - 54.152.208.245
Tue Apr 3 18:20:36 2018 warn: enp3s0f1 - connection warning
Tue Apr 3 18:20:46 2018 info: enp3s0f1 - ping check on server #1 passed - 8.8.8.8
Tue Apr 3 18:22:47 2018 info: system - heartbeat...
Tue Apr 3 18:27:48 2018 info: system - heartbeat...
Tue Apr 3 18:30:11 2018 debug: enp3s0f1 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 18:30:13 2018 info: enp3s0f1 - ping check on server #1 failed - 8.8.8.8
Tue Apr 3 18:30:20 2018 info: enp3s0f1 - ping check on server #2 failed - 54.152.208.245
Tue Apr 3 18:30:20 2018 warn: enp3s0f1 - connection warning
Tue Apr 3 18:30:34 2018 info: enp3s0f1 - ping check on gateway failed - 192.168.2.1
Tue Apr 3 18:30:34 2018 info: enp3s0f1 - ping check on server #1 passed - 8.8.8.8
Tue Apr 3 18:32:55 2018 info: system - heartbeat...
Tue Apr 3 18:34:17 2018 debug: enp3s0f1 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 18:37:58 2018 info: system - heartbeat...
Tue Apr 3 18:42:59 2018 info: system - heartbeat...
Tue Apr 3 18:48:00 2018 info: system - heartbeat...
Tue Apr 3 18:48:43 2018 debug: enp3s0f1 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 18:50:26 2018 debug: enp3s0f1 - ping check on server #1 failed - 8.8.8.8 (ping size: 1)
Tue Apr 3 18:50:28 2018 info: enp3s0f1 - ping check on server #1 failed - 8.8.8.8
Tue Apr 3 18:50:35 2018 info: enp3s0f1 - ping check on server #2 failed - 54.152.208.245
Tue Apr 3 18:50:35 2018 warn: enp3s0f1 - connection warning
Tue Apr 3 18:50:47 2018 info: enp3s0f1 - ping check on server #1 passed - 8.8.8.8

Did you restart networking? I guess you did.

What happens if you run a continuous ping on 192.168.0.1?

ping 192.168.0.1 -I enp3s0f0

Also are you seeing errors on enp3s0f0?

ifconfig enp3s0f0

Yes, I actually rebooted the server. Ping results are attached.



[root@gateway ~]# ifconfig enp3s0f0
enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.254 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::1aa9:5ff:fe65:6468 prefixlen 64 scopeid 0x20<link>
ether 18:a9:05:65:64:68 txqueuelen 1000 (Ethernet)
RX packets 5453685 bytes 6058817169 (5.6 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 17375009 bytes 12664791805 (11.7 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Yes, I actually rebooted the server. Ping results are attached.



[root@gateway ~]# ifconfig enp3s0f0
enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.254 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::1aa9:5ff:fe65:6468 prefixlen 64 scopeid 0x20<link>
ether 18:a9:05:65:64:68 txqueuelen 1000 (Ethernet)
RX packets 5453685 bytes 6058817169 (5.6 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 17375009 bytes 12664791805 (11.7 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0


I wanted to attach the text file with ping results but it's not as a supported format for uploads, but here is the result. While the ping was running, the internet connection on my pc broke twice.
statistics ---
152 packets transmitted, 152 received, 0% packet loss, time 150999ms
rtt min/avg/max/mdev = 0.178/0.225/0.464/0.039 ms

This is odd. Your pings have not attached but the result is OK.

What is the output to:

ifconfig | grep ^e -A 1

What are your external connections? Are you able to put the device connected to enp3s0f0 into bridge/passthrough mode so that enp3s0f0 gets a public IP? If it is an ADSL connection, you'll need to switch enp3s0f0 from DHCP to PPPoE.

Please see below for the output. enp3s0f0 is a wimax connection. I live in a remote location so wimax is the only solution. The ISP does not allow any changes to the device, so that's not possible here. Anything else I can try? I know the connections are good because, when I bypass clearos, they both work fine individually. If you want, I can give you remote access to the server.



enp2s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.1 netmask 255.255.255.0 broadcast 192.168.1.255
--
enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.254 netmask 255.255.255.0 broadcast 192.168.0.255
--
enp3s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.2 netmask 255.255.255.0 broadcast 192.168.2.255

A couple of thoughts. What are your ping times to 8.8.8.8.

ping 8.8.8.8 -I enp3s0f0

I wonder if syswatch is is not waiting long enough before it times out. (If you can read code, have a look in /usr/sbin/syswatch) Is your other connection WiMax as well?

The other thing to do is just stop the syswatch daemon for the moment with a:

systemctl stop syswatch.service

systemctl disable syswatch.service

Below are the ping results. I am sorry, I tried going through syswatch but cant make much sense out of it. I have turned it off and disabled it, but symptoms are still the same.


[root@gateway ~]# ping 8.8.8.8 -I enp3s0f0
PING 8.8.8.8 (8.8.8.8) from 192.168.0.254 enp3s0f0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=19.4 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=16.6 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=20.5 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=54 time=19.6 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=54 time=18.9 ms
64 bytes from 8.8.8.8: icmp_seq=6 ttl=54 time=18.6 ms
64 bytes from 8.8.8.8: icmp_seq=7 ttl=54 time=18.4 ms
64 bytes from 8.8.8.8: icmp_seq=8 ttl=54 time=18.9 ms
64 bytes from 8.8.8.8: icmp_seq=9 ttl=54 time=18.1 ms
64 bytes from 8.8.8.8: icmp_seq=10 ttl=54 time=23.1 ms
64 bytes from 8.8.8.8: icmp_seq=11 ttl=54 time=24.5 ms
64 bytes from 8.8.8.8: icmp_seq=12 ttl=54 time=17.6 ms
64 bytes from 8.8.8.8: icmp_seq=13 ttl=54 time=21.4 ms
64 bytes from 8.8.8.8: icmp_seq=14 ttl=54 time=18.0 ms
64 bytes from 8.8.8.8: icmp_seq=15 ttl=54 time=17.7 ms
64 bytes from 8.8.8.8: icmp_seq=16 ttl=54 time=19.9 ms
64 bytes from 8.8.8.8: icmp_seq=17 ttl=54 time=24.4 ms
64 bytes from 8.8.8.8: icmp_seq=18 ttl=54 time=16.5 ms
64 bytes from 8.8.8.8: icmp_seq=19 ttl=54 time=18.4 ms

Can you check syswatch is stopped because it can restart itself (or at least service watch can):

systemctl status syswatch.service

It is syswatch which restarts the interfaces so I am surprised you still have the same issue. I think syswatch logging should also have stopped.

You can check the interfaces are up:

ifup enp3s0f0

ifup enp3s0f1

service firewall restart

I don't know what to do after that and I'm out for the rest of the day. I also don't have MultiWAN so I can't see what it is doing.

I know its not multiwan because I uninstalled it and the symptems were still there.


Apr 03 18:07:38 gateway.vmakol.lan systemd[1]: Starting System Watch Daemon...
Apr 03 18:07:38 gateway.vmakol.lan systemd[1]: Started System Watch Daemon.
Apr 04 05:08:05 gateway.vmakol.lan syswatch[1323]: RTNETLINK answers: File exists
Apr 04 11:04:39 gateway.vmakol.lan systemd[1]: Stopping System Watch Daemon...
Apr 04 11:04:39 gateway.vmakol.lan systemd[1]: Stopped System Watch Daemon.


ifup enp3s0f0

Determining IP information for enp3s0f0...dhclient(1161) is already running - exiting.

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report for this software via the CentOS Bugs Database:
http://bugs.centos.org/

exiting.
failed.


ifup enp3s0f1

returned nothing.

Another observation: I ran a ping test from multiple(6) machines concurrently, and the result is not consistent. Connection break is not consistent, it's purely random. Any other ideas?

I'm afraid I am completely at a loss as to why this is happening. I asked Ben last week and he had no idea either.

Are you now running with syswatch enabled or disabled?

Have you tried changing your load-balancing rules, say to 1:1? I don't see it should matter.

I have just had a look at the Webconfig and the options don't match the documentation I don't understand the difference in the roles between Backup and Standby.

Been following this with interest as there are 5 machines with Multi-Wan - sorry nothing to add about the OPs problems at the moment. Never seen a problem quite like this on any of the systems here. Three machines act as firewalls, Multi-Wan to 2 different ISPs (ADSL2+ and Cable) - the other two machines connecting the office to where the firewalls are placed - one interface on each via a WiFi connection - the interface uses Ethernet-over-Power. All Multi-Wan interface are private sub_nets (192.168.x.x) - work equally well whether static or dynamic. All Multi-Wan interfaces are configured as primary.

However, this is my understanding regarding the modes and based on observation of the machines here - happy to be corrected...

Both primary - both interfaces in use. Traffic passed through each interface roughly dependant on weights. If one fails all traffic is directed to the working interface. Traffic is again shared when the failing interface recovers.

[root@karien ~]# ip rule

0:	from all lookup local 

50:	from all lookup 50 

100:	from all fwmark 0x8000 lookup 100 

101:	from all fwmark 0x8001 lookup 101 

200:	from 192.168.6.98/24 lookup 200 

201:	from 192.168.2.98/24 lookup 201 

250:	from all lookup 250 

32766:	from all lookup main 

32767:	from all lookup default 

[root@karien ~]# ip route show table 250

default proto static 

	nexthop via 192.168.6.253 dev enp3s0 weight 100 

	nexthop via 192.168.2.35 dev enp4s6f0 weight 100 

[root@karien ~]# 

One interface primary, one backup. All traffic goes through the primary - the backup is up (hot) and ready to take-over immediately if the primary fails.
If the backup has taken over as a result of the primary failing, when the primary comes back on-line traffic switches back to the primary.

[root@karien ~]# ip rule

0:	from all lookup local 

50:	from all lookup 50 

100:	from all fwmark 0x8000 lookup 100 

101:	from all fwmark 0x8001 lookup 101 

200:	from 192.168.6.98/24 lookup 200 

201:	from 192.168.2.98/24 lookup 201 

250:	from all lookup 250 

32766:	from all lookup main 

32767:	from all lookup default 

[root@karien ~]# ip route show table 250

default via 192.168.2.35 dev enp4s6f0 

[root@karien ~]# 

One interface primary, one standby - standby interface is 'dead' while primary active and carries all traffic - note no table 250.
In my tests if the primary dies - no traffic - requires manual re-configuration of Multi-Wan to restore traffic by changing 'standby' to primary or backup.

[root@karien ~]# ip rule

0:	from all lookup local 

200:	from 192.168.6.98/24 lookup 200 

201:	from 192.168.2.98/24 lookup 201 

32766:	from all lookup main 

32767:	from all lookup default 

[root@karien ~]# 

Question for the OP...

You have stated one modem/router is connected to WiMax - is that the same for the other WAN connection?
Are both going to the same ISP?
If so, can you check the ISP supplied addresses on the WAN connection for each modem/router. Are they all within the same ISP's sub-net?

They are both WIMAX and from different ISPs. One is static and the other one is dynamic. I spoke with both of the ISPs and was able to convince them to share the settings to eliminate the modem/router from the middle, but still no luck.
I have a deadline to make this work so I am going to start looking at other options.


Thanks everyone for your help... If I find a solution to this, I will definitely post.