Forums

Resolved
0 votes
This is a bit complex (uh, sorry) but I've tried to distill it all down to the relevent stuff. Some IP addresses are obfuscated and lots of stuff omitted.

- WAN on port enp5s0, local 192.168.3.5, gateway 192.168.3.1, Wireless
- WAN on port enp6s0, local 192.168.2.5, gateway 192.168.2.1, Wireless
- WAN on dev ppp0 (ADSL) local 444.64.31.32, gateway 303.29.31.1 (I think)
- LAN on port enp4s0, local 192.168.0.5, with VLANs .10 and .20.

- ClearOS 7, last updated late 2016. OpenVPN 2.3.12 x86_64-redhat-linux-gnu
- Busy gateway server with three WANs, two VLANs, businesses, and paying guests using internet at all hours, plus VoIP.
- Default traffic is shared across the three WANs using COS Multi-WAN which works well.
- Physically a long drive (3hrs) to get to server, I administer it from afar via VPN, SSH, etc.
- ADSL modem connects to ISP who are also the VoIP exchange. Modem is on dev ppp0.
- VoIP only works through ADSL/ppp0, other WANs unavoidably have multiple NATs, which kills VoIP dead.
- The ADSL on ppp0 goes down more than occasionally, disrupting phones and making everyone antsy including me.
- We are trying to get an OpenVPN client to tunnel thru a public VPN gateway so VoIP bypasses the NATs on the other WANs.
- Incidentally, there's an OVPN server running here as well, on port 1194 which I use for admin, via modem and ppp0.
- Experimentation is tricky, need to keep this box ticking over.

One problem is that using a public VPN gateway (safervpn.com) we have no control over their OpenVPN server config, however they do provide sample configs for linux and I've used that with a certain amount of success. Plainly we can't have their OVPN server pushing a default route, so I've added a route-nopull to the local client config plus a few routes that we want going through the VPN. It produces a bunch of errors as a result, but I they seem to be benign. This is the stuff I'm least sure about.

Result is no VoIP traffic routed. Traceroute shows the traffic for the VoIP network (333.111.14.0/24) is still going out ppp0. Pinging remote end of tunnel gives nothing. tunnel interface (tun1) looks as dead as king Tut. OVPN seems happy though, reports tunnel set up over one of the WANs.

Occasionally ppp0 goes down (I lose the inwards VPN) after a couple of hours of starting the OVPN client which seems a bit strange. Getting the local guy to reboot the whole box fixes it but VoIP is dead (ping is silent) until I stop the OVPN client and then it all comes right.

It looks to me like a clash of Multi-WAN with the static routes added by OVPN. It sometimes works with metric set to zero but I need to use metrics so that I can change routes without taking down ppp0 and screwing up VoIP calls and my own VPN. Also, I think the route back from tun1 to the sender isn't working - tun1 shows traffic both ways but no ping response (logs and config below).

I've battled with network-scripts/route-xxx files, but that system seems unable to cope with tun1, mostly I think because there's no ifcfg-tun1 (perhaps should I make one?)

Does anyone have a clue how to fix this?

FYI, 104.24.3.14 is just a random website which will return my pings and is handy for testing. 333.111.x.x is the VoIP provider.


This ping goes out some other WAN:
# ping 104.24.3.14
PING 104.24.3.14 (104.24.3.14) 56(84) bytes of data.
64 bytes from 104.24.3.14: icmp_seq=1 ttl=57 time=65.1 ms

Forcing it out tun1 goes nowhere:
# ping -I tun1 104.24.3.14
PING 104.24.3.14 (104.24.3.14) from 10.7.0.6 tun1: 56(84) bytes of data.
--- 104.24.3.14 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 6999ms

But there was traffic (tun1 had zero traffic just prior):
# ifconfig tun1
tun1: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1500
inet 10.7.0.6 netmask 255.255.255.255 destination 10.7.0.5
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 100 (UNSPEC)
RX packets 8 bytes 672 (672.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 672 (672.0 B)

And when I look closely at the ping traffic:
# tcpdump -vvv -i tun1
tcpdump: listening on tun1, link-type RAW (Raw IP), capture size 65535 bytes
17:22:53.910472 IP (tos 0x0, ttl 64, id 58729, offset 0, flags [DF], proto ICMP (1), length 84)
mako.lan > 104.24.3.14: ICMP echo request, id 19801, seq 1, length 64
17:22:54.378427 IP (tos 0x0, ttl 60, id 33888, offset 0, flags [none], proto ICMP (1), length 84)
104.24.3.14 > mako.lan: ICMP echo reply, id 19801, seq 1, length 64



# cat /etc/openvpn/safervpn.config
client
dev tun
proto tcp
cipher AES-256-CBC
remote-cert-tls server
remote blah1.safervpn.net 443 # - Your server IP and OpenVPN Port
resolv-retry infinite
nobind
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
persist-key
persist-tun
auth-user-pass blahblah-pass
comp-lzo
user nobody
group nobody
verb 3

# ifconfig 10.7.0.6 10.7.0.5 # Grizzles if it's included
route 10.7.0.1
route 104.24.3.14 255.255.255.255
route-metric 40
route 333.111.14.0 255.255.255.0
route 333.111.13.0 255.255.255.0
<certificate stuff>




# ip route
default via 192.168.2.1 dev enp6s0
10.7.0.1 via 10.7.0.5 dev tun1 metric 40
10.7.0.5 dev tun1 proto kernel scope link src 10.7.0.6
10.8.10.0/24 via 10.8.10.2 dev tun0
10.8.10.2 dev tun0 proto kernel scope link src 10.8.10.1
10.10.24.0/24 dev enp4s0.20 proto kernel scope link src 10.10.24.5
104.24.3.14 via 10.7.0.5 dev tun1 metric 40
333.111.14.0/24 via 10.7.0.5 dev tun1 metric 40
333.111.14.0/24 dev ppp0 metric 80
333.111.14.0/24 dev enp6s0 metric 100
303.29.31.1 dev ppp0 proto kernel scope link src 444.64.31.32
303.38.90.35 dev ppp0
192.168.0.0/24 dev enp4s0.10 proto kernel scope link src 192.168.0.5
192.168.0.3 via 192.168.11.5 dev enp4s0
192.168.2.0/24 dev enp6s0 proto kernel scope link src 192.168.2.211
192.168.3.0/24 dev enp5s0 proto kernel scope link src 192.168.3.5



# cat route-ppp0
27.111.14.0/24 via 0.0.0.0 dev ppp0 metric 80



# cat ifcfg-ppp0
DEVICE=ppp0
TYPE="xDSL"
USERCTL="no"
BOOTPROTO="dialup"
NAME="DSLppp0"
ONBOOT="yes"
PIDFILE="/var/run/pppoe-ppp0.pid"
FIREWALL="NONE"
PING="."
PPPOE_TIMEOUT="80"
LCP_FAILURE="5"
LCP_INTERVAL="20"
CLAMPMSS="1412"
CONNECT_POLL="6"
CONNECT_TIMEOUT="80"
DEFROUTE="yes"
SYNCHRONOUS="no"
ETH="enp10s0"
PROVIDER="DSLppp0"
PEERDNS="yes"
USER="zzzzzz"
MTU="1492"



# tail -n 200 /var/log/messages | grep openvpn
OpenVPN 2.3.12 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Aug 23 2016
library versions: OpenSSL 1.0.1e-fips 11 Feb 2013, LZO 2.06
WARNING: file 'safervpn-pass' is group or others accessible
Socket Buffers: R=[87380->87380] S=[16384->16384]
NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Attempting to establish TCP connection with [AF_INET]185.121.168.47:443 [nonblock]
TCP connection established with [AF_INET]185.121.168.47:443
TCPv4_CLIENT link local: [undef]
TCPv4_CLIENT link remote: [AF_INET]185.121.168.47:443
TLS: Initial packet from [AF_INET]185.121.168.47:443, sid=003655db 9a7c7966
WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
VERIFY OK: depth=1, C=US, ST=CA, L=SanFrancisco, O=SaferVPN, OU=SaferVPN, CN=SaferVPN, name=SaferVPN, emailAddress=support@s
Validating certificate key usage
++ Certificate has key usage 00a0, expects 00a0
VERIFY KU OK
Validating certificate extended key usage
++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
VERIFY EKU OK
VERIFY OK: depth=0, C=US, ST=CA, L=SanFrancisco, O=SaferVPN, OU=SaferVPN, CN=SaferVPN, name=SaferVPN, emailAddress=support@s
Data Channel Encrypt <messages omitted>
[SaferVPN] Peer Connection Initiated with [AF_INET]185.121.168.47:443
SENT CONTROL [SaferVPN]: 'PUSH_REQUEST' (status=1)
PUSH: Received control message: 'PUSH_REPLY,dhcp-option DNS 8.8.8.8,dhcp-option DNS 8.8.4.4,explicit-exit-notify,redirect-ga.1,topology net30,ping 5,ping-restart 30,ifconfig 10.7.0.6 10.7.0.5'
Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
Options error: option 'redirect-gateway' cannot be used in this context ([PUSH-OPTIONS])
Options error: option 'route' cannot be used in this context ([PUSH-OPTIONS])
OPTIONS IMPORT: timers and/or timeouts modified
OPTIONS IMPORT: --explicit-exit-notify can only be used with --proto udp
OPTIONS IMPORT: --ifconfig/up options modified
ROUTE_GATEWAY 192.168.2.1/255.255.255.0 IFACE=enp6s0 HWADDR=dg:ch:id:11:ff:bb
TUN/TAP device tun1 opened
TUN/TAP TX queue length set to 100
do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
/usr/sbin/ip link set dev tun1 up mtu 1500
/usr/sbin/ip addr add dev tun1 local 10.7.0.6 peer 10.7.0.5
/usr/sbin/ip route add 10.7.0.1/32 metric 40 via 10.7.0.5
/usr/sbin/ip route add 333.111.14.0/24 metric 40 via 10.7.0.5
GID set to nobody
UID set to nobody
Initialization Sequence Completed
In OpenVPN
Saturday, October 28 2017, 01:33 PM
Share this post:
Responses (1)
  • Accepted Answer

    Sunday, November 26 2017, 09:12 AM - #Permalink
    Resolved
    0 votes
    Hi Lindsay,
    Welcome to the forum and sorry for ignoring you. New users posts get moderated and if they are not released very quickly I tend to miss them if the don't appear high up on the first page.

    If you have not managed to do anything, I think this is beyond me unless I spend a huge amount of time on it, but one of the devs has done some clever routing in the ibVPN app to route particular LAN IP's via ibVPN. It may give you some ideas. Have a look at the /etc/clearos/ibvpn.d/route-up script in app-ibvpn-core in clearos contribs
    The reply is currently minimized Show
Your Reply