Nick Howitt wrote:

What sort of devices are you trying to connect to? If they are Windows devices, can you please try dropping their firewalls. Also you could try sniffing the packets on the ClearOS LAN interface, either with tcpdump of a firewall rule, but I doubt if you have a problem as the pings are getting through.

It isn't a client firewall issue. The majority of systems running on the other side are Macs or linux boxes (mostly CentOS 7) with firewall disabled since they are hosting private internal services on a variety of ports for a ton of things.

If we go back to the start of this problem as well - there were no issues with client connectivity between sites when using ClearOS 6. It was the move to ClearOS 7 that started to create this issue - so it is safe to rule out anything on the client side as none of that has changed between when I had a perfectly working tunnel under COS6 and now. Also, it is important to note that when using OpenVPN to do remote client connectivity everything works as expected to all clients at either site - so client configuration isn't an issue.

Nick Howitt wrote:
What may be interesting is seeing the source IP's on the packets as the exit the remote firewall.

I'd be very interested in some example tests that I could run with TCPDump to determine what might be happening. I'm admittedly a bit of a neophyte with the tool and the documentation listed in this thread is open enough that it is hard to determine where I should even start. Internal IP to external IP? All traffic? From where? To where? I'm happy to do any tests recommended to get this sorted.

Nick Howitt wrote:

Your traceroutes are completely wrong as nothing is going through the tunnel. You should not see any hops between the IPsec endpoints. If you want to check the tunnel working you should be doing a traceroute to the remote ClearOS LAN IP directly, From A do a traceroute to rightsourceip=192.168.10.1 or any non-Windows device on the remote LAN. The problem with Windows is its firewall, where it may reject pings and other connections from outside its own LAN. You would need to check its settings.

That's likely because I did external interface to external interface. Doing internal to internal looks like this:

Firewall up and down are identical in this case (site A to site B, from 192.168.3.1 internal LAN):


Doing the reverse also works identical with firewall up or down (site B to site A, from 192.168.10.1 internal LAN)



These are both router to router trace routes, so the tunnel is up and the two end points are responding. Going onto a client at site A and doing a trace route to a known good running machine at site B gets me this:



However I cannot access any services on that machine (192.168.10.5) through the tunnel - the firewall appears to be blocking all of them.

If I do the inverse and trace route a machine on site A from a machine on site B I get the following:



In both cases, the firewall being on/off has no impact to the trace route results. But I can't actually access any services on any machines on the opposite side of the tunnel.

As soon as I turn off the firewall at both sites, my tunnel works as expected, but now my NAT/forwarding doesn't work at either site (which is expected). So there is a config in iptables that is taking precedent over the tunnel and blocking communications. At this point that is pretty clear because when it is removed from the equation everything works. What I can't figure out is what that setting is or what it is happening the way it is.

Nick Howitt wrote:
Similarly if you want to access a web server at B from A, the FQDN at A should resolve to a LAN IP at B and vice versa. This is achieved through the DNS Server settings (or the hosts file)

It isn't a DNS issue. As I said earlier, completely eliminating DNS 100% from all configs and using only IPs results in the exact same behavior under the same tests. That said, I have full working forward/reverse DNS at both sites for the LANs.

Nick Howitt wrote:
Can we start again with the troubleshooting as things may have changed over 2 years. Please can you give the conn definitions from either end from /etc/ipsec.d.*.conf and the connection log from both ends.

Site A


Site B

Nick Howitt wrote:
With the firewall up and down at B, can you give a traceroute from A to B.
[quote]

Firewall Up


Firewall Down


So the firewall is successfully stopping the traceroute it looks to me.

[quote]Nick Howitt wrote:
Thinking about this some more, is this a DNS issue? Are you trying to access machines by FQDN when you get the issue? Can you try pinging the remote host by IP and FQDN? Is pinging by FQDN pinging your remote external IP?

If this is the case, in Site A you need a DNS entry for the machines in Site B to go directly to the LAN IP. I am not sure how this work with port forwards where you only forward certain IP's to LAN machines and it can be different IP's. I have a feeling it will fail. Try setting the DNS entry for machines you for which have port forwarding rules to point to the remote ClearOS LAN IP.

Does not seem to be DNS related. If I remove DNS from the equation completely (both on the public side and internally) I still get the same results.

After 2 years (!?) I've finally had some time to sit down and diagnose this some more and am still having the same issues. That said, I HAVE nailed this down to being a rule in the side B firewall. If I stop the firewall service with , I instantly get access to everything on both sides. Obviously this also brings down my router's function as an internet gateway at the same time and leaves a pretty big security hole, so not a good viable option.

I'm ready to take a look deeper now that I at least have an idea of where the problem is. I've spent quite some time today trying to compare iptables outputs between both sides and do not see anything obvious, but turning off the firewall on one side makes it all work so I know it is hiding in there somewhere (maybe even a rule order)? If there are specific tests that can help nail this down with tcpdump, happy to run them as well. Any help is appreciated. Below is the output of the Site B firewall config as of today:

Nick Howitt wrote:

From the logs I am not interested in anything down to "loading ipsec.secrets". That is just Libreswan (not Openswan) start up stuff. It is the tunnel negotiation after that where the interesting (or not) stuff happens. Your logs don't match as they are a few minutes out and both logs show Libreswan initiating and not responding. I could do with the response log, say from site B, at around the time you restarted IPsec at site A. I only have a responding set up so I can't necessarily compare, but your logs are missing what I would expect but that could only be a responding thing.

In your conn, have you defined the subnets you want to link? I suspect not. Can you post the contents of any conf file in /etc/ipsec.d.

Yes, I have defined this. Here are the configs from both sides:

Side A



Side B

Nick Howitt wrote:
I think you are using one of the ClearOS packages because your log file is being redirected. Can you clarify which one? Is it the free Static IPsec VPN for home?

I'm using the ClearOS Static IPSEC VPN for Home package.

Nick Howitt wrote:
[edit]
BTW, if you do a tcpdump on the external interface you should only see encrypted packets.
[/edit]

[edit2]
Also, if not set, please set your Local LAN IP to your ClearOS LAN IP. It helps server to server communication. There is no need to set the Remote LAN IP. This does nothing.
[/edit2]

I've removed the Local LAN IP directives on both sides and still don't have a working two way tunnel.

Additionally, here are the startup logs for both sites /var/log/ipsec

Side A



Side B

Here is the dumps from side A





Here are the dumps from side B



[code]
[root@router ~]# iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 755K packets, 96M bytes)
pkts bytes target prot opt in out source destination
17323 1108K DNAT tcp -- * * 0.0.0.0/0 my.public.ip tcp dpt:443 to:192.168.10.73:443
713 42632 DNAT tcp -- * * 0.0.0.0/0 my.public.ip tcp dpt:80 to:192.168.10.37:80
15 960 DNAT tcp -- * * 0.0.0.0/0 my.public.ip tcp dpt:22222 to:192.168.10.20:22

Chain INPUT (policy ACCEPT 141K packets, 12M bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 157K packets, 10M bytes)
pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 39386 packets, 2509K bytes)
pkts bytes target prot opt in out source destination
1031 65951 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 policy match dir out pol ipsec
1917 122K ACCEPT all -- * tun+ 0.0.0.0/0 0.0.0.0/0
12592 806K SNAT tcp -- * * 192.168.10.0/24 192.168.10.73 tcp dpt:443 to:192.168.10.1
41 2624 SNAT tcp -- * * 192.168.10.0/24 192.168.10.37 tcp dpt:80 to:192.168.10.1
0 0 SNAT tcp -- * * 192.168.10.0/24 192.168.10.20 tcp dpt:22 to:192.168.10.1
471K 37M MASQUERADE all -- * eno1 0.0.0.0/0 0.0.0.0/0
[/node]

Where my.public.ip is obviously different between site A and B.

Thanks Dave.

I do not have any proxy servers running right now, so that is why the port forward issue is a bit perplexing. I feel like there is a firewall issue on one side blocking traffic on the tunnel, but that isn't exactly scientific.

Given the issue, I'm not entirely sure which tcpdump path I should take. I ran ping tests and pings to hosts behind the tunnel at site B are actually received by the local router.

Some site info:

Site A - subnet 192.168.3.0/24
Site B - subnet 192.168.10.0/24

Here is what I did so far:

Fired up tcpdump -i <interface> icmp[icmptype]=icmp-echo on all four of my ClearOS interfaces (internal and external on both systems)

Pinged from a client behind ClearOS A to a client behind ClearOS B
I see the pings showing up on the appropriate interfaces, but I get timeouts on the ping

Pinged from a client behind ClearOS B to a client behind ClearOS A
I see the exact same tcpdump behavior (in reverse this time for obvious reasons) but the pings actually return.

I'm open for any/all other tests I can run here to determine what might be going on. The tunnel is up and no errors in the logs, so this seems to be firewall related. Again, I'm guessing a bit here.