Tux Brothers GFORCE1 wrote:
Can you ask dev team what differences there were?

No one would know. We use the upstream kerne kernel unaltered. We get it from Centos. I don't think they do anything to it and they get it from RedHat

Hi Nick,

With the update to kernel-3.10.0-1160.42.2.el7 after a reboot all went well. Can you ask dev team what differences there were?
I checked the 3 servers but 2 have Intel network cards and one has a Broadcom.

Sorry, but I've no ideas. Ethernet should be connecting faster than PPPoE in my experience.

Nick Howitt wrote:

The problem may be before your logging, but I don't like the segfault in app-firewall and don't know how to troubleshoot it. It does not look like you have internet connectivity at the beginning of that log snippet. Are you on PPPoE or Ethernet for your WAN?

On all servers with this problem have Ethernet.

The problem may be before your logging, but I don't like the segfault in app-firewall and don't know how to troubleshoot it. It does not look like you have internet connectivity at the beginning of that log snippet. Are you on PPPoE or Ethernet for your WAN?

Nick Howitt wrote:

Really strange and I have no idea what the cause is if a second boot fixes it. Is /boot out of space:

df -h

[root@net ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.8G 0 3.8G 0% /dev/shm
tmpfs 3.8G 9.1M 3.8G 1% /run
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
/dev/sda3 923G 646G 278G 70% /
/dev/sda1 497M 224M 274M 45% /boot
tmpfs 772M 0 772M 0% /run/user/991
tmpfs 772M 0 772M 0% /run/user/0

Logfile snip:

Nov 20 13:30:40 net ntpd_intres[795]: host name not found: time4.clearsdn.com
Nov 20 13:30:40 net systemd: firewall.service: main process exited, code=killed, status=15/TERM
Nov 20 13:30:42 net systemd: Unit firewall.service entered failed state.
Nov 20 13:30:42 net systemd: firewall.service failed.
Nov 20 13:30:42 net kernel: ip_set: protocol 7
Nov 20 13:30:43 net kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team
Nov 20 13:30:43 net kernel: tun: Universal TUN/TAP device driver, 1.6
Nov 20 13:30:43 net kernel: tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Nov 20 13:30:43 net kernel: PPP generic driver version 2.4.2
Nov 20 13:30:43 net kernel: PPP MPPE Compression module registered
Nov 20 13:30:43 net nmbd[1051]: [2021/11/20 13:30:43.766322, 0] ../../source3/nmbd/asyncdns.c:158(start_async_dns)
Nov 20 13:30:43 net nmbd[1051]: started asyncdns process 1430
Nov 20 13:30:43 net mariadb-prepare-db-dir: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Nov 20 13:30:43 net mariadb-prepare-db-dir: If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Nov 20 13:30:43 net mariadb-prepare-db-dir: Database MariaDB is probably initialized in /var/lib/system-mysql already, nothing is done.
Nov 20 13:30:43 net mariadb-prepare-db-dir: If this is not the case, make sure the /var/lib/system-mysql is empty before running mariadb-prepare-db-dir.
Nov 20 13:30:44 net kernel: app-firewall[1336]: segfault at 0 ip 0000000000404893 sp 00007ffebcc35bf0 error 4 in app-firewall[400000+10000]
Nov 20 13:30:44 net nmbd[1051]: [2021/11/20 13:30:44.710403, 0] ../../lib/util/become_daemon.c:136(daemon_ready)
Nov 20 13:30:44 net nmbd[1051]: daemon_ready: daemon 'nmbd' finished starting up and ready to serve connections
Nov 20 13:30:44 net ntpd[794]: Listen normally on 8 tun0 10.8.0.1 UDP 123
Nov 20 13:30:44 net ntpd[794]: Listen normally on 9 tun1 10.8.10.1 UDP 123
Nov 20 13:30:44 net ntpd[794]: Listen normally on 10 tun1 fe80::8564:fb63:a34b:b334 UDP 123
Nov 20 13:30:44 net ntpd[794]: Listen normally on 11 tun0 fe80::6799:73d6:aa7f:27e5 UDP 123
Nov 20 13:30:44 net systemd: firewall.service: main process exited, code=exited, status=1/FAILURE
Nov 20 13:30:44 net systemd: Failed to start ClearOS Firewall Engine.
Nov 20 13:30:44 net systemd: Unit firewall.service entered failed state.
Nov 20 13:30:44 net systemd: firewall.service failed.
Nov 20 13:30:45 net clearsyncd[983]: System Events: Socket hang-up: 32
Nov 20 13:30:45 net mysqld_safe: 211120 13:30:45 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.

Really strange and I have no idea what the cause is if a second boot fixes it. Is /boot out of space:

df -h

Hi Nick,
After installing latest kernel the problem is back: IPv4 Firewall Error - Restricted Access Only 2021-11-20 13:35:06
I had to reboot again and the the firewall is OK.
Seems to be a kernel update thing, everytime i install a new kernel on certain machines the problem comes back.

Hi Nick,

After installing latest updates and new kernel the problems seem te be gone!
A second reboot is also not needed anymore to get all ports working.

Cheers,

Hi Nick,

FYI
After latest updates the error seems to be gone now after a reboot (when installed a new kernel)!

It should be fine. If it won't downgrade, post back with the error. If it does not fix the problem, just let it upgrade automatically overnight or do a "yum update iptables".

Hi Nick,

I will give it a go this weekend! You are sure its harmless to revert iptables in combination with Clear??

.... and it looks like there is another kernel update from upstream being released today. They seem to be coming quite quickly at the moment.

[edit]
Correction. It looks like a centos-plus kernel.
[/edit]

I understand the issue with rebooting to different kernels. I've uploaded the previous version of iptables to my server. you can probably downgrade with:

wget https://www.howitts.co.uk/clearos/iptables-1.4.21-34.el7.x86_64.rpm

wget https://www.howitts.co.uk/clearos/iptables-devel-1.4.21-34.el7.x86_64.rpm

yum downgrade iptables*

You won't need the devel package if you don't currently have it installed. If rebooting works OK, you'll need to block the update in /etc/yum.conf with a line something like:

exclude=iptables*-1.4.21-35*

Rebooting with other kernels is hard because with the corona thing its hard to get onsite. And i dont like to do it remote to change kernel sequence.

They are all Community versions. And i noticed it due last kernel and firewall app update. Really strange thing is that yesterday i updated a same kind of server with 100% the same rule and i didnt get error.

This weekend i can try to remove the rule completely and put in as a new rule.Will that make a change?

There was an app-firewall update recently and all it did was add a port to the standard port list (tcp:11625/Stellar Core). The last update was last summer. Iptables did update with the 7.9 update. The kernel updated with 7.9. Paid has had one update since 7.9. Community has had 2 (paid should be getting the second one tomorrow).

@Tux Brothers, can you see what happens when booting to older kernels. Try the earlier 7.9 kernels (3.10.0-1160) and at least one of the 7.8 kernels (3.10.0-1127).

Also do you have a paid version? If you do, we can try downgrading iptables to a previous version. If you're on Community, I'd need to try to make the earlier package available.

What updates have you received recently?

Never seen an error like the one discussed here in any of the CentOS or Fedora newsgroups, so first suspect IMHO would lie with app_firewall. Isn't that a ClearOS application? Who in ClearOS wrote it, or was it 'borrowed' from elsewhere. Also note in the original append - "Since i installed the latest updates on one of my production machine". Has app-firewall been updated recently? The OP neglected to list what those updates were in that append... However, later "I see it on a second machine also, when i applied the new firewall app and the new kernel" it would seem that it has indeed been updated. Surely whatever the changes were to produce the update(s) need to be investigated first, who made them? Shouldn't that person(s) be involved with this issue? Also a simple test to see if it is the newer kernel alone or just the app_firewall update is responsible (or combination of the two).

Good info, but what can be done about the call failing to get an iptables lock? Is it because we are getting so many firewall restarts during boots that we are getting contention of some sorts as all the "iptables -w" comands back up?

iptc_init is called by most of the iptables binaries. Also /usr/sbin/app-firewall and /usr/sbin/app-firewall6

Hi Nick,

Yes all machines are on ClearOS 7 latest!

Assuming you are on ClearOS7, that line in firewall.lua calls iptc_init(), but I can't find the iptc_init function anywhere if any ClearOS code. There is a function of that name in libiptc but it seems to require a table name so is probably not relevant. I am a bit stumped.

Hi Nick ,

Ive been testing this morning, now this is really strange. On a test machine it appeared one time (the error) and now its gone ...
In the log files i found this:
Error: /usr/clearos/apps/firewall/deploy/firewall.lua:3067: Unable to acquire xtables lock.

I have no idea then. I'll see if I can give it a try sometime.

Hi Nick,

Ofcourse is the 0.0.0.0 is only fictional (cause i dont want to publish the real ip here), so lats say the ip is 123.123.123.123 and the port is 22222.

The rule looks really odd as 0.0.0.0 should never be in use. What are you trying to achieve?

Hi Nick,

I see it on a second machine also, when i applied the new firewall app and the new kernel and a reboot i noticed some network services (httpd and ftp) did not work. After a second reboot they came back.
Anyway on this machine same rule and same error after a reboot.

I don't see that that rule can be causing a persistent error as the rule itself is OK. Just ignore the event during start. When booting, the firewall does multiple restarts, but should finish up working OK.

Hi Nick,

The error only is visible after a reboot! (in the Event and Notifications)

The rule out have shown works directly in my system, even with 000.000.000.00.

So what is the exact rule? Try it from the command line replacing "$IPTABLES" with "iptables".

Hi Nick,

When i disable the rule the error is gone!

The "00" was a typo! Ps the rule worked perfect for 2 years !

I created the rule on your advice ;-) Its for opening 1 port for one external ip adress only.

If you disable the rule, does it work? If it does, then it is the error. If it still does not work, start the firewall in debug mode "firewall-start -d".

That rule is odd. If you have leading zeros in an IP address it indicated you are using octal notation. I don't know if you need three character for that in which case your "00" is invalid. You'd do better to stick to the conventional 0.0.0.0.

What are you trying to achieve? If you are trying to disable the firewall, what about UDP, ICMP and so on? Better and simpler would be:

$IPTABLES -I INPUT -j ACCEPT