Nick Howitt wrote:

Blame that one on me.The cyrus-imap jail was not working correctly (at all) so I pushed a couple of updates. It looks like I may have missed the check to see if the file existed in the first place or something similar has gone wrong. I guess I'll have to have another look at it. It will be harder to fix now!

Thanks for the heads up.

Thank you Nick for your incredible work !

Blame that one on me.The cyrus-imap jail was not working correctly (at all) so I pushed a couple of updates. It looks like I may have missed the check to see if the file existed in the first place or something similar has gone wrong. I guess I'll have to have another look at it. It will be harder to fix now!

Thanks for the heads up.

I post this here because it is the first (among very few) that I found while googling my issue on ClearOs7.

After rebooting a test server that was sleepy for a few weeks and after all the updates were completed I check the Attack Detector app to find out that fail2ban wasn’t running.
Each attempt to start it from the webconfig or terminal failed right away.

I used

/usr/bin/fail2ban-client -v -v start

to have more information and saw:

INFO     Loading files: ['/etc/fail2ban/jail.d/clearos-cyrus-imap.conf']

ERROR  Failed during configuration: File contains no section headers.

file: /etc/fail2ban/jail.d/clearos-cyrus-imap.conf, line: 1

'port = imap,imap3,imaps,pop3,pop3s\n'

To check I deleted "/etc/fail2ban/jail.d/clearos-cyrus-imap.conf" and fail2ban restarted right successfully.

I don’t use a mail server or have any need for incoming mail on this unit.
Should I tried to understand what was wrong with "clearos-cyrus-imap.conf" or it shouldn’t be here anyway ?

Bernard

It could be. local fires off after 90 and custom. You certainly don't need it anymore now ClearOS provide the 90 file and you are using ipset jails.

Nick,

I've continued to check this out and haven't been able to figure anything more until today.

I noticed in the /etc/clearos/firewall.d/local there is a line

service fail2ban restart

and nothing else.

Also there is a /etc/clearos/firewall.d/90-attack-detector file with a bunch of code.

Is it possible that the 90-attack-detector has started fail2ban and local tries to start fail2ban causing it to hang?

Thanks again for your help and patience as I try to figure this out.

I've just tested and yes, there is a firewall restart when adding a block. That's a bit brutal.

The equivalent of the 5.x rc.firewall.local is /etc/clearos/firewall.d/local. Each time you save changes to it the firewall restarts as well. Again use $IPTABLES or "iptables -w". Stick them in the ipv4 section.

You appear to have a mixture of ipset and iptables blocks. ClearOS seems to work better with ipset jails so I'd be tempted to change the banaction in jail.conf or jail.local to "iptables-ipset-proto6". I've done that for mine.

Otherwise I don't know how to tidy things up. I'd be tempted to try a "firewall-start -d" if it is the firewall which is causing the problem. This will start the firewall in debug mode and give you some more output. as it starts. You can correlate the ouput to /var/log/system and/var/log/fail2ban.log

Nick Howitt wrote:
It looks like a firewall restart has happened and this has wiped a lot of f2b firewall bits. Why it restarted, I don't know. When that happens, when f2b tries to shut down, lots of its rules which it wants to delete no longer exist generating those errors. Ignore them.

It restarted when I added an IP to the Firewall:Incoming list in the Firewall App. This happens each and every time I add an IP to the list. It isn't in the custom firewall menu.

Nick Howitt wrote:Correct me if I'm wrong, but this is ClearOS 7. Do you have app-attack-detector installed any more?

Yes, this is COS 7.x. I reinstalled app-attack-detector from the MarketPlace after I realized that the problems that I was having was with using systemd log. I left my changes in the jail.local but everything else is standard.

Nick Howitt wrote:What firewall files do you have related to fail2ban or attack-detector in /etc/clearos/firewall.d/?

I haven't added anything to the firewall.d manually here is the list:

ll /etc/clearos/firewall.d/*

-rwxr-xr-x 1 root root   95 Feb  4 11:36 /etc/clearos/firewall.d/10-ntp

-rw-r--r-- 1 root root 1156 Aug 20 11:10 /etc/clearos/firewall.d/10-snortsam

-rwxr-xr-x 1 root root 1580 Feb 14 13:37 /etc/clearos/firewall.d/90-attack-detector

-rwxr-xr-x 1 root root  326 Jan 24 03:09 /etc/clearos/firewall.d/custom

-rwxr-xr-x 1 root root  212 Jan 11 17:24 /etc/clearos/firewall.d/local

-rwxr-xr-x 1 root root 1467 Dec 12 14:52 /etc/clearos/firewall.d/types

Nick Howitt wrote:Also what is the contents of /etc/clearos/firewall.d/custom and /etc/clearos/firewall.d/local, if any.

cat /etc/clearos/firewall.d/custom 

#######################################

# Created by API - Please Do NOT Edit #

#######################################



# IPv4 Custom Firewall Rules

#===========================



if [ "$FW_PROTO" == "ipv4" ]; then true

fi



# IPv6 Custom Firewall Rules

#===========================



if [ "$FW_PROTO" == "ipv6" ]; then true

fi

cat /etc/clearos/firewall.d/local 

# This script is run after every firewall restart.  Add custom rules here.

# Ensure you use $IPTABLES instead of calling iptables directly if you wish

# to avoid xtable locking problems.

service fail2ban restart

Nick Howitt wrote:AFAIK, the firewall only restarts when deleting a rule, not when adding it.

In my case it restarts after adding a rule. I haven't tried deleting any rules.
It should be able to delete a rule without restarting too.

Nick Howitt wrote:Although it restarts when you change and save /etc/clearos/firewall.d/local, so it may do the same with /etc/clearos/firewall.d/custom, which I think is what you are effectively editing.

I haven't a clue where the block rules that are added in the Firewall:Incoming are added. But each time I add one, the firewall restarts and kills fail2ban.

On COS5.2, I had a list of IPs that I blocked because they were a pest. It was some sort of local file rc.firewall.local that had a list of commands with IPTABLES -A INPUT -s ipaddress -j -DROP. I used to manually add the IPs there so they would reblock upon a system restart. The IPTABLES command would add the block manually without restarting the firewall. I don't know where to add these now with this new firewall.d.

It looks like a firewall restart has happened and this has wiped a lot of f2b firewall bits. Why it restarted, I don't know. When that happens, when f2b tries to shut down, lots of its rules which it wants to delete no longer exist generating those errors. Ignore them.

Correct me if I'm wrong, but this is ClearOS 7. Do you have app-attack-detector installed any more? What firewall files do you have related to fail2ban or attack-detector in /etc/clearos/firewall.d/?

Also what is the contents of /etc/clearos/firewall.d/custom and /etc/clearos/firewall.d/local, if any.

AFAIK, the firewall only restarts when deleting a rule, not when adding it.

[edit]
Although it restarts when you change and save /etc/clearos/firewall.d/local, so it may do the same with /etc/clearos/firewall.d/custom, which I think is what you are effectively editing.
[/edit]

So here is what happened when I added an ip address to the Network:Firewall:Incoming Firewall:Blocked Incoming Connections (Add).

2018-02-15 15:20:30,159 fail2ban.transmitter [12468]: DEBUG Command: ['stop']

2018-02-15 15:20:30,159 fail2ban.asyncserver [12468]: DEBUG Removed socket file /var/run/fail2ban/fail2ban.sock

2018-02-15 15:20:30,159 fail2ban.asyncserver [12468]: DEBUG Socket shutdown

2018-02-15 15:20:30,159 fail2ban.server [12468]: INFO Stopping all jails

2018-02-15 15:20:30,159 fail2ban.server [12468]: DEBUG Stopping jail cyrus-imap

2018-02-15 15:20:30,229 fail2ban.filterpoll [12468]: DEBUG cyrus-imap filter terminated

2018-02-15 15:20:30,689 fail2ban.actions [12468]: DEBUG Flush ban list

2018-02-15 15:20:30,689 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap

2018-02-15 15:20:30,793 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- stdout: ''

2018-02-15 15:20:30,794 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- stderr: "iptables v1.4.21: Couldn't load target `f2b-cyrus-imap':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\niptables: No chain/target/match by that name.\niptables: No chain/target/match by that name.\n"

2018-02-15 15:20:30,794 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- returned 1

2018-02-15 15:20:30,794 fail2ban.actions [12468]: ERROR Failed to stop jail 'cyrus-imap' action 'iptables-multiport': Error stopping action

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/fail2ban/server/actions.py", line 241, in run

action.stop()

File "/usr/lib/python2.7/site-packages/fail2ban/server/action.py", line 372, in stop

raise RuntimeError("Error stopping action")

RuntimeError: Error stopping action

2018-02-15 15:20:30,810 fail2ban.actions [12468]: DEBUG cyrus-imap: action terminated

2018-02-15 15:20:30,810 fail2ban.jail [12468]: INFO Jail 'cyrus-imap' stopped

2018-02-15 15:20:30,811 fail2ban.server [12468]: DEBUG Stopping jail postfix-sasl

2018-02-15 15:20:31,246 fail2ban.filterpoll [12468]: DEBUG postfix-sasl filter terminated

2018-02-15 15:20:31,594 fail2ban.actions [12468]: DEBUG Flush ban list

2018-02-15 15:20:31,594 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl

2018-02-15 15:20:31,699 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- stdout: ''

2018-02-15 15:20:31,699 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- stderr: "iptables v1.4.21: Couldn't load target `f2b-postfix-sasl':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\niptables: No chain/target/match by that name.\niptables: No chain/target/match by that name.\n"

2018-02-15 15:20:31,699 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- returned 1

2018-02-15 15:20:31,700 fail2ban.actions [12468]: ERROR Failed to stop jail 'postfix-sasl' action 'iptables-multiport': Error stopping action

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/fail2ban/server/actions.py", line 241, in run

action.stop()

File "/usr/lib/python2.7/site-packages/fail2ban/server/action.py", line 372, in stop

raise RuntimeError("Error stopping action")

RuntimeError: Error stopping action

2018-02-15 15:20:31,700 fail2ban.actions [12468]: DEBUG postfix-sasl: action terminated

2018-02-15 15:20:31,700 fail2ban.jail [12468]: INFO Jail 'postfix-sasl' stopped

2018-02-15 15:20:31,701 fail2ban.server [12468]: DEBUG Stopping jail sshd

2018-02-15 15:20:32,150 fail2ban.filtersystemd [12468]: DEBUG sshd filter terminated

2018-02-15 15:20:32,380 fail2ban.actions [12468]: DEBUG Flush ban list

2018-02-15 15:20:32,381 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd src -j DROP

ipset flush f2b-sshd

ipset destroy f2b-sshd

2018-02-15 15:20:32,485 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd src -j DROP

ipset flush f2b-sshd

ipset destroy f2b-sshd -- stdout: ''

2018-02-15 15:20:32,485 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd src -j DROP

ipset flush f2b-sshd

ipset destroy f2b-sshd -- stderr: 'iptables: Bad rule (does a matching rule exist in that chain?).\n'

2018-02-15 15:20:32,485 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd src -j DROP

ipset flush f2b-sshd

ipset destroy f2b-sshd -- returned successfully

2018-02-15 15:20:32,485 fail2ban.actions [12468]: DEBUG sshd: action terminated

2018-02-15 15:20:32,486 fail2ban.jail [12468]: INFO Jail 'sshd' stopped

2018-02-15 15:20:32,486 fail2ban.server [12468]: DEBUG Stopping jail sshd-ddos

2018-02-15 15:20:32,491 fail2ban.actions [12468]: DEBUG Flush ban list

2018-02-15 15:20:32,492 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd-ddos src -j DROP

ipset flush f2b-sshd-ddos

ipset destroy f2b-sshd-ddos

2018-02-15 15:20:32,596 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd-ddos src -j DROP

ipset flush f2b-sshd-ddos

ipset destroy f2b-sshd-ddos -- stdout: ''

2018-02-15 15:20:32,596 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd-ddos src -j DROP

ipset flush f2b-sshd-ddos

ipset destroy f2b-sshd-ddos -- stderr: 'iptables: Bad rule (does a matching rule exist in that chain?).\n'

2018-02-15 15:20:32,596 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd-ddos src -j DROP

ipset flush f2b-sshd-ddos

ipset destroy f2b-sshd-ddos -- returned successfully

2018-02-15 15:20:32,596 fail2ban.actions [12468]: DEBUG sshd-ddos: action terminated

2018-02-15 15:20:33,151 fail2ban.filtersystemd [12468]: DEBUG sshd-ddos filter terminated

2018-02-15 15:20:33,152 fail2ban.jail [12468]: INFO Jail 'sshd-ddos' stopped

2018-02-15 15:20:33,153 fail2ban.server [12468]: DEBUG Remove PID file /var/run/fail2ban/fail2ban.pid

2018-02-15 15:20:33,153 fail2ban.server [12468]: INFO Exiting Fail2ban

The firewall has restarted as far as I can tell. Attack Detector shows "Restarting" but isn't. [So here I am full circle back to the start]

fail2ban-client status

ERROR  Failed to access socket path: /var/run/fail2ban/fail2ban.sock. Is fail2ban running?

systemctl status -l fail2ban.service

● fail2ban.service - Fail2Ban Service

   Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled; vendor preset: disabled)

   Active: inactive (dead) since Thu 2018-02-15 15:20:33 EST; 18min ago

     Docs: man:fail2ban(1)

  Process: 14983 ExecStop=/usr/bin/fail2ban-client stop (code=exited, status=0/SUCCESS)

  Process: 12465 ExecStart=/usr/bin/fail2ban-client -x start (code=exited, status=0/SUCCESS)

 Main PID: 12468 (code=killed, signal=TERM)



Feb 15 15:01:47 domain.com systemd[1]: Starting Fail2Ban Service...

Feb 15 15:01:47 domain.com fail2ban-client[12465]: 2018-02-15 15:01:47,746 fail2ban.server         [12466]: INFO    Starting Fail2ban v0.9.5

Feb 15 15:01:47 domain.com fail2ban-client[12465]: 2018-02-15 15:01:47,746 fail2ban.server         [12466]: INFO    Starting in daemon mode

Feb 15 15:01:48 domain.com systemd[1]: Started Fail2Ban Service.

Feb 15 15:20:30 domain.com systemd[1]: Stopping Fail2Ban Service...

Feb 15 15:20:33 domain.com fail2ban-client[14983]: Shutdown successful

Please note: at 15:01 I added DEBUG to the log level and did a condrestart. It is now 15:41 and attack detector hasn't started.

If I do

systemctl start fail2ban

it hangs. If I do

systemctl stop fail2ban

systemctl start fail2ban

it starts again.

Nick Howitt wrote:

There is no way adding a firewall rule should stop f2b.

I am wondering how much work it would be to reload ClearOS? Brutal but it may be the best. Otherwise we can start again with the troubleshooting now netify has gone.

Hi Nick.

I really don't want to reinstall COS. My installation, now 5 months old, is only COS + a few Marketplace apps. Getting websites up, mail running etc. etc. will take days that I don't have time for at the moment. I haven't spent any time doing any customizations like I did with COS5.2. It's a vanilla install. Interestingly enough this all started this year around the time of the firmware updates due to the Intel security issue. Before that, everything ran OK.

I left the logging for f2b as default but will change that right after I post this message.

While I agree that adding a firewall rule shouldn't stop f2b, it appears that the behaviour is something like this: add a firewall rule, restart the firewall, restart netify, restart attack detector etc.

I agree it should be "just add this ip address to the block list". Is there a way to trace the firewall add rule behaviour in detail?

Since netify was hanging (I have a strong suspicion that it hung at the reference to the non-existant dat file ...) and systemd never got a message that netify hadn't started, I think the restart of the attack detector timed out. ???

As a debugging thought, can you change loglevel to DEBUG in /etc/fail2ban/fail2ban.conf?

There is no way adding a firewall rule should stop f2b.

I am wondering how much work it would be to reload ClearOS? Brutal but it may be the best. Otherwise we can start again with the troubleshooting now netify has gone.

I've finally got all the protocol filter and netify stuff removed.

I'm still seeing a problem with the Attack Detector/fail2ban.

If I add an IP to the block list on the incoming firewall rules, then the Attack Detector shuts down and hangs on the restart. I think the shut down and restart makes sense but I'm still at a loss why the Attack Detector won't restart.

It feels like I've spent days trying to figure this out and end up back in the same place. Aaaaaaahhhhhh.

To remove the protocol filter, you can do a "yum remove app-protocol*, but it won't remove the netify bits. If you do a "yum remove netify" it will remove the protocol filter and possibly the application filter. It wil also remove app-netify-core and app-netify-fwa core. I don't know what these are. You can then do a "yum reinstall app-application*" to reinstall the application filter and it will bring back in the dependencies it needs.

You're way ahead of me as I don't use gateway management or the application filter so I have not seen the interactions, if any.

If you enable jails through jails.local, I agree, the get enabled even if the clearos- setting is disabled and vice versa! Go figure.

My hack is:

#!/bin/sh



# IPv4 only for now

#------------------



if [ "$FW_PROTO" != "ipv4" ]; then

    return 0

fi



# Bail if fail2ban is not running

#--------------------------------



if [ ! -e /var/run/fail2ban/fail2ban.sock ]; then

    return 0

fi



# Config

#-------



ACTION_ADD="/var/clearos/attack_detector/run/f2b-add"

ACTION_DELETE="/var/clearos/attack_detector/run/f2b-delete"



rm -f $ACTION_ADD

rm -f $ACTION_DELETE



# Regenerate iptables commands

#-----------------------------



JAILS=`LANG=en_US fail2ban-client status | grep "Jail list" | sed -e 's/.*Jail list://' -e 's/,//g'`



for JAIL in $JAILS; do

    ACTION=`fail2ban-client get $JAIL actions | tail -n 1`

    if [[ $ACTION =~ ipset ]]; then

        fail2ban-client get $JAIL action $ACTION actionstart | grep "<iptables>" >> $ACTION_ADD

        #PROPERTIES=`fail2ban-client get $JAIL actionproperties $ACTION | sed -e 's/.*://' -e 's/,//g'`

        PROPERTIES="iptables chain name blocktype lockingopt protocol port returntype"



        for PROPERTY in $PROPERTIES; do

            CHECK=`echo $PROPERTY | grep -v ^known | grep -v ^action`

            if [ -n "$CHECK" ]; then

                VALUE=`fail2ban-client get $JAIL action $ACTION $PROPERTY`

                VALUE_SED=`echo $VALUE |  sed 's/\//\\\\\//g'`

                sed -i -e "s/<$PROPERTY>/$VALUE_SED/" $ACTION_ADD

            fi

        done

    else

        fail2ban-client reload $JAIL > /dev/null 2>&1

    fi

done

# the "&" in the above fail2ban-client command is optional and alows the script to finish a bit quicker



# Stops script throwing an error if no ipset type jails are enabled as ACTION_ADD will not exist

if [ -f $ACTION_ADD ]; then

    grep " \-I " $ACTION_ADD | sed "s/ \-I / -D /" > $ACTION_DELETE



    sh $ACTION_DELETE > /dev/null 2>&1

    sh $ACTION_ADD >/dev/null 2>&1

fi



# Exit 0

:

Compare it to the original /etc/clearos/firewall.d/90-attack-detector. I've only added a few lines and it is irrelevant if every jail uses ipset sets.

And one more update.
I wanted to remove the protocol filter but could figure out what it was called in the marketplace. Ended up on the Application filter which is also stopped. Tried to start the Application filter (which I was using) and it killed the Attack Detector too. This Gateway Management Community is running though.

I'm stumped.

An update.

I reinstalled Attack Detector. It runs OK when netify-fwa is stopped. I think it is using my jails.local rather than the clearos-*.conf but I'm not sure since I tried to restart the protocol filter and it brought everything crashing down again. Now I have to figure out the command sequence to stop/halt netify-fwa starting and get the fail2ban going again.

I think the protocol filter has to go. Maybe that will fix things. I haven't been using it anyway so maybe less complex is better. It's still frustrating since all this stuff ran great on COS5.2.

When it has been removed, I'll check if the iptables are properly blocking stuff.

Nick Howitt wrote:
app-attack-detector should play well with your existing f2b installation. Just check where your default banaction is set and what to. Mine is set in jail.conf and is set to "iptables-ipset-proto6". IMHO this is better than iptables as a ban action. Also it plays well with the configlet ClearOS drop into /etc/clearos/firewall.d to restart the jails. This configlet only works with ipset type jails.

I copied the iptables-ipset-proto6-allports when defined in the Clearos actions and put them into the jail.local. This is where I enabled the jails that I am using. I also changed the ban action to DROP in the iptables-common.local following on from your recommendation.

I've been searching for what all these actions.d are and when & why you use them. So far I haven't found anything helpful in the fail2ban docs. They are still at v0.8 and we have 0.9.5.

Nick Howitt wrote:
This answers your other point. There is no need to update the ClearOS set up for the firewall rules. If the jails are ipset type they already take care of that. You can hack their script to make it reload non-ipset jails as well. It needs an if...then...do_the_ClearOS_stuff...else reload_the_jail. I have done that then converted everything to ipset anyway. The ClearOS view is that if the user installs app-attack-detector then f2b runs for its bits. If the user wants other jails then it is up to the user to get them going.

I'm not quite sure what your hack does but that is part of re-learning how fail2ban works.

Thanks again for all your support.

app-attack-detector should play well with your existing f2b installation. Just check where your default banaction is set and what to. Mine is set in jail.conf and is set to "iptables-ipset-proto6". IMHO this is better than iptables as a ban action. Also it plays well with the configlet ClearOS drop into /etc/clearos/firewall.d to restart the jails. This configlet only works with ipset type jails.

This answers your other point. There is no need to update the ClearOS set up for the firwall rules. If the jails are ipset type they already take care of that. You can hack their script to make it reload non-ipset jails as well. It needs an if...then...do_the_ClearOS_stuff...else reload_the_jail. I have done that then converted everything to ipset anyway. The ClearOS view is that if the user installs app-attack-detector then f2b runs for its bits. If the user wants other jails then it is up to the user to get them going.

Nick Howitt wrote:

I suspect the final solution will be to reinstall app-attack-detector.

Thanks Nick. That makes sense.
If I reinstall app-attack-detector from the Marketplace will it play nice with the existing fail2ban installation.
I assume that it won't over write the .local files for jail and iptables-common.local ?

Given what you needed to do to get fail2ban ie. create the 30-fail2ban file, shouldn't this solution be updated in COS 7.4? The Attack Detector is a Marketplace app in the end.

I suspect the final solution will be to reinstall app-attack-detector. netify-fwa is playing round with the firewall. If it restarts the firewall your f2b chains get wiped. What I used to do was have a file, /etc/clearos/firewall.d/30-fail2ban, and in it I put:

[ -e /var/run/fail2ban/fail2ban.pid ] && systemctl reload fail2ban.service &

This forces a fail2ban reload every time f2b reloads. app-attack-detector does something similar but cleverer for ipset jails.

What your output is showing is that a lot of the iptables firewall set up is not there as f2b shuts down. To be fair it always generates errors as you can't easily test the existence of a chain, but you can test the existence of firewall rules.

An update.

This appears to be somehow related to a problem I'm having with netify-fwa. Link to other discussion.
After watching fail2ban for a few days I know that it is running properly now. I think I could probably install the Attack Detector again.

What I have noticed from the logs is that fail2ban is identifying issues but it isn't blocking or dropping anything. I adjusted the settings in the jail so that some of the repeat offenders would get blocked but nothing happens. Interestingly the repeat offenders come back just over 30 min. The default jail time is 30 min. So I've increased the time and I get the ban statement but nothing is showing up in the iptables.

Can you check the following log entries and confirm if what I think is happening is actually happening? What I think is that fail2ban thinks there should be some banned IPs so when it goes through a shut down it is trying identify the ban/rejected IPs and finds none. So it throws an error. This shows only the two jails that are running.

2018-01-31 09:22:54,731 fail2ban.server [12324]: INFO Stopping all jails

2018-01-31 09:22:55,491 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- stdout: ''

2018-01-31 09:22:55,491 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- stderr: "iptables v1.4.21: Couldn't load target `f2b-cyrus-imap':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\niptables: No chain/target/match by that name.\niptables: No chain/target/match by that name.\n"

2018-01-31 09:22:55,491 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- returned 1

2018-01-31 09:22:55,492 fail2ban.actions [12324]: ERROR Failed to stop jail 'cyrus-imap' action 'iptables-multiport': Error stopping action

2018-01-31 09:22:55,492 fail2ban.jail [12324]: INFO Jail 'cyrus-imap' stopped

2018-01-31 09:22:56,506 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- stdout: ''

2018-01-31 09:22:56,506 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- stderr: "iptables v1.4.21: Couldn't load target `f2b-postfix-sasl':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\niptables: No chain/target/match by that name.\niptables: No chain/target/match by that name.\n"

2018-01-31 09:22:56,507 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- returned 1

2018-01-31 09:22:56,507 fail2ban.actions [12324]: ERROR Failed to stop jail 'postfix-sasl' action 'iptables-multiport': Error stopping action

2018-01-31 09:22:56,507 fail2ban.jail [12324]: INFO Jail 'postfix-sasl' stopped

2018-01-31 09:22:56,509 fail2ban.server [12324]: INFO Exiting Fail2ban

Tracing the rules back, you need to find where blocktype is defined. It looks like mine is in /etc/fail2ban/action.d/iptables-blocktype.local but that may have been my addition. /etc/fail2ban/action.d/iptables-common.conf indicates iptables-blocktype.local is obsolete but in it I have a section which says:

[INCLUDES]



after = iptables-blocktype.local

        iptables-common.local

# iptables-blocktype.local is obsolete

In iptables-blocktype.local I have:

# Fail2Ban configuration file

#

# Author: Daniel Black

#

# This is a included configuration file and includes the defination for the blocktype

# used in all iptables based actions by default.

#

# The user can override the default in iptables-blocktype.local

[Init]

blocktype = DROP

This suggest to me it may be my override.

I can't reliably comment on systemd and journalmatch as it takes us back to where we were at the start. All you can do is try something and see if it works. Mine does with the default configs.

Nick Howitt wrote:
Personally I'd change the default rule from REJECT to DROP and remove the reject-with icmp-port-unreachable bit. If I am blocking them, I don't see the point in telling them. I am almost tempted to blackhole them!

Thanks again, Nick.
I didn't set the iptable command. This was created by default fail2ban or the banaction, I think.

Your comment makes total sense. I'm not sure how to go about changing the default rule to DROP and remove the icmp-port-unreachable.

I've been reading about this jailmatch in jail.conf, journalctl man page and systemd.journal-fields man page. It would appear there is a problem with using fail2ban and systemd journals if the filter doesn't have a journalmatch definition. I can't figure out this journal match definition so I've change the following in my jail.local

[postfix-sasl]

enabled = true

port     = smtp,465,submission,imap3,imaps,pop3,pop3s

logpath  = /var/log/maillog

backend  = polling

journalmatch =

bantime  = 43200

findtime = 1800

maxretry = 3



[cyrus-imap]

enabled = true

port   = imap3,imaps

logpath = /var/log/maillog

backend = polling

journalmatch =

bantime  = 43200

findtime = 1800

maxretry = 3

I've directed both to the maillog and the errors are gone.

In the iptables I have this

# iptables -L -n

Chain INPUT (policy DROP)

target     prot opt source               destination         

f2b-cyrus-imap  tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 220,993

f2b-postfix-sasl  tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 25,465,587,220,993,110,995



...



Chain f2b-cyrus-imap (1 references)

target     prot opt source               destination         

RETURN     all  --  0.0.0.0/0            0.0.0.0/0           



Chain f2b-postfix-sasl (1 references)

target     prot opt source               destination         

RETURN     all  --  0.0.0.0/0            0.0.0.0/0         

I'll watch this over the next few days and see what happens.

No it is not blocking everything. The key bit is the "match-set f2b-cyrus-imap src". This tells me that ipset rules are being used. It will block everything from the source address (src) listed in the ipset set f2b-cyrus-imap. I have a little whizz-bang to check these sets. Put the following in a file, make it executable and run it:

for SET in `ipset list -name | grep f2b` ; do

	ipset list $SET -output save | grep add | awk '{print $2 " "  $3}'

done

Personally I'd change the default rule from REJECT to DROP and remove the reject-with icmp-port-unreachable bit. If I am blocking them, I don't see the point in telling them. I am almost tempted to blackhole them!

Nick Howitt wrote:

There is a fail2ban-systemd in epel-unverified. I have not researched it but I'm not sure why you want it.

I thought it was necessary to get fail2ban to be able to be enabled by systemctl. After reading a bit more it looks like fail2ban > 0.9 is systemctl enabled.

Anyway, I was able to get it going. I'm still getting the following notice for cyrus-imap and proftpd

fail2ban.filtersystemd [16999]: NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.

For postfix-sasl it is OK.

fail2ban.jail           [8727]: INFO    Creating new jail 'postfix-sasl'

fail2ban.jail           [8727]: INFO    Jail 'postfix-sasl' uses systemd

fail2ban.jail           [8727]: INFO    Initiated 'systemd' backend

fail2ban.filter         [8727]: INFO    Set maxRetry = 5

fail2ban.actions        [8727]: INFO    Set banTime = 43200

fail2ban.filter         [8727]: INFO    Set findtime = 1800

fail2ban.filtersystemd  [8727]: INFO    Added journal match for: '_SYSTEMD_UNIT=postfix.service'

I see in iptables

REJECT     tcp  --  anywhere             anywhere             multiport dports imap3,imaps match-set f2b-cyrus-imap src reject-with icmp-port-unreachable

REJECT     tcp  --  anywhere             anywhere             multiport dports smtp,urd,submission,imap3,imaps,pop3,pop3s match-set f2b-postfix-sasl src reject-with icmp-port-unreachable

REJECT     tcp  --  anywhere             anywhere             multiport dports ftp,ftp-data,ftps,ftps-data match-set f2b-proftpd src reject-with icmp-port-unreachable

Isn't this blocking everything?

There is a fail2ban-systemd in epel-unverified. I have not researched it but I'm not sure why you want it.

Nick Howitt wrote:

Posts crossed. Delete everything in /etc/fail2ban except /etc/fail2ban/jail.d then do a:

yum install fail2ban-server

Done.
Is there nothing like

fail2ban-systemd

available?

Posts crossed. Delete everything in /etc/fail2ban except /etc/fail2ban/jail.d then do a:

yum install fail2ban-server

Be careful if you go it alone. Files in /etc/fail2ban/jail.d are added by their apps and not by app-attack-detector. The other thing which you'll need to get round is that a firewall restart will wipe the f2b firewall rules so you may want to drop a configlet into /etc/clearos/firewall.d to restart f2b if it is running. That will add the rules again. You can make it run asynchronously by adding a "&" at the end of the restart command.

nuke wrote:
I'm going to uninstall again and see if I can get fail2ban to run by itself without the Attack Detector envelope.

Which doesn't work.

yum install fail2ban fail2ban-systemd

Loaded plugins: clearcenter-marketplace, fastestmirror

ClearCenter Marketplace: fetching repositories...

Loading mirror speeds from cached hostfile

 * clearos: mirror2-newyork.clearos.com

 * clearos-centos: download4.clearsdn.com

 * clearos-centos-sclo-rh: download4.clearsdn.com

 * clearos-centos-updates: download4.clearsdn.com

 * clearos-contribs: mirror2-newyork.clearos.com

 * clearos-epel: download4.clearsdn.com

 * clearos-fast-updates: download4.clearsdn.com

 * clearos-infra: mirror2-newyork.clearos.com

 * clearos-updates: mirror2-newyork.clearos.com

 * private-clearcenter-dnsthingy: download4.clearsdn.com:80

 * private-clearcenter-dyndns: download4.clearsdn.com:80

 * private-clearcenter-plex: download4.clearsdn.com:80

No package fail2ban available.

No package fail2ban-systemd available.

Error: Nothing to do

Grrrrr.

It installed as part of the Attack Detector, so why can't yum find it now? Do I have to manually download the RPMs or add another repository?

Nick Howitt wrote:
The jail takes the definition from jail.conf initially then overrides with the settings in jail.d/clearos-postfix-sasl.conf. It looks like you still have the default settings. Does it keep running like this or is it still failing?

I haven't had any time to look at this for a few days. This morning I got another email warning

/etc/cron.daily/logrotate:



ERROR  Failed to access socket path: /var/run/fail2ban/fail2ban.sock. Is fail2ban running?

and when I checked, Fail2ban has stopped again and it looks like it's stuck trying to restart.

This is frustrating.

I'm going to uninstall again and see if I can get fail2ban to run by itself without the Attack Detector envelope.

The jail takes the definition from jail.conf initially then overrides with the settings in jail.d/clearos-postfix-sasl.conf. It looks like you still have the default settings. Does it keep running like this or is it still failing?

If you wanted to test, you could try overriding the findtime for the jail e.g by adding a file /etc/fail2ban/jail.local and putting in it:

[postfix-sasl]

findtime = 86400

maxretry = 2

It is a bit aggressive, but if you have set up your mobile devices, they should never trip it so, only bad attempts will be logged.

As a lateral jump, do you have postfix authentication on? I would turn it off and open incoming port tcp:465. ClearOS will still allow authentication on SMTPS/port 465. Then you switch all mobile devices to SMTPS. Fixed LAN devices can also be switched or, if you're a little lazy like me, you could add your LAN to the trusted networks so no authentication is needed on SMTP/port 25.

Nick Howitt wrote:
What does your [postfix-sasl] jail look like?

Thanks again Nick.

I'm not sure I understand correctly. Is the [postfix-sasl] jail what is written in the jail.conf file & clear postfix conf file?

The info from the jail.conf is default. I wasn't going to change anything until I got it working right.

findtime = 600
bantime = 600
maxretry = 5

[postfix-sasl]



port     = smtp,465,submission,imap3,imaps,pop3,pop3s

# You might consider monitoring /var/log/mail.warn instead if you are

# running postfix since it would provide the same log lines at the

# "warn" level but overall at the smaller filesize.

logpath  = %(postfix_log)s

backend  = %(postfix_backend)s

and the jail.d/clearos-postfix-sasl.conf

# This file is controlled by the ClearOS API, please do not edit!

# If you would like to customize parameters, add a new configlet file.

[postfix-sasl]

enabled = true

bantime  = 86400

Eventually, I want to get some of the jails for Apache going.

"Found" means one of your jails has found a single match. You then need to look in your jails to see how many messages need to be found (maxretry) in what period (findtime). Note that the default findtime is 600 seconds and the default maxretry is 5 so you need 5 "Found" messages in 5 minutes in a jail before there is a ban. All your "Found" entries are well spaced. The Attack Detector app logs only show the bans. If you go to the log viewer Webconfig > Reports > Performance and Resources > Log Viewer you can view the full fail2ban log. You can then filter it for "Ban" (capital B or you get everything) to see your bans. Or do as you did and view the file in /var/log/fail2ban.

What does your [postfix-sasl] jail look like?

The webconfig log shows nothing even when there are items in the fail2ban.log file.

Nick Howitt wrote:

I don't particularly want to try it for the moment, but I don't believe the webconfig will uninstall the fail2ban-server component which is the one I was trying to refresh. I could be wrong, but have a look in your yum log

Nick, the webconfig does uninstall all three files and reinstalls all three again. From my yum log:

--------------------- yum Begin ------------------------ 

 Packages Installed:

    1:app-attack-detector-core-2.3.1-1.v7.noarch

    1:app-attack-detector-2.3.1-1.v7.noarch

    fail2ban-server-0.9.5-3.el7.noarch

 

 Packages Updated:



 ....



 Packages Erased:

    1:app-attack-detector-core-2.3.1-1.v7.noarch

    1:app-attack-detector-2.3.1-1.v7.noarch

    fail2ban-server-0.9.5-3.el7.noarch

 ---------------------- yum End ------------------------- 

It appears to be running, ie. service status is green, but I'm still seeing the journal match warning.

I don't think it is actually blocking anything. If I remember the last time I dug into fail2ban, the log would show that it found an issue, that it had been blocked and after some period of time, that it was released again. The log only show that it found an issue. The logging is the same level of detail as the COS5.2 install.

Here is the log from the start up to now. At the bottom you can see postfix-sasl and cyrus-imap have "Found" issues but no action occurs. Am I reading this correctly?

2018-01-22 15:01:11,099 fail2ban.server [17582]: INFO Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.5

2018-01-22 15:01:11,099 fail2ban.database [17582]: INFO Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'

2018-01-22 15:01:11,101 fail2ban.jail [17582]: INFO Creating new jail 'sshd'

2018-01-22 15:01:11,120 fail2ban.jail [17582]: INFO Jail 'sshd' uses systemd

2018-01-22 15:01:11,142 fail2ban.jail [17582]: INFO Initiated 'systemd' backend

2018-01-22 15:01:11,143 fail2ban.filter [17582]: INFO Set maxRetry = 5

2018-01-22 15:01:11,144 fail2ban.actions [17582]: INFO Set banTime = 86400

2018-01-22 15:01:11,145 fail2ban.filter [17582]: INFO Set findtime = 600

2018-01-22 15:01:11,145 fail2ban.filter [17582]: INFO Set maxlines = 10

2018-01-22 15:01:11,223 fail2ban.filtersystemd [17582]: INFO Added journal match for: '_SYSTEMD_UNIT=sshd.service + _COMM=sshd'

2018-01-22 15:01:11,230 fail2ban.jail [17582]: INFO Creating new jail 'sshd-ddos'

2018-01-22 15:01:11,230 fail2ban.jail [17582]: INFO Jail 'sshd-ddos' uses systemd

2018-01-22 15:01:11,231 fail2ban.jail [17582]: INFO Initiated 'systemd' backend

2018-01-22 15:01:11,232 fail2ban.filter [17582]: INFO Set maxRetry = 5

2018-01-22 15:01:11,233 fail2ban.actions [17582]: INFO Set banTime = 86400

2018-01-22 15:01:11,233 fail2ban.filter [17582]: INFO Set findtime = 600

2018-01-22 15:01:11,235 fail2ban.filtersystemd [17582]: INFO Added journal match for: '_SYSTEMD_UNIT=sshd.service + _COMM=sshd'

2018-01-22 15:01:11,242 fail2ban.jail [17582]: INFO Creating new jail 'proftpd'

2018-01-22 15:01:11,242 fail2ban.jail [17582]: INFO Jail 'proftpd' uses systemd

2018-01-22 15:01:11,243 fail2ban.jail [17582]: INFO Initiated 'systemd' backend

2018-01-22 15:01:11,243 fail2ban.filter [17582]: INFO Set maxRetry = 5

2018-01-22 15:01:11,244 fail2ban.actions [17582]: INFO Set banTime = 86400

2018-01-22 15:01:11,244 fail2ban.filter [17582]: INFO Set findtime = 600

2018-01-22 15:01:11,262 fail2ban.jail [17582]: INFO Creating new jail 'postfix-sasl'

2018-01-22 15:01:11,262 fail2ban.jail [17582]: INFO Jail 'postfix-sasl' uses systemd

2018-01-22 15:01:11,263 fail2ban.jail [17582]: INFO Initiated 'systemd' backend

2018-01-22 15:01:11,263 fail2ban.filter [17582]: INFO Set maxRetry = 5

2018-01-22 15:01:11,264 fail2ban.actions [17582]: INFO Set banTime = 86400

2018-01-22 15:01:11,264 fail2ban.filter [17582]: INFO Set findtime = 600

2018-01-22 15:01:11,268 fail2ban.filtersystemd [17582]: INFO Added journal match for: '_SYSTEMD_UNIT=postfix.service'

2018-01-22 15:01:11,274 fail2ban.jail [17582]: INFO Creating new jail 'cyrus-imap'

2018-01-22 15:01:11,274 fail2ban.jail [17582]: INFO Jail 'cyrus-imap' uses systemd

2018-01-22 15:01:11,275 fail2ban.jail [17582]: INFO Initiated 'systemd' backend

2018-01-22 15:01:11,275 fail2ban.filter [17582]: INFO Set maxRetry = 5

2018-01-22 15:01:11,276 fail2ban.actions [17582]: INFO Set banTime = 86400

2018-01-22 15:01:11,276 fail2ban.filter [17582]: INFO Set findtime = 600

2018-01-22 15:01:11,287 fail2ban.jail [17582]: INFO Jail 'sshd' started

2018-01-22 15:01:11,290 fail2ban.jail [17582]: INFO Jail 'sshd-ddos' started

2018-01-22 15:01:11,291 fail2ban.filtersystemd [17582]: NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.

2018-01-22 15:01:11,292 fail2ban.jail [17582]: INFO Jail 'proftpd' started

2018-01-22 15:01:11,295 fail2ban.jail [17582]: INFO Jail 'postfix-sasl' started

2018-01-22 15:01:11,301 fail2ban.filtersystemd [17582]: NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.

2018-01-22 15:01:11,335 fail2ban.jail [17582]: INFO Jail 'cyrus-imap' started

2018-01-22 15:35:57,214 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.140.12.121

2018-01-22 16:21:24,571 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.222.209.14

2018-01-22 17:10:05,104 fail2ban.filter [17582]: INFO [cyrus-imap] Found 60.166.12.117

2018-01-22 19:29:14,303 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.222.209.14

2018-01-22 20:19:37,799 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.140.12.121

2018-01-22 20:22:41,692 fail2ban.filter [17582]: INFO [cyrus-imap] Found 116.248.41.55

2018-01-22 21:12:16,228 fail2ban.filter [17582]: INFO [cyrus-imap] Found 61.166.110.62

2018-01-22 22:19:46,027 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.222.209.14

2018-01-23 01:06:43,758 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.140.12.121

2018-01-23 01:12:02,954 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.222.209.14

2018-01-23 03:03:01,138 fail2ban.filter [17582]: INFO [cyrus-imap] Found 60.223.252.6

2018-01-23 05:51:19,053 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.140.12.121

I don't particularly want to try it for the moment, but I don't believe the webconfig will uninstall the fail2ban-server component which is the one I was trying to refresh. I could be wrong, but have a look in your yum log

I guess I'm an impatient guy. I just left it race the FF interface and came back a while later and found it was uninstalled.
So have installed it again (forgot to delete the /etc/fail2ban folder ) ) and it looks like it works with all the jails.
In the Services list it shows the green light.
So I'm left scratching my head as to what caused all this time suck.
Thanks again for your help, Nick. I appreciate all your patience with helping me through this.

Removing from the Webconfig will only remove the app- packages and not fail2ban, I believe. Instead do a:

yum remove fail2ban-server

Abort it if it tries to remove more than fail2ban-server and app-attack*.

Be very careful if you ever use yum to remove packages. It can wreck your system if it removes too many dependencies (don't ever try "yum remove gcc*").

Nick Howitt wrote:

You could possibly uninstall fail2ban-server and then delete everything in /etc/fail2ban except for anything in /etc/fail2ban/jail.d then reinstall it (make a backup of /etc/fail2ban first).

I just tried to uninstall and it hangs the GUI using FF and Chrome. No idea if it has removed app-attack-detector; app-attack-detector-core; and fail2ban-server. Maybe a manual uninstall is required?

Can't check this manually as the webGUI uninstall is locking yum.

Another app is currently holding the yum lock; waiting for it to exit...

[edit]added new info[/edit]