Hi guys, I have ClearOS 5.1 and have the Intrusion detection/prevention modules up and running and a ClearSDN subscription for the intrusion detection updates. Here is my problem:
I have a Windows server 2003 machine (internal website server, library card catalog, and a few other small services running) that is being attacked. The security log in event viewer shows hundreds of "Failure Audit" type 529 errors (login type 10, which is remote/terminal services I believe.) These go on for 2 or 3 hours, usually in the early hours of the morning. The login attempts occur every 3 seconds for these 2-3 hrs. They try hacking the Adminstrator account mostly (which i've disabled) but also try other common accounts "helpdesk" "admin" etc.
These all come from external IP addresses from all across the world. I've been putting them into the Block Incoming Firewall portion of clearos, but they change all the time. One IP will try for 2 hrs, then the next day it's a new IP. I've put in 10-15 addresses so far to block, but this seems useless. They are obviously using proxy servers or something to route their hacking through, so I'll never be able to stop them like this - they'll just keep changing IPs. They haven't broken in yet, and probably won't - there's only 5 accounts on the machine and they are not generic sounding - but the brute force of the attacks alone is putting a heavy burden on the server and causing it to lock up occasionally.
Is there a way in ClearOS to stop this?? Why isn't the intrusion prevention picking up on this? I would think it would look like suspicious behavior that an IP would try every 3 seconds for 3 hrs to get into my network.
Any suggestions?
thanks!
I have a Windows server 2003 machine (internal website server, library card catalog, and a few other small services running) that is being attacked. The security log in event viewer shows hundreds of "Failure Audit" type 529 errors (login type 10, which is remote/terminal services I believe.) These go on for 2 or 3 hours, usually in the early hours of the morning. The login attempts occur every 3 seconds for these 2-3 hrs. They try hacking the Adminstrator account mostly (which i've disabled) but also try other common accounts "helpdesk" "admin" etc.
These all come from external IP addresses from all across the world. I've been putting them into the Block Incoming Firewall portion of clearos, but they change all the time. One IP will try for 2 hrs, then the next day it's a new IP. I've put in 10-15 addresses so far to block, but this seems useless. They are obviously using proxy servers or something to route their hacking through, so I'll never be able to stop them like this - they'll just keep changing IPs. They haven't broken in yet, and probably won't - there's only 5 accounts on the machine and they are not generic sounding - but the brute force of the attacks alone is putting a heavy burden on the server and causing it to lock up occasionally.
Is there a way in ClearOS to stop this?? Why isn't the intrusion prevention picking up on this? I would think it would look like suspicious behavior that an IP would try every 3 seconds for 3 hrs to get into my network.
Any suggestions?
thanks!
Share this post:
Responses (43)
-
Accepted Answer
-
Accepted Answer
What services are you exposing on your Windows Server 2003 machine which are open to authentication? your website?
Are you port forwarding specific services? or using 1 to 1 nat? Have your forwarded the remote / terminal services port?
If you don't need them remotely then first thing is to close the ports.
ClearOS has brute force scripts for things like SMTP, POP, SSH but will monitor only on specific ports. Thats why we need to understand what these authentication requests are, and for what service. Once you have that, with a bit of tcpdump detection work a small rule can be created in Snort (ClearOS IDS) to detect them... -
Accepted Answer
Can you give the sid of the rule or the exact error message as it appears in /var/log.secure. Munge your own IP but leave it clear which is yours and which is the attacking one.
Nick Howitt
That's the thing - ClearOS isn't catching it, so there's no SID.
What services are you exposing on your Windows Server 2003 machine which are open to authentication? your website?
Are you port forwarding specific services? or using 1 to 1 nat? Have your forwarded the remote / terminal services port?
If you don't need them remotely then first thing is to close the ports.
ClearOS has brute force scripts for things like SMTP, POP, SSH but will monitor only on specific ports. Thats why we need to understand what these authentication requests are, and for what service. Once you have that, with a bit of tcpdump detection work a small rule can be created in Snort (ClearOS IDS) to detect them...
We do have some internal website stuff that requires a single login, and we have RDP/terminal services enabled so I can check on the server occasionally from home (not a huge deal.)
We are just using 1 to 1 nat on the clearos box - no port forwarding on that server.
I can give you all the info from the security log on the 2003 server, but the high points were:
Event ID 529
Type: Failure Audit
Reason: Unknown username or bad password
Logon Type: 10 (i think this means remote/terminal services)
Logon process: User32
Source address: (IP address varies, I've listed a few below)
Source port: 55793 (these vary wildly, but usually between 30000-60000)
Here's a few of the IPs trying to hack in:
151.196.62.219
205.119.111.46
212.0.212.219
212.7.208.81
217.148.161.162
66.193.240.40
78.90.116.86
79.28.30.120
83.243.220.136
88.247.38.153
89.2.117.230
93.152.151.75
95.143.198.129
Probably doesn't help - but maybe someone will recognize one on a web search one day and run a virus scan on their zombie server! -
Accepted Answer
-
Accepted Answer
-
Accepted Answer
Bob Stangarone wrote:
If you know what IPs you use for RDP , it is possible to create custom firewall rules that only allow RDP from specific external addresses...
Would I create these rules on the clearos box or the server itself? If it is on the clearos machine, will it still work even though I have the server in 1-to-1 NAT?
thanks for all the help so far! -
Accepted Answer
Hi Blake,
You can setup a 1-to-1 nat rule or Advance rule too.
for example, if you are just working on the server from specific IPs you create a rule that will open and port forward to the Windows server behind the wall. IT will only open the ports if the Source IP match what you place in there.
I have something like this setup when I need to access ClearOS from client's locations.
But if you have Remote Users that travel this will not work for you.
You could also try setting up different Ports and have those ports forwarded? -
Accepted Answer
Kevin Dika wrote:
Hi Blake,
You can setup a 1-to-1 nat rule or Advance rule too.
for example, if you are just working on the server from specific IPs you create a rule that will open and port forward to the Windows server behind the wall. IT will only open the ports if the Source IP match what you place in there.
I have something like this setup when I need to access ClearOS from client's locations.
But if you have Remote Users that travel this will not work for you.
You could also try setting up different Ports and have those ports forwarded?
Thanks for the info! How do I go about setting up a rule for the 1-to-1 Nat server? We don't have any traveling users that would need to RDP - just me from my house occasionally. Would the rule effect just the server in question, or all the other servers I have set up in 1-to-1 NAT?
Lastly, would it be easier/better to do this or figure out how to tell ClearOS to ban an IP that tries to brute force login to the server (if that's even possible)? -
Accepted Answer
-
Accepted Answer
Nick Howitt wrote:
If it is just you doing RDP would it not be easier to set up OpenVPN. Then you can close the WAN RDP port entirely. If you set up OpenVPN you can address your servers directly using their LAN IP addresses.
I just looked at OpenVPN briefly, but not sure it's what I need exactly. It seemed to create a whole new network connection on my computer and I'm afraid of installing something like that on the server, if I can keep from it. I would love to be able to try to block these attacks through our ClearOS gateway/firewall if it is possible. One reason is, I'm guessing even if I disable RDP on the server - isn't the attacker still going to keep trying to brute force it's way in, regardless of whether RDP is enabled or not, the server still has to say "NO" 3,600 times right? - which causes the server to be bogged down. If i could kill the request at the gateway and ban the IP - that would save three hours of constant logon requests to the server. -
Accepted Answer
I don't have alot of experience with OpenVPN.
and I'm sure as Nick suggested you can do both options really. As which one is more secure... I'm not sure.
Under the menu NETWORK, then FIREWALL >> ADVANCE. You just enter the source IP, plus the Destination Port and away you go.
I'm sure once the port is close that the hackers will stop trying that port, but if you other ports they try again. Maybe suggest changing IP address from the ISP.
To check which ports are open you can use the GRC.com tester. Doesn't search all ports just commonly used one. Or you can try mxtoolbox.com too.
Here are some docs on how to setup the advance firewall.
http://www.clearcenter.com/support/documentation/clearos_enterprise_5.2/user_guide/advanced_firewall
But I think Nick is right too, if you setup the VPN to ClearOS, you should be able to access the network behind ClearOS once you log into the VPN and then log in from there.
Either way I hope that helps. -
Accepted Answer
Awesome, thanks! I didn't have the 'Advanced Firewall' Module downloaded, but I just got it installed.
So, are you saying that if I just leave everything as is on the server (not disabling RDP, not changing RDP's port, etc) and set an advanced rule to allow access from certain IPs to port 3389 (RDP) - then it will automatically block all others? Even if the server is set up in 1-to-1 Nat?
thanks so much everyone for their help, I think I'm close to working this out. -
Accepted Answer
For OpenVPN you would not install it on your server, just the ClearOS gateway and your home PC and it is the most secure option.
If you have a fixed IP at home, do as Kevin suggested and only forward port 3389 from the one WAN IP address in the ClearOS to your server. The server will not see any connection attempts from anywhere else on the internet. They will be blocked by the ClearOS firewall. If you have a dynamic IP then OpenVPN is the way to go and you can completely close port 3389.
The next line of defence is snort. There appears to be one RDP rule in the Miscellaneous rule set. Do you have that rule set enabled? If you use the Emerging Threat rules (search the forum for Emerging Threat) there are three more RDP rules but none of them will trigger a firewall block. If you read the Emerging Threat thread on the forum you will see there is a way of adding firewall block rules to any rule.
If Tim pops back into the thread, with a bit of packet sniffing, he could help you set up a custom rule to trigger a firewall block.
[edit]
I was still typing when you did your last post.
I am not sure about 1:1 NAT as I don't use it but there must be a way to block 3389. Do you need 1:1 NAT or can you just port forward http and https? Alternatively you may have to craft a negative firewall rule so you block all 3389 except from your allowed address. That may work along side the 1:1 NAT.
[/edit] -
Accepted Answer
I do have the Misc rule checked in Intrusion Detection. I'm assuming I need 1:1 Nat - I'm not sure how else to map a public/external IP to the server's local IP (192.168.x.x). I looked at the port forwarding and advanced port forwarding, but didn't see any way of saying this external IP goes to this local IP. Maybe I'm missing something. I just thought if I used port forwarding, I would have to go to the ClearOS IP : (whatever port) - instead of people just being able to go to www.mywebsite.com. -
Accepted Answer
Hi Blake,
By your response, maybe I misunderstood what you are trying to do.
I thought you said that you are the only one using RDP? If you aren't, then yes, user will have to go enter something like this:
ort" target="_blank">https://clearOSip.comort
But I understood you differently. You mentioned that you would like access to the server from home . Than enter the IP of your home in the source field, the destination IP address 192.168.x.x (ServerIP) and port 3389. This when when you are home, and you enter http://mywebsite.com or IP address of the ClearOS in the RDP, it will open the port to access the server. All other IPs, it will appear that the port doesn't exist. Now unless someone hacks your home computer than they will have access to the server.
If you want user to be able to connect to LAN behind ClearOS, then VPN is the only choice, and you'll have to configure it in a way to do so. But if you want to access the server via RDP only, then the Advance Firewall (or Port Forwarding option) is the one way...
If you are using the Windows Server as the main computer for user to work on (As a Terminal Server etc) there are a few ideas that come to mind, but that is another subject. -
Accepted Answer
Kevin, yes I will be the only one RDPing into the server. I was just responding to the suggestion that I take the server off of 1:1 nat and just port forward. If I did that, how would people visit the website that runs on the server? The only connection that ties the external IP/dns name/website name to the servers internal address is the 1:1 nat module. We have 4 servers (mail, website, help desk, etc) with external (and internal) IPs that are in the 1:1 nat page. If I take them out, they are no longer reachable to those outside our network. Some have suggested using port forwarding instead of 1:1 nat. I guess I don't understand how I would do that, without contacting our dns hosting company and having them point mywebsite.com to clearosip:80 instead of 1.2.3.4. Which would be a huge hassle I think, considering all the ports.
This is all moot however if the firewall rules and intrusion prevention work regardless of the server being in 1:1 nat. From reading the clearos help page on 1:1 nat, it seems like they are not protected by the firewall, and thus any rule I put in wouldn't work for the servers listed in 1:1 nat. Hopefully I'm wrong. -
Accepted Answer
Yeah Port Forwarding would be a clear options, but the 1 to 1 NAT'ing was designed to do what you are looking for, or at least what I "assumed" it should do.
Not sure what else I can help with, as it sounds like you need to provide a topology of your network to ensure that everything is covered and the min. amount of ports are opened etc. This doesn't sound like a simply setup... which makes it harder to explain and figure out via forms. -
Accepted Answer
The 1-to-1NAT documentation suggests you can 1-1 NAT with just selective ports. Perhaps this is what you want for your server. Port forwarding would also work (and in effect may be the same) but you may have to craft your rule carefully so it is probably better to stay with 1-1 NAT but with selective ports only. -
Accepted Answer
Nick Howitt wrote:
The 1-to-1NAT documentation suggests you can 1-1 NAT with just selective ports. Perhaps this is what you want for your server. Port forwarding would also work (and in effect may be the same) but you may have to craft your rule carefully so it is probably better to stay with 1-1 NAT but with selective ports only.
You're probably right. I probably need to do 1-1 NAT with selective ports. If I do that, and then do an advanced firewall rule specifying my home address as the only address to allow in 3389 (RDP) - do you think that would work?
It would still be cool to figure out a way to stop the constant querying of the server by snort. -
Accepted Answer
Just to confirm that 1-to-1 NAT creates a DNAT and SNAT rule that passes ALL traffic destined to that public IP to your private IP. This is very similar to a DMZ setup
If you wish to restrict it - use the 1-to-1 port mappings as suggested above for the services you wish to expose. If it was me I'd leave the RDP off this for the time being then create an advanced rule to permit only specific remote IP's.
If you wish to keep your current config - consider changing your Windows firewall to restrict port 3389 to only one IP. -
Accepted Answer
Not sure if I should post this here or start a new thread. I came here looking for this exact same thing, except that RDP will be coming from many users, most with dynamic IPs, so the solutions posted so far won't work for me. I floated the idea of a VPN but was shot down so now I'm wondering, is it possible to create a snort rule that will blacklist an IP if opens a connection to a port a certain number of times in a given time period? For instance, if an IP opens a connection on port 3389 5 times in a minute then it's blacklisted. I think something like that would be ideal. -
Accepted Answer
Fail2ban would give you that functionality. It can be found here:
http://www.fail2ban.org/wiki/index.php/Main_Page -
Accepted Answer
-
Accepted Answer
There is an easy way.
I've attached a fail2ban RPM that the good folks over at PBX in a Flash built for their PBX distribution which runs on CentOS 5. It works perfectly with ClearOS.
Just do the following:
1. Copy the RPM to your ClearOS box.
Access your ClearOS box via SSH and run the following commands as the root user:
2. rpm -ivh fail2ban-0.6.1-2jik.noarch.rpm
3. /etc/init.d/fail2ban start
That's it!
The other fail2ban RPMs on the Internet typically require Shorewall, the frontend application to IPTables, and you don't want to use that because it would probably interfere with the way ClearCenter does things with IPTables. I could be wrong but I'm just trying to keep it simple.
[file name=fail2ban_0.zip size=58320]http://www.clearfoundation.com/media/kunena/attachments/legacy/files/fail2ban_0.zip[/file] -
Accepted Answer
-
Accepted Answer
After you unzip the file and install it on your ClearOS Box, edit the /etc/fail2ban.conf file and look for the SSH section. Change the port from "ssh" to whatever your port number is. Here's a copy of the relevant part of the SSH section from the conf file:
[SSH]
# Option: enabled
# Notes.: enable monitoring for this section.
# Values: [true | false] Default: true
#
enabled = true
# Option: logfile
# Notes.: logfile to monitor.
# Values: FILE Default: /var/log/secure
#
logfile = /var/log/secure
# Option: port
# Notes.: specifies port to monitor
# Values: [ NUM | STRING ] Default:
#
port = ssh <------ YOU NEED TO CHANGE THIS TO YOUR CORRECT PORT NUMBER
After you do that, start the fail2ban service. -
Accepted Answer
-
Accepted Answer
That's awesome Alex, thank you. Fail2ban says it monitors logs for password attempts. Are there logs generated on ClearOS that it would be able to monitor to recognize password attempts on a server behind the firewall?
Also, thanks for the heads-up on the PBX in a Flash project, that looks like something I would be interested in as well. -
Accepted Answer
MBP, you're very welcome. I'm glad I could help.
The only way a ClearOS firewall running fail2ban would be able to monitor password attempts in the logs of another server is if the ClearOS firewall was also configured to act as a syslog server. The server behind the firewall would then need to be configured to send its logs to the ClearOS syslog server. -
Accepted Answer
Is there a way to monitor the logs for connection attempts on a specific port? For instance, if I have port 5000 forwarded through the firewall will ClearOS log that there have been 20 connections to port 5000 in the last 30 seconds (example) which would then be monitored by Fail2ban and blocked? -
Accepted Answer
There is a way to configure IPTables to write information to the system log but you need a way to examine the actual traffic to determine if the packets to a port are just a connection attempt or part of an established session. I think the easiest way to accomplish what you want is to configure a Snort (IDS) rule to look for packets that contained information on failed login attempts. However, since SSH traffic is encrypted you won't be able to examine the packets so you might want to look for a lot of reset (RST) packets. I haven't manually configured Snort in a very long time and I'm no expert on Snort rules but I think there might be a way to achieve your goal with Snort and Snortsam, the tool that automatically adjusts the firewall rules to block unwanted traffic. Both Snort and Snortsam are already built-in to ClearOS, in case you didn't know. -
Accepted Answer
Bingo!
I just a quick search and found a specific example on how to use Snort to achieve exactly what you want. The following example was taken from http://www.devx.com/security/Article/22476/1954
-----
Put your ssh.rules file at /home/white/snort/ssh.rules, and add its include option to Snort's configuration file at /usr/local/etc/snort.conf:
include /home/white/snort/ssh.rules
The following is the rule set itself (/home/white/snort/ssh.rules):
alert tcp any any -> $HOME_NET 22 ( \
msg:"Potential SSH Brute Force Attack"; \
flow:to_server; \
flags:S; \
threshold:type threshold, track by_src, count 3, seconds 60; \
classtype:attempted-dos; \
sid:2001219; \
rev:4; \
resp:rst-all; \
)
This rule set file sends TCP_RST packets in both directions when somebody tries to perform more than three connections during a 60-second span.
Moreover, Snort also logs the attempts. You can see them in the /var/log/snort/alert log file:
-----
It appears that you could just modify the above rule to fit your configuration, changing port 22 to port 5000 and changing 60 seconds to 30 seconds, then save the rule to /etc/snort/ssh.rules on the ClearOS gateway. I think you would also need to modify the /etc/snort.conf file and add the following line to the end of the file and then restart the IDS (snort) service.
include $RULE_PATH/ssh.rules
I hope that gives you a good start.
By the way, fail2ban could be configured to read the source IP information logged by Snort in order to permanently ban those IP addresses. I think Snortsam is only configured to block the IP addresses for 24 hours but I'm pretty sure that could modified as well. -
Accepted Answer
-
Accepted Answer
Thanks for the advice, Nick.
The only local.rules file I see on my ClearOS box is in /var/lib/suva/services/intrusion-protection/rules/local.rules. Also, I don't see a commented line for local.rules in the /etc/snort.conf file. I know that it can easily be added but I'm curious to know if you added it or if was there by default for you. -
Accepted Answer
I'm lazy. /etc/snort/ symlinks to /var/lib/suva/services/intrusion-protection/rules/. I can remember the short path name and not the long one. The two files are the same.
To be honest I can't remember if I added the line or uncommented it. I thought I uncommented it, but it does not matter. Just make sure it is there (and restart snort). -
Accepted Answer
Hi folks.
Just an FYI : the SSH brute force rule is already included in ClearOS 5.x (telnet.rules). There's also an FTP rule (ftp.rules) but that one cause quite a bit of grief when it was added to the intrusion prevention system, so it's just a passive rule.
Regardless, Snort can't be used to catch all of these types of login shenanigans so implementing the fail2ban is a good extra layer of defense. I don't want to hijack this thread, so I started a new one here. -
Accepted Answer
Hey Peter I don't know if you have seen it but check out this nixcraft article
http://www.cyberciti.biz/tips/linux-unix-bsd-openssh-server-best-practices.html
It would be one of the best articles on locking down ssh I have ever read -
Accepted Answer
Very nice article! Version 6 will add the following security features:
#1: Disable OpenSSH Server (via SSH Server App)
#3: Limit Users' SSH Access (via Shell Extension App)
#5: Disable .rhosts
#7: Disable root login (via SSH Server App)
#11: Use Public Key Based Authentication (via SSH Server App) -
Accepted Answer
They (nixcraft) also have an article about locking down php too
Also I have been looking at this too
http://blog.adslweb.net/serendipity/article/286/CentOS-5-enabling-Two-factor-SSH-authentication-via-Google
Two factor ssh auth via google B) -
Accepted Answer
Hi. Just started reading the various posts on fail2ban.
Alex, where would that RPM be found?
The file you've attached is a zip that contains a number scripts and conf files. If this is the file to install, then your instructions didn't work for me. The attached file is a zip not RPM. What should I be doing with the ZIP?
As I'm a noob, I'm sure I'm missing something or too dense to know.
Thanks
Nuke.
Alex wrote:
There is an easy way.
I've attached a fail2ban RPM that the good folks over at PBX in a Flash built for their PBX distribution which runs on CentOS 5. It works perfectly with ClearOS.
Just do the following:
1. Copy the RPM to your ClearOS box.
Access your ClearOS box via SSH and run the following commands as the root user:
2. rpm -ivh fail2ban-0.6.1-2jik.noarch.rpm
3. /etc/init.d/fail2ban start
That's it!
The other fail2ban RPMs on the Internet typically require Shorewall, the frontend application to IPTables, and you don't want to use that because it would probably interfere with the way ClearCenter does things with IPTables. I could be wrong but I'm just trying to keep it simple.
[file name=fail2ban_0.zip size=58320]http://www.clearfoundation.com/media/kunena/attachments/legacy/files/fail2ban_0.zip[/file]
Please login to post a reply
You will need to be logged in to be able to post a reply. Login using the form on the right or register an account if you are new here.
Register Here »