DOS-Deflate - Mitigate (D)DOS atacks

[elix]

*Note*

The home of this project has been relocated to:

http://deflate.medialayer.com/

Thanks,

[Zaf]

Hi Friends,

a quick update was required of the script. It is now v 0.2

If you already installed the script, just re-install by issuing the following commands:
./uninstall.sh
./install.sh

If you dont have uninstall.sh:
wget http://www.inetbase.com/scripts/ddos/uninstall.sh
chmod 755 uninstall.sh

Changelog:
Changed the method of unbanning for the apf.
Fixed a few bugs

Thanks all

[@how@]

Thanks for the post.



Wael

[elix]

Glad to hear you like the post..

Zaf is currently working on a new version of this that will be *much* better.

[yeshoward]

Hello elix

how to stop system sending cron and ban IP email notice please?

We got thousands a day

thanks

[Zaf]

Open /usr/local/ddos/ddos.sh in your favorite editor (vi or pico) and search for the line which says "mail -s" (that is the command which sends you the mail). Put # at the beginning of that line. Save the file and exit the editor.
Done!

Sidenote: It is too bad if you have to block thousands of ips using this script. Infact, you are calling for trouble by doing so, as too many ips in the block list would make your VPS unstable.

Edit: Too many in my opinion would be anything more than 500.

[elix]

Open /usr/local/ddos/ddos.sh in your favorite editor (vi or pico) and search for the line which says "mail -s" (that is the command which sends you the mail). Put # at the beginning of that line. Save the file and exit the editor.
Done!

Sidenote: It is too bad if you have to block thousands of ips using this script. Infact, you are calling for trouble by doing so, as too many ips in the block list would make your VPS unstable.

Edit: Too many in my opinion would be anything more than 500.
Could it be that it is constantly trying to ban the same IP? Perhaps it'll be good to up the ban period?

[Zaf]

oh well elix, i really did not consider that aspect....the ban is indeed time bound, and yes you are right...if its the same ip.....one should actually ban it manually. What i was talking about was banning hundreds of ips at the same time and for longer period of time.

[yeshoward]

Hello

nope, it is mainly about "cron" email instead of ban IP notice....

I have disable it by puting a #, will this stop sending cron email too?

thanks

[yeshoward]

system keeps sending me cron notice? I gues this is controlled by system instead of script though? or any way to disable it? it nows sends cron notice every 2 minutes....

[@how@]

email sent to root@server4.domain.tld hot to set admin@domain.tld
i use centos 4 with DirectAdmin

[Zaf]

The script does not send any mails except when it bans Ip address(es). The mails are sent to root account for the server and if you had your root mails forwarded, these mails would be forwarded too. All the other mails you receive are from the Cron service and they are usually sent when a cron job experiences an error during its execution.

[@how@]

Thanks Zaf

[Zaf]

Hi Friends,
Happy New Year to all of you and hope you have a great year ahead. I've recently updated this script and it now runs much faster than before....especially if your server is a busy one. The speeds can go upto 8-10 times faster than the previous version. I have also added a few features like:
Frequency setting: The script runs as a cron job every minute by default and you might find that too much for your requirements, so change it if you like (though I would not recommend you do that. As per the feedback I received, some might want to run it every 30 seconds)
Email: The previous version would send an email to 'root' account, whereas you might prefer it on another mail id. You might even wish to make that null. This would stop sending out mails everytime it banned any ip.

*Please note that the script only mails when it bans ip address(es). All other mails come from the cron service and this script has nothing to do with it.

Edit: Feeback & suggestions would be greatly appreciated (even if negative).

Edit 1: To update the script (if you are already running it), uninstall the previous version and re-install the newer version. The steps involved are:
# wget http://www.inetbase.com/scripts/ddos/uninstall.ddos
# sh uninstall.ddos
# wget http://www.inetbase.com/scripts/ddos/install.ddos
# sh install.ddos

[@how@]

Thanks Zaf
Update done.



Wael

[resellertr]

Hi,

It is an excellent script. Is there a option for white list IPs? We have some servers connecting to mysql on another server. This script blocks mysql connections.

[elix]

Hi,

It is an excellent script. Is there a option for white list IPs? We have some servers connecting to mysql on another server. This script blocks mysql connections.
if you are using APF just do

apf -a 1.2.3.4

[resellertr]

I am using iptables.

[elix]

i recommend you switch to APF in that case.

[Zaf]

resellertr
I will add that feature you asked for asap....
thanks a lot for your feedback

[relevance4u]

Thanks for this nice script ... I like it but have the following problem

- max ips is set to 100
- 2 ips (two other hosts that do MYSQL queries) are constantly over 100
- the get banned/unbanned every minute
- I have both IPs in the
hosts.allow
AND apf's
/etc/apf/allow_hosts.rules


However - after the BAN the ip address is removed from the allow_hosts.rules and then moved to the deny_hosts.rules

My understandign would be, that the ddos.sh should not only check the deny_hosts.rules if the ip is already banned, but also read thru the allow_hosts.rules to find out if that ip is ALLOWED to be banned, because currently that's what's missing in the functionality...

please let me know - I'm willing to do the fix myself, but wonder if there are any thoughts from you that this is a bad idea...

uninstalling the script for now...

thanks in advance, relevance4u

[Zaf]

After writing the above post, I went on to modify the code and yes I almost completed it.....but i did not get enough time for testing. My work timings have been prohibiting me quite a bit in the past few weeks....but I shall try and squeeze in some time right away to test and roll it out.

Relevance4u thank you very much for your valuable feedback and the offer to fix it yourself....I've already worked on it and its just about a few hrs away I hope that you would be able to install the script again. I really appreciate your feedback.

[Zaf]

Version 0.6 released
Changes:
- whitelisting possible by adding ips in /usr/local/ddos/ignore.ip.list
- it uses this file to avoid banning the ip again (it was handling this differently and was a bit slower too)

Installation / Update Method:
It remains the same as last time (uninstall and reinstall the script), and you need to make changes to the conf to suit your preferences (The default values ban an ip with 150 connections (or more) for 600 seconds and run the script every minute)

Steps for installing/updating the script (ignore the first two steps if you are installing first time)
# wget http://www.inetbase.com/scripts/ddos/uninstall.ddos
# sh uninstall.ddos
# wget http://www.inetbase.com/scripts/ddos/install.ddos
# sh install.ddos

Feedback and suggestions are welcome

Thanks and Regards

[danfulton]

Is this a silly question?

Probably, but here I go anyway ...

I install this script while logged into my VPS as root, is that correct ? I don't install it per account on the VPS ?

Thaks

Dan Fulton

[elix]

Is this a silly question?

Probably, but here I go anyway ...

I install this script while logged into my VPS as root, is that correct ? I don't install it per account on the VPS ?

Thaks

Dan Fulton
You install it on root, correct.

[danfulton]

Thanks for the confirmation.

Dan Fulton

[SlAiD]

WOW.

I have loads of 20, and now i have 6/7....
Just one question/suggestion:
- i cannot find the config file, because:
- maybe you can make a script that show the output of the config file when finished, because when i make the "q" for quit, i will not see anymore the localization of config file.

Great, keep your work!

Rui

[elix]

WOW.

I have loads of 20, and now i have 6/7....
Just one question/suggestion:
- i cannot find the config file, because:
- maybe you can make a script that show the output of the config file when finished, because when i make the "q" for quit, i will not see anymore the localization of config file.

Great, keep your work!

Rui
try

cat /usr/local/ddos/ddos.conf | more

;)

also do make sure that it isn't blocking legit users.

[SlAiD]

Hello.

Thanks for your quickly awser

One nore thing. When i make some changes, i need to restaret someting? Ir it will take efect when the next cron job start?

Rui

[elix]

Hello.

Thanks for your quickly awser

One nore thing. When i make some changes, i need to restaret someting? Ir it will take efect when the next cron job start?

Rui
no need to restart, it will take affect on the next cronjob, yes

however, if you change the frequency the cronjob needs to be regenerated

so run:

ddos --cron

[Zaf]

Thanks Rui for trying the script and your feedback. I think when you type 'q' after the installation, you do see the config file location....i will check that again later and if required will make appropriate changes as required.

Thanks elix for your support in responding for queries related to the script. can you also pls update ur first post with info about the 0.6 version

[SlAiD]

Hello again.

Today i buy another VPS and i will install this great script on my new vps.

But i use the same adress to recive this type of stuff.

So, how can i change the email send to me, and "add" the server name or the ip name?

Thanks,
Rui

Banned the following ip addresses on Mon Jan 23 17:17:01 WET 2006

xxx.xxx.xxx.xxx with 94 connections

[Zaf]

The quickest way to achieve that would be to access the /usr/local/ddos/ddos.sh file in your favorite editor and change the quoted text on line no. 114 and / or line no. 144. For instance you might want to change the line as followsecho "Banned the following ip addresses on `date`" > $BANNED_IP_MAIL
to
echo "Banned the following ip addresses on server.name.com `date`" > $BANNED_IP_MAIL
cat $BANNED_IP_MAIL | mail -s "IP addresses banned on $dt" $EMAIL_TO
to
cat $BANNED_IP_MAIL | mail -s "IP addresses banned on server.name.com $dt" $EMAIL_TO
Regards

[elix]

the way i tell is the e-mail is generated from root@server.name.com so you can confirm the server through that :)

[jols]

Great script, great idea anyway. I have a few questions if you don't mind.

I installed this on a few of our servers and saw that it was working to ban some IPs at the rate of about 6 per hour or so, then I reduced the NO_OF_CONNECTIONS= value down to 50, and we received a swarm of complaints from our hosted customers who were not able to reach their sites.

I am wondering how, or why, any legitimate connection from a web site owner would incur as many as 50 simultaneous connections? In one case I saw, someone was working on their shopping cart, and this was the message that came in via email:


Banned the following ip addresses on Thu Jan 26 18:08:59 CST 2006

71.111.159.125 with 247 connections


247 connections?

How could this be. Does this mean that the owner of the hosting account has a virus on their computer?


Next question:

With the ddos.conf file set up this way:

FREQ=1
NO_OF_CONNECTIONS=50
APF_BAN=1
KILL=1
EMAIL_TO="root"
BAN_PERIOD=600


The IPs were not being cleared after 600 seconds. Would I need to change APF_BAN=1 to APF_BAN=0 in order to effect this?


Next question:

KILL=0 (Bad IPs are'nt banned, good for interactive execution of script)

If IPs are not banned, then what is the effect of the script? Would it have any effect at all? What do you mean by "interactive execution of script"?

Also:

If a hosted customer tries to do a mass emailing through their account, would this evoke a ban of their IP.

Thanks very much. Our servers have been hit very hard lately by all kinds of spam-bot probes and other similar things that look like they originate from external scripts. Hopefully your script will provide some relief.

[Zaf]

Hi
Thank you so much for your feedback. I have tried to answer your specific questions by email (in reply to your mail) too.

The script uses netstat command to find the number of connections each ip makes to the server (ranks them by the number of connection) and then uses APF or IPTABLES to ban the ips that exceed the limit set in conf file. The ip would be unbanned using APF or IPTABLES depending on your conf setting, however it requires APF v 0.96 or better installed on the server for the unbanning to work (if using APF_BAN).
Since the script uses netstat command, it looks for a network connection (mass emailing or multiple http connections or any other connection)

[elix]

zaf - first post updated.

i think an interesting idea would be to try having a script which checks the # of connections to httpd, if it's over X then run ddos.sh, if not, then don't run it.

this could prevent having to run the script and block legit IPs and only run it when you are actually being (d)dosed.

[yaax]

I've just tested this script and it is very nice with big potential to develop it more.
I've did some modifications there:
First it should only check ESTABLISHED connections and not all connections, so it just need to modify line 117 in the file ddos.sh and make it as next:
netstat -ntu | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr > $BAD_IP_LIST

Then it require to modify NO_OF_CONNECTIONS in the ddos.conf file. I've did equal as 30 - of Established only connections - only then it will block this bad ip.

Also I removed unban function as it is not function properly with APF because APF is not supporting -u option it cannot remove ip's from bad ip's list this way.
Also it is too dangerous to unban ip while DDOS attack not finished yet.
So the best way is to remove it manually or to make another function to open apf config file deny_host.rules, delete there line with unbaned ip and then restart apf as apf -r

Also this script require to kill all bunch of dead processes remaining after DDOS attack. For example after DDOS http attack on my server it remained in the RAM more then 500 httpd processes, so this script can easy track on what port was connected banned ip and match this port with processes in the RAM and killall all dead processes and then restart killed service.

It need to run command like this:
ps -ef | awk '{print $8}' | sort | uniq -c | sort -nr > $PROC_LIST

and then killall -9 there. As I figured httpd restart will make server load to jump up to 200-300, while simple killall -9 httpd and then restart apache - can very easy and fast resolve such problem.

[charles]

apf supports -u. If yours doesn't I suggest upgrading.

charles

[Zaf]

Hi yaax
Thank you for your feedback and I really appreciate your interest in the script.

First it should only check ESTABLISHED connections and not all connections, so it just need to modify line 117 in the file ddos.sh and make it as next:
netstat -ntu | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr > $BAD_IP_LISTAs suggested by you...if we add that code...the netstat command would be tad slower as it has to grep ESTAB too.

Also I removed unban function as it is not function properly with APF because APF is not supporting -u option it cannot remove ip's from bad ip's list this way.
Also it is too dangerous to unban ip while DDOS attack not finished yet.
So the best way is to remove it manually or to make another function to open apf config file deny_host.rules, delete there line with unbaned ip and then restart apf as apf -rAs mentioned by charles earlier, apf version 0.96 onwards -u works and that was probably mentioned by me somewhere earlier in the thread too I believe.

Also this script require to kill all bunch of dead processes remaining after DDOS attack. For example after DDOS http attack on my server it remained in the RAM more then 500 httpd processes, so this script can easy track on what port was connected banned ip and match this port with processes in the RAM and killall all dead processes and then restart killed service.

It need to run command like this:
ps -ef | awk '{print $8}' | sort | uniq -c | sort -nr > $PROC_LIST

and then killall -9 there. As I figured httpd restart will make server load to jump up to 200-300, while simple killall -9 httpd and then restart apache - can very easy and fast resolve such problem.
I agree that the script should kill the processes running for that bad Ip and I will look into this as soon as possible.

Once again thanks for your feedback and i appreciate your interest

[elix]

Also this script require to kill all bunch of dead processes remaining after DDOS attack. For example after DDOS http attack on my server it remained in the RAM more then 500 httpd processes, so this script can easy track on what port was connected banned ip and match this port with processes in the RAM and killall all dead processes and then restart killed service.

It need to run command like this:
ps -ef | awk '{print $8}' | sort | uniq -c | sort -nr > $PROC_LIST

and then killall -9 there. As I figured httpd restart will make server load to jump up to 200-300, while simple killall -9 httpd and then restart apache - can very easy and fast resolve such problem.
That is is a *VERY* good idea, but it takes away from the lightweight appeal the script has atm.

[yaax]

As suggested by you...if we add that code...the netstat command would be tad slower as it has to grep ESTAB too.


As I checked on my server, the most CPU time is used by netstat -ntu command itself and not so much by other grep,awk,uniq and other. Also after many lines from netstat result are deleted by grep - there will be much less work to do for other five commands - awk,cut,sort,uniq - they will not need to do the job on all hundreds of lines on dead connections - and only will work on really live - established connections.
I think such grep command only cause for faster work and for less load on CPU.

Also regarding unban - apf version is only part of problem. The real problem is to know whether DDOS attack really finished or not - whether unban of bad ip will not cause immediate ban again of same ip.
I think it will be good to make some simple AI - to check how many connections were done by one ip and regarding from this number setup different unban delays. For example - if from one ip are done about 30-40 established connections - it is still not sure that some hacker is attacking you from this ip, so it can be banned for few minutes only. But if from one ip made more then 100 established connections - thats is for sure serious DDOS attack and this ip must be banned for very long period - may be even forever (it can be unbanned manually anytime).
However after each unban will be good to keep history of this ip and check how many attacks were made by each ip and each ISP and make appopriated unban delays.
For implementation of ban history - is enough to log all ban messages into one log file, and then it will be possible to grep it for some bad ip's and check when and how much they were banned.

Also I want to add a link to Cpanel forums where I posted my similar script for preventing CPU high-load:
http://forums.cpanel.net/showthread.php?t=22568&page=5
I think this script is doing similar job and it maybe be useful to merge some parts of it... or to use both these scripts together as I do on my servers.

[CRUEL]

The forums new messages I am looking at 10-15 opening the page, immediately banned:(
The meaning of these what is?;
no of connections=?
Ban period=?

[charles]

The forums new messages I am looking at 10-15 opening the page, immediately banned:(
The meaning of these what is?;
no of connections=?
Ban period=?

I honestly don't understand you at all. Can you elaborate?

charles

[elix]

I honestly don't understand you at all. Can you elaborate?

charles
Yes, I agree.....you can increase the no of connections though if you feel that you it is too low.

[asterisk]

The forums new messages I am looking at 10-15 opening the page, immediately banned:(
The meaning of these what is?;
no of connections=?
Ban period=?

That is normal. Just wait for the specified time period (10 to 15 seconds) before you are able to check for New Posts on PVPS's forum. I reckon the time delay is there to avoid overloading the server with forum searches on new posts etc.

By the way, I reckon the abovementioned comment may have nothing to do with DOS-Deflate.

Hope this helps.

Also, I'd like to ask. What is the difference between DOS-Deflate and another script I noticed around here, dos_evasive please? I saw that dos_evasive breaks frontpage extensions. What about DOS-Deflate?

Any comments would be much appreciated.

[elix]

deflate will probe netstat every minute and search for an attack - it generally doesn't have as many false positives and is better at catching 'bad' IPs. it shouldn't have an issue with FP.

dosevasive runs as an apache module, while this just probes netstat.

[asterisk]

Thanks for the comments, Elix. And thanks Zaf for such a nifty prog. Would definitely love to give it a go soon enough. :D

[elix]

Zaf, I did some testing and

netstat -ntu | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr > $BAD_IP_LIST

is actually a little faster in my case, and perhaps in other cases too.

This is because once you grep it down to just the ESTAB connections there is less data for all the other sorting, awk, cut, etc. to work with and it excecutes faster.

I added

netstat -ntu | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr > $BAD_IP_LIST

to my install and I think you may want to release then in the default.

This way you can lower the amount of connections that trigger a block of the IP as it is much more accurate and less prone to false positives.

[resellertr]

Hi,

It is possible to run this script on Freebsd 5.3 .

Thanks.

[elix]

Hi,

It is possible to run this script on Freebsd 5.3 .

Thanks.
I don't believe freebsd supports iptables (ipfw is what is generally used on bsd, iirc) therefore no this will not work. but you are free to modify this script - atleast the banning functions.

[resellertr]

it is easy to modify for banning but can it count correcty # of connections?

[elix]

it is easy to modify for banning but can it count correcty # of connections?
you can test that by trying the netstat command

netstat -ntu | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

does that run fine on BSD?

[resellertr]

Hi,

netstat -na | awk '{print $5}' | cut -d. -f1-4 | sort -n | uniq -c | sort -n

I can see # of connections with this command.

[elix]

Should work fine if you modify it.

[ozgreg]

Can we have the subject of the Banned Email to instead be $IPADD Banned on $date?

[elix]

Sure...

Line 144 in /usr/local/ddos/ddos.sh

cat $BANNED_IP_MAIL | mail -s "SUBJECT HERE" $EMAIL_TO

[ozgreg]

Already tried that, except cannot get the banned IP from any of the current IP BAN variables because of the loop..

[elix]

afaik multiple ips can be banned in one shot therefore it would be hard to print it in the title when there are line breaks. you can output the list to a file and tail -1 it though. then you will get 1 ip.

[ozgreg]

It got nothing to do with the multiple IP's. If you look at the script you will see the current IP is being changed when you loop back through the deny.host thus by the time the email is outputted the original (offending) IP has long gone..

[Zaf]

The script loops to find multiple bad ips and bans them all in one go. Therefore, it would be hard to display the ip in the subject as elix mentioned.

However, modifications can be made to show either the first banned ip or the last banned ip in the subject line....but when multiple ips are blocked....it might not make sense...

[yaax]

I've did number of modifications of this script.
Main of them is that now it can show you what sites where attacked.
Also I am using only against HTTP DDOS attacks so I do grep of all established connections on port 80 And also I completely disabled unban of banned ip's, I can unban them manually if I will see that DDOS attack really has finished and ip is not from blacklisted servers.

Last point that I did finding of target site of DDOS attack by three different ways:
1. By PID (so I also modified netstat parameters to be -ntup - with addition of 'p' it will show process ID's of all connection and after knowing of PID script can enter to /proc/PID/cwd folder and see there from what path where initiated this connection. This way is working properly only in part of HTTP connections.

2. By full apache status page - where script also can find target ip's and connection PIDs, however in case of serious DDOS attack apache status may be anavailable or will just show "???" instead of site name.

3. By scan of last apache access logs (done for CPanel servers where all sites logs are in /usr/local/apache/domlogs directory) - so it only search for all active files during last 5 minutes and doing grep of 100 last lines of each file searching there banned ip's
This way is most realiable and is working almost in case of any HTTP DDOS attack.

Here is my midified ddos.sh file:
http://k1g.com/scripts/ddos.sh

[elix]

Nice job on the modifications, yaax, I am going to be putting this through some testing.

[elix]

Bump.

We now have an official site for this project.

http://projects.medialayer.com/

Thanks to everybody for their support.

In addition, we plan on releasing a new version shortly with quite a few new features.

[SlAiD]

Nice site.

Simple but with all information.


Btw, yaax, how can i use your modificated version? I've download it and run, but it just make a list of ip's. How can i *install* it?


SL

[elix]

Nice site.

Simple but with all information.


Btw, yaax, how can i use your modificated version? I've download it and run, but it just make a list of ip's. How can i *install* it?


SL
Hey,

thanks for the comments on the site. It was actually intended to be something quick thrown up for now - but yes, it was meant to be a simple but intuitive design..

you may use yaax's version if you want by doing the following:

# cd /usr/local/ddos
# cp ddos.sh ddos.sh.bak
# wget http://k1g.com/scripts/ddos.sh -O ddos.sh


Now you will be running his modified version.

HOWEVER...I do recommend waiting for an official reason as we will be incorporating some of yaax's tweaks including some more 'interesting' features (;)) very shortly.

Thanks,

[yaax]

Nice site.

Simple but with all information.


Btw, yaax, how can i use your modificated version? I've download it and run, but it just make a list of ip's. How can i *install* it?


SL

If you get just list of ip's this mean that script worked properly - it found all ip's connected by established connections to port 80 to your server and show how much connections did each ip.
When some ip reach critical number of connections (this parameter is setup in ddos.conf file - NO_OF_CONNECTIONS), this script will print banning ip... and then will display information on what domains and paths was found this ip and also will send email with detailed report.

[resellertr]

Hi,

I have a problem on my new cent os 4.3 box.

When I run netstat -an on ssh;

I get result like this,

tcp 0 0 ::ffff:72.36.165.140:80 ::ffff:195.175.16.26:59981 TIME_WAIT

and

[root@server136 etc]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
580
8 213.243.4.50
2 85.98.20.201
1 servers)
1 Address
1 85.102.97.231
1 81.215.243.31
1 81.214.21.34
1 194.67.52.19
1 127.0.0.1
[root@server136 etc]#

What is this 580 ??

[elix]

I wouldn't worry about the 580 honestly, it wouldn't cause the script to malfunction AFAIK.

[yaax]

Hi,

I have a problem on my new cent os 4.3 box.

When I run netstat -an on ssh;

I get result like this,

tcp 0 0 ::ffff:72.36.165.140:80 ::ffff:195.175.16.26:59981 TIME_WAIT

and

[root@server136 etc]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
580
8 213.243.4.50
2 85.98.20.201
1 servers)
1 Address
1 85.102.97.231
1 81.215.243.31
1 81.214.21.34
1 194.67.52.19
1 127.0.0.1
[root@server136 etc]#

What is this 580 ??

If you will add grep ESTAB to netstat command, then you will not get such junk.

[elix]

If you will add grep ESTAB to netstat command, then you will not get such junk.
yes, that will probably help.

netstat -ntu | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

[zeusmaster]

Hi,

I have a problem on my new cent os 4.3 box.

When I run netstat -an on ssh;

I get result like this,

tcp 0 0 ::ffff:72.36.165.140:80 ::ffff:195.175.16.26:59981 TIME_WAIT

and

[root@server136 etc]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
580
8 213.243.4.50
2 85.98.20.201
1 servers)
1 Address
1 85.102.97.231
1 81.215.243.31
1 81.214.21.34
1 194.67.52.19
1 127.0.0.1
[root@server136 etc]#

What is this 580 ??

yes and apf deny_host.rules

# added 447 on 05/12/06 23:17:01
447
# added 411 on 05/12/06 23:18:03
411
# added 663 on 05/12/06 23:19:01
663
# added 580 on 05/12/06 23:20:02
580
# added 507 on 05/12/06 23:21:01
507 ---> NOT IP ?

[CRUEL]

Top Process %CPU 26.0 netstat -ntu
high load?? :s


Top Process %CPU 38.0 netstat -nlp
Top Process %CPU 26.0 netstat -ntu
Top Process %CPU 24.0 netstat -ntu


What is -ntl ?

[yaax]

If you are checking Cpanel cpu-usage index, as for me it is not correct indicator for real CPU usage.
Ok indeed netstat -ntu command may use some CPU percent but overall how much script used - this depend on many factors - by frequency of its executions and by server load of other tasks and others etc.

[elix]

Top Process %CPU 26.0 netstat -ntu
high load?? :s


Top Process %CPU 38.0 netstat -nlp
Top Process %CPU 26.0 netstat -ntu
Top Process %CPU 24.0 netstat -ntu


What is -ntl ?
Those are done by RFX Networks scripts and have NO relation to (D)DOS-Deflate.

[resellertr]

Hi zeusmaster ,

Recompile kernel Without ipv6. I solved this problem by disabling ipv6

[tata13]

Is there any setting in this scripts to ignore some IP to be banned ? coz my own IP always banned by this scripts .

Thanks

btw
this is my /usr/local/ddos/ddos.conf

NO_OF_CONNECTIONS=150
APF_BAN=0
EMAIL_TO= "myname@domain.com"
BAN_PERIOD=259200

not using apf

[elix]

you can add any ips that you want to be whitelisted in:

/usr/local/ddos/ignore.ip.list

[tata13]

Already do that, still this scripts banned my own IP that I add into /usr/local/ddos/ignore.ip.list , is there any special format to add whitelist IP in that file ? and how to add multiple IP to the whitelist ?

[elix]

ill have a look at the script later today if possible and see if there is a possible bug with the handling of the whitelist.

[resellertr]

Already do that, still this scripts banned my own IP that I add into /usr/local/ddos/ignore.ip.list , is there any special format to add whitelist IP in that file ? and how to add multiple IP to the whitelist ?

I have same problem :)

[elix]

Hello,

There should be a new version of this script coming out shortly to correct these issues as well as add new features.

Thanks,

[stevechen123]

I have a question,

If searching spider visit our site and the connections or time is more than the limit, how to do? I dont' want to ban them.

Please help!!!

Steve.

[Zaf]

You could whitelist the ip, but dont think its going to be as simple as i said it.

[yaax]

I have a question,

If searching spider visit our site and the connections or time is more than the limit, how to do? I dont' want to ban them.

Please help!!!

Steve.

For such limits you may use another tools as apache modules like limitipconn which can limit connections per ip or bandwidth throttling module.

[elix]

..we are pleased to announce the release of our new site!

http://deflate.medialayer.com/

[ozgreg]

Elix,

You might want to add grep ESTAB into your netstat command as per the previous page of this thread :-)

[elix]

ill see what i can do in the morning

[Tony]

Nice script, only modification i'd make is to add some logic to remove the first two lines of netstat output so "servers)" and "Address" don't appear in the listing:

Something along the lines of


netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | grep -v -e server -e Address


Should suffice

One thing I would also filter from your command is 127.0.0.1 and 0.0.0.0 as it would be bad to block those, and i've seen stupid amounts of local connections in the past, due to extremely poor code.


netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | grep -v -e server -e Address -e 127.0.0.1 -e 0.0.0.0

[Zaf]

Nice script, only modification i'd make is to add some logic to remove the first two lines of netstat output so "servers)" and "Address" don't appear in the listing:I never bothered about those lines as I was more worried about performance of the script....always thought why add another '|' when its only for cosmetic.One thing I would also filter from your command is 127.0.0.1 and 0.0.0.0 as it would be bad to block those, and i've seen stupid amounts of local connections in the past, due to extremely poor code.Yes I'm now getting tempted to add another '|' to the command for this, but those addresses can also be taken care of by whitelisting them in /usr/local/ddos/ignore.ip.list

[elix]

I agree with Zaf on this. The servers) and Address are purely cosmetic and they won't actually cause a problem per say.

Yes I'm now getting tempted to add another '|' to the command for this, but those addresses can also be taken care of by whitelisting them in /usr/local/ddos/ignore.ip.list
Yeah, perhaps we should ship ignore.ip.list with a default of 127.0.0.1 and 0.0.0.0 if it's not already?

[Fred]

hi guys,

sorry if this question is already answered, but i don't want to pass through 10 pages of posts ;)

Is any ban is followed by a mail sent to the admin to notice him ?
:)

[elix]

hi guys,

sorry if this question is already answered, but i don't want to pass through 10 pages of posts ;)

Is any ban is followed by a mail sent to the admin to notice him ?
:)
That's a feature within the script, yes, by default it is sent to 'root', but you can change this by editing /usr/local/ddos/ddos.conf

hth

[Fred]

Just installed it...

So far, it doesn't cause any problem :)
I did add 127.0.0.1 and 0.0.0.0 in the ignore list...

:)

[Fred]

oh... just a little question:

By this line:
##### APF_BAN=1 (Make sure your APF version is atleast 0.96)

did you meant apf 0.9.6 ???

[Tony]

I never bothered about those lines as I was more worried about performance of the script....always thought why add another '|' when its only for cosmetic.

Check it with time { }, but I don't see it changing performance in anyway but for perhaps 0.001seconds +/-.

[elix]

oh... just a little question:

By this line:
##### APF_BAN=1 (Make sure your APF version is atleast 0.96)

did you meant apf 0.9.6 ???
Yes, APF 0.9.6 :)

We will have that edited

[Fred]

Just had a small problem minutes ago...

It banned one of my customers... for having more than 150 connections. I verified the IP in the logs and found it was one of my customers.

I didn't verified but i believe he was uploading by ftp...

Ever met that problem ? What should i do to avoid that ?



EDIT: Forget some words in my last sentence :)

[elix]

It's not really a 'problem', it's just with the nature of the script and how it bans IPs...try increasing the # of connections in /usr/local/ddos/ddos.conf to your needs...

[dwh-kenl]

i gave the script a tried, after installing i started getting all these server warning emails that i never use to get

Status warning from host.xxx.com

they seems to keep coming every 2 to 3 minutes and i have uninstalled the script but it just wouldnt stop..

how can i stop it?

looks like something started sim.log and it keeps warning me about
[10/03/06 23:55:01]: FTP restart failed, could not find /etc/init.d/proftpd.

[Daniel]

i gave the script a tried, after installing i started getting all these server warning emails that i never use to get

Status warning from host.xxx.com

they seems to keep coming every 2 to 3 minutes and i have uninstalled the script but it just wouldnt stop..

how can i stop it?

looks like something started sim.log and it keeps warning me about
[10/03/06 23:55:01]: FTP restart failed, could not find /etc/init.d/proftpd.
Are you on cPanel? Which FTPd are you using? proftpd or pure-ftpd? This is normally caused from an issue in the SIM Config where it thinks you're using proftpd and you're using pure-ftpd.

[elix]

i gave the script a tried, after installing i started getting all these server warning emails that i never use to get

Status warning from host.xxx.com

they seems to keep coming every 2 to 3 minutes and i have uninstalled the script but it just wouldnt stop..

how can i stop it?

looks like something started sim.log and it keeps warning me about
[10/03/06 23:55:01]: FTP restart failed, could not find /etc/init.d/proftpd.

This has nothing to do with the script.

[stephan]

Hi,

First of all, thanks for the work on the script.

I'm having a little bit of trouble sadly. I'm using CentOS 4.4

When I do a test DDOS to my server to see if it works, I get e-mails like this:

"Banned the following ip addresses on Wed Oct 4 23:42:01 BST 2006

316 with 316 connections


."


I think the issue is with the netstat grep command on my server.



If I run this manually:

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

I get:

8
1 servers)
1 Address


Is there another command I can substitute?

My server is:

CentOS release 4.4 (Final)
Linux serverXXXXXXXX.XXXXXXXXX.com 2.6.9-42.0.2.plus.c4 #1 Fri Aug 25 17:33:49 CDT 2006 i686 i686 i386 GNU/Linux

netstat -V gives:

net-tools 1.60
netstat 1.42 (2001-04-15)
Fred Baumgarten, Alan Cox, Bernd Eckenfels, Phil Blundell, Tuan Hoang and others
+NEW_ADDRT +RTF_IRTT +RTF_REJECT +FW_MASQUERADE +I18N
AF: (inet) +UNIX +INET +INET6 +IPX +AX25 +NETROM +X25 +ATALK +ECONET +ROSE
HW: +ETHER +ARC +SLIP +PPP +TUNNEL +TR +AX25 +NETROM +X25 +FR +ROSE +ASH +SIT +FDDI +HIPPI +HDLC/
LAPB


The net-tools.i386 package is 1.60-37.EL4.8


Does anyone have any tips please? This is a great script, I'd like to use it.

[elix]

hi,

try removing some of the pipes in the command ( the | stuff) and see what happens


netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

particularly the cut and the awk.

[stephan]

Hi,

Thanks.

I've tried this:

[root@server30080 ~]# netstat -ntu | awk '{print $4}' | sort | uniq -c | sort -nr

and that gives:

14 ::ffff:83.170.75.95:80
4 ::ffff:83.170.75.95:9999
1 (w/o
1 Local

Is that the kind of output you get too when you run the normal (original) command?

[stephan]

Or, maybe this is better:

netstat -an | cut -c 45-68 | awk -F\: '{print$4}' | sort | uniq -c | sort -n | grep -v 127.0.0.1


It gives an output like this:

[root@server30080 ~]# netstat -an | cut -c 45-68 | awk -F\: '{print$4}' | sort | uniq -c | sort -n | grep -v 127.0.0.1
1 82.195.113.20
2 62.252.128.28
5 *
134



Could someone paste me what the original command's output is on their computer please so I can compare? I need to know what kind of format to get in.

I think I'm nearly there, apart from the "5 *" and "134" bit which I suppose would need to be removed for the script to like it.

[SlAiD]

bash-2.05b# netstat -an | cut -c 45-68 | awk -F\: '{print$4}' | sort | uniq -c | sort -n | grep -v 127.0.0.1
138
bash-2.05b#

O.o

Strange...
SL

[stephan]

It is strange isn't it.

Why does everyone else's work ok apart from mine!

I don't see what is different about my version of netstat, awk, sort, or whatever is doing it:(

Maybe it's my locale that's changing the formatting of the output of netstat ever so slightly? I don't see how though...

I think it's back to the drawing board for me! If anyone has any input, I'd like to hear it, I really want to use this script.

[ecsportal]

So is there any particular reason that:

netstat -ntu | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | grep -v -e server -e Address -e 127.0.0.1 -e 0.0.0.0

Would be a bad idea?

[elix]

Well first of all for the IP addresses in this statement:

grep -v -e server -e Address -e 127.0.0.1 -e 0.0.0.0

you should be using /usr/local/ddos/ip.ignore.list. there is already functionality available for this.

the server and address part is purely cosmetic so you aren't increasing functionality, just reducing performance through the grep

[ozgreg]

My server has been hammered lately by a slightly different style of DDOS attack. The attacker is generating between 50-70 threads across generally 4-8 IP's. The IP's will all be in the class C range (ie 192.168.1.100, 192.168.1.102 etc) since the number of threads is kept low the DDOS attack does not trigger the ban..

Can the script be altered to handle something like..

If total number of connections exceeds x number change the BAN threads to be a different number (a much lower number, say 30) that way the ban will kill the attacker.. (It may effect others but at least this is better than not stopping them at all)

[elix]

If total number of connections exceeds x number change the BAN threads to be a different number (a much lower number, say 30) that way the ban will kill the attacker.. (It may effect others but at least this is better than not stopping them at all)

we can code something like that as an addon to the script, yes.

[ozgreg]

Did I mention that I would like it by tomorrow :-)

[oaf357]

I too have CentOS and am having the random three digit number generated by this script. I have no clue what the deal is with it but I would imagine it's CentOS specific. The problem is that I really need this script as I've been having some random drive-bys on my server lately (thus the reason I'm here now).

If anyone has cracked this cookie please let me know.

Thanks!

[elix]

I too have CentOS and am having the random three digit number generated by this script. I have no clue what the deal is with it but I would imagine it's CentOS specific. The problem is that I really need this script as I've been having some random drive-bys on my server lately (thus the reason I'm here now).

If anyone has cracked this cookie please let me know.

Thanks!

I've ran this on numerous CentOs servers and have never had such a problem...I will have zaf look at this

[oaf357]

Any updates?

[jdk]

http://forums.deftechgroup.com/showpost.php?p=10761&postcount=73

yes and apf deny_host.rules

# added 447 on 05/12/06 23:17:01
447
# added 411 on 05/12/06 23:18:03
411
# added 663 on 05/12/06 23:19:01
663
# added 580 on 05/12/06 23:20:02
580
# added 507 on 05/12/06 23:21:01
507 ---> NOT IP ?

I am having the same problem is there any fix for this ?

[stephan]

Hi,

I've come back to this issue and had a go at seeing if I can fix it.

I've no idea what it is about this command that is making it not work properly with CentOS, but it's annoying.

I have come up with this and it seems to work, but it's hard for me to say. Right now, my websites haven't been updated for months and due to their nature, no updates = no traffic.

I think this command works, but I can't say for sure as I'm basically the only person on the server right now!

Can someone look and let me know please? Note that it's only really CentOS people that are having the problems. It would be nice to know if it works on other distributions though, so if people could try the command and reply with the result, and the name/version of their distribution, that would be appreciated.

netstat -ntu | grep ffff | awk '{print $5}' | cut -d: -f4 | sort | uniq -c | sort -nr

Thanks :cool:

It's pretty similar in the end.




*** I just tested it by DOS-ing my own server, it seems to work. I got blocked for 10 minutes, then unblocked again :-)

Thanks for the script.




*** Update: It only works on connections to port 80 it seems. It might work on SSL as well, I haven't tried. I tried making connections to port 25 and they appear without the :ffff: thing, so it doesn't work. I'll leave it like this for now. Does anyone have any ideas on how to get round this please?

[stephan]

Ok, how about this!


netstat -ntu | grep ':' | awk '{print $5}' | awk '{sub("::ffff:","");print}' | cut -f1 -d ':' | sort | uniq -c | sort -nr



gives:

2 82.195.113.201
1 74.6.86.172
1 74.6.85.164
1 212.20.230.11

It's looking right for the moment. Can someone test it on a server getting lots of hits and let me know how it looks please?

[Tony]

bash-2.05b# netstat -an | cut -c 45-68 | awk -F\: '{print$4}' | sort | uniq -c | sort -n | grep -v 127.0.0.1
138
bash-2.05b#

O.o

Strange...
SL

You've got 138 blank lines.

netstat -an | cut -c 45-65 gives you:

0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
blished)
I-Node Path
G 588724998 /var/lib
G 2764 public/clea
G 2771 private/rew
G 2775 private/bou
G 2779 private/def
G 2783 private/tra
G 2787 private/ver
G 2791 public/flus
G 2795 private/pro
G 2799 private/smt
G 2803 private/rel
G 2807 public/show
G 2811 private/err
G 2815 private/loc
G 2819 private/vir
G 2823 private/lmt
G 2827 private/anv
G 2831 private/mai
G 2835 private/old

then you throw an awk that's trying to print the 4th 'column' of data separated by a :... which causes every line to be blank as that doesn't exist.

I haven't read the whole thread, so i'm not sure exactly what output you're trying to get. Just thought I'd clear up why you're getting that 'weird' output tho. :)

[stephan]

Thanks.

I'm guessing you are using an OS other than CentOS, for some reason the output that I get on either Redhat or CentOS is different than what I get on another distribution. I don't know why. The command above works ok on my server and another server I tested it on (redhat), but not on your computer, as you pasted. It's weird....

I've just DOS'd my own server again to test it and it looks like I'm blocked. I'm just waiting to see if I get unblocked automatically now. I think it should work.



*** Edit: I just re-read your message, you were posting about the one I did a while back which was wrong. Duh.... I am way too tired!

[SlAiD]

Hi Tony.

You dont read the thread, I cant remember ny own messege. LOL.

Maybe it was fixed, or maybe it fixe itself. Anyway, I think its solved. Thank you anyway.


SL

[stephan]

I have a further improvement:



netstat -ntu | grep ':' | awk '{sub("::ffff:","");print $5}' | cut -f1 -d ':' | sort | uniq -c | sort -nr


It's only an improvement if the original command doesn't work for you though.



*** grrrr... that doesn't work, it sometimes prints numbers on their own. I'll go away and hide...

[kone]

Hello, I have installed this script, got a few questions hope someone can help.

1. how can I test it and make sure it is running?

2. I set the NO_OF_CONNECTIONS=80, is this too low? I have got people scanning my server with some tool and connected less than 150 per minute. Just wonder will this setting 80 ban legitimate users too?

3. NO_OF_CONNECTIONS=80, is this referring to per IP per 1 minute with a connection of 80 will get banned or does it refer to per IP at any time with a connection of 80 will get banned?

thanks

[elix]

Hello, I have installed this script, got a few questions hope someone can help.

1. how can I test it and make sure it is running?

2. I set the NO_OF_CONNECTIONS=80, is this too low? I have got people scanning my server with some tool and connected less than 150 per minute. Just wonder will this setting 80 ban legitimate users too?

3. NO_OF_CONNECTIONS=80, is this referring to per IP per 1 minute with a connection of 80 will get banned or does it refer to per IP at any time with a connection of 80 will get banned?

thanks

I would say that it is a bit low. The script checks every minute by polling netstat, if there are more than X connections that are open at that time, then the IP address will be be banned.

[kone]

Hi thanks for replying,

I have some problems...

under my ignore list

127.0.0.1
62.xx.xx.xx
63
89
56
12

the program add those 2 digits numbers to my ignore list and email me the ip got banned.

I checked the deny_host file nothing was added there.

I have an incident that the program successfully added a ip address to deny_host file but it also ADDED the same ip to ignore list....

Please help something is wrong..

[elix]

Hi thanks for replying,

I have some problems...

under my ignore list

127.0.0.1
62.xx.xx.xx
63
89
56
12

the program add those 2 digits numbers to my ignore list and email me the ip got banned.

I checked the deny_host file nothing was added there.

I have an incident that the program successfully added a ip address to deny_host file but it also ADDED the same ip to ignore list....

Please help something is wrong..


It is supposed to do this. so it doesnt ban it again extraneously until the time period is up. then its removed.

other issue is fixed in latest release, but we are still working on it - it is not public yet

[cigarOC]

Few issues here..

I am running Redhat ES4, and have installed DoS-Deflate. When it goes to block an ip it sends me the following email:

Banned the following ip addresses on Fri Jan 11 17:40:03 PST 2008

232 with 232 connections

and adds the following to my ignore.ip.list

371
421
672
458
572
338
232
307

Seems very odd.. also, how would I check which IP's are being blocked? I am currently using DoS-Deflate with iptables. Also, forgot to mention my server is taking on a huge DDoS attack right now :( Sometimes getting up too 98.34 Mbits/s.

Thanks!

[Bogdan]

Any csf implementation in the future?

[stking]

*bump*

is there still no fix?

[BornOnline]

CSF already does this - check connection tracking

[stking]

CSF already does this - check connection tracking


and if I dont use CSF? and perhaps want to use this with APF?