netrider
03-27-2006, 07:55 AM
Somewhere between occassionally and frequently, our VPS will go into extreme and very un-normal load (as shown by TOP and VZPP). Load number of 8+ and Apache requests timing out with 500 - server error messages as a result.
The other day, I opened a support docket when just such an occurrance was happening. By the time the docket was actioned though, server load seemed to have returned to normal (1 - 2) though. Bugger!
Anyway, the support response also noted that we had many concurrent connections from single IP's, and suggested we might want to block them :eek:
I know some/many/all browsers will create 5-8 concurrent connections to a webserver, so by my thinking any IP's with more concurrent connections than that should be proxy servers or business gateway servers - correct?
So I had a look at some of the IP's in the list support provided with lots of concurrent connections to our server, and they don't appear (by NSLOOKUP and TRACEROUTE) to be proxy's or gateway. In fact some clearly seem to be individual client (eg. ADSL) connections.
We don't have POP/IMAP or FTP active, so connections to our server should solely be HTTP. Since HTTP is instantaneous/stateless, it's connection to our server should stop/finish as soon as apache has respond to the request/get - correct?
Here's the releavant part of the support ticket response....
When I checked, your VPS load was around 1-2 but these were the currently active connections (with connection counts):
1 127.0.0.1
1 202.146.253.4
1 220.233.0.26
1 59.167.86.84
1 64.156.215.18
1 66.249.177.179
1 68.142.249.134
1 68.142.249.153
2 144.131.177.179
2 144.139.109.105
3 203.164.166.88
3 203.4.250.160
3 220.253.112.104
4 203.206.99.162
4 64.104.252.130
5 202.128.117.7
5 202.63.37.57
5 60.225.94.173
7 58.165.35.64
10 165.228.125.243
10 66.249.66.108
16 59.167.126.79
18 202.7.183.130
22 58.165.207.49
24 144.139.163.168
35 0.0.0.0
36 218.185.69.130
42 211.28.158.230
45 202.139.145.86
50 60.230.181.228
As you can see there are quite a few IPs which do appear to have a lot of connections. You can monitor the IPs and connection counts using the following command:
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
You might want to look into some of these IPs and block them if necessary.
Even right now, if I run the suggested netstat command above, I get...
10 220.101.52.144
11 220.253.54.17
12 211.28.173.237
24 203.129.155.171
30 202.161.22.138
35 0.0.0.0
45 211.28.33.200
46 210.49.252.152
48 220.238.65.200
As an example, the last IP on the "right now" list above is 220.238.65.200. With NSLOOKUP, that translates to d220-238-65-200.dsl.vic.optusnet.com.au which tells me that it is assigned to an ADSL client and thus wouldn't be a proxy or business gateway (particularly as it is right now almost 11pm)
Could anyone explain to me why these IP addresses would have so many concurrent connections open to our server? Why and should the IP address have that many? Can, and how would, I track/monitor what that IP address is performing/requesting with each connection? Should I be blocking them as support suggests, and if so why?
Thanks.
The other day, I opened a support docket when just such an occurrance was happening. By the time the docket was actioned though, server load seemed to have returned to normal (1 - 2) though. Bugger!
Anyway, the support response also noted that we had many concurrent connections from single IP's, and suggested we might want to block them :eek:
I know some/many/all browsers will create 5-8 concurrent connections to a webserver, so by my thinking any IP's with more concurrent connections than that should be proxy servers or business gateway servers - correct?
So I had a look at some of the IP's in the list support provided with lots of concurrent connections to our server, and they don't appear (by NSLOOKUP and TRACEROUTE) to be proxy's or gateway. In fact some clearly seem to be individual client (eg. ADSL) connections.
We don't have POP/IMAP or FTP active, so connections to our server should solely be HTTP. Since HTTP is instantaneous/stateless, it's connection to our server should stop/finish as soon as apache has respond to the request/get - correct?
Here's the releavant part of the support ticket response....
When I checked, your VPS load was around 1-2 but these were the currently active connections (with connection counts):
1 127.0.0.1
1 202.146.253.4
1 220.233.0.26
1 59.167.86.84
1 64.156.215.18
1 66.249.177.179
1 68.142.249.134
1 68.142.249.153
2 144.131.177.179
2 144.139.109.105
3 203.164.166.88
3 203.4.250.160
3 220.253.112.104
4 203.206.99.162
4 64.104.252.130
5 202.128.117.7
5 202.63.37.57
5 60.225.94.173
7 58.165.35.64
10 165.228.125.243
10 66.249.66.108
16 59.167.126.79
18 202.7.183.130
22 58.165.207.49
24 144.139.163.168
35 0.0.0.0
36 218.185.69.130
42 211.28.158.230
45 202.139.145.86
50 60.230.181.228
As you can see there are quite a few IPs which do appear to have a lot of connections. You can monitor the IPs and connection counts using the following command:
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
You might want to look into some of these IPs and block them if necessary.
Even right now, if I run the suggested netstat command above, I get...
10 220.101.52.144
11 220.253.54.17
12 211.28.173.237
24 203.129.155.171
30 202.161.22.138
35 0.0.0.0
45 211.28.33.200
46 210.49.252.152
48 220.238.65.200
As an example, the last IP on the "right now" list above is 220.238.65.200. With NSLOOKUP, that translates to d220-238-65-200.dsl.vic.optusnet.com.au which tells me that it is assigned to an ADSL client and thus wouldn't be a proxy or business gateway (particularly as it is right now almost 11pm)
Could anyone explain to me why these IP addresses would have so many concurrent connections open to our server? Why and should the IP address have that many? Can, and how would, I track/monitor what that IP address is performing/requesting with each connection? Should I be blocking them as support suggests, and if so why?
Thanks.