Nodes resource usage is not supposed to affect other nodes!?! Not Happy!

[netrider]

From http://www.powervps.com/what.php comes ....

This resource management allows us to define a minimum guaranteed level of RAM and CPU for every VPS. So no matter what another VPS does your VPS is always guaranteed to be unaffected and function normally. [...]

Quality of Service to provide dedicated SLAs for each VPS

* Standard: Includes CPU, disk space and network guarantees
* Unique: Guarantees on memory - user and kernel, physical and virtual
* Unique: Guarantees on disk I/O and many other critical resources (over 20).
[...]
* Performance:
In a shared environment resources goes to the site 'that shouts the loudest' in other words a big site can drain all resources and leave yours performing badly. In a virtual private server environment you are guaranteed a set of resources to ensure your site always performs at full speed, these resource guarantees ensure other virtual private servers have no affect on the performance of your site


A few times recently, and also just now, our HTTP site is extremely slow (to the point of giving false 404's), I get Virtuozzo QOS alerts for ftpd failure, SSH takes for ever to give a login, SSH command line tools such as "top" take 30+ seconds to load, and Virtuosso page loads take 2+ minutes. Clearly somethign is wrong with our node/server ... I open a support docket (#FGR-16774-740).

Then comes this response from support to my docket ...

Hi,

There is an issue with another VPS on your node using extreme resources and affecting your VPS performance. I am addressing this now.

Regards
Tony Dodd
DTG/PowerVPS Support Team

Now, according to http://www.powervps.com/what.php that I quoted above this is not supposed to occur. It's the very reason we moved from a shared hosting model!

"top" consistently shows our node/vps using less than 10% of CPU and Virtuozzo 'resources' are always well within the the green. We should NOT be experiencing this problem, and NOT be affected by other nodes/vps's.

Now, another support response has just come in ...

[...]We ensure - as best we can - that things like this can't happen, by constantly load balancing all of our nodes. However, if a user suddenly gets many thousands of hits, this will affect everyone else.[...]
Once again, this contradicts http://www.powervps.com/what.php and that which I quoted above from that page.

Not very happy and would like for someone to explain WTF is going on? What's the point of a VPS if it behaves like a shared model and we can be so grossly affected by other resource hungry vps's on our node?

This is soooo disappointing, especially as it was only just a few days ago I was compelled to write this ...
http://forums.deftechgroup.com/showpost.php?p=2908&postcount=5

Now I'm just flabergasted that despite http://www.powervps.com/what.php it all just seems to be complete BS and this is not the case at all. Any VPS can and does affect other VPS's on the same node and bring our VPS to it's knee's!

:eek: :mad: :confused:

[elix]

Jeez..calm down there!

Virtuozzo is designed to make it seem as if no other VPS can affect your performance. But, there are cases in which there are flaws such as what you outlined above. I'm sure a staff member would have a better explanation...note that if ftpd was failiing--some of this may be on your end too.

[charles]

The technology does provide isolation. There are a few ways that other VPS can end up influencing your VPS, but not cripple it as you describe. Judging from your resource use today, I can only surmise that your VPS was only getting its guarantee at the time (or not much more that your guarnatee like you usually do), at which time your own resource use was high too, and thus you felt it so badly.

So please don't lay all the blame on us or virtuozzo here. Nothing is perfect, but it does provide isolation and guarantees.

charles

[netrider]

Jeez..calm down there!
Jeez ... have your VPS 'outage' 3+ times in almost as many days, with the latest being from something that just shouldn't happen and see how calm you are.

Virtuozzo is designed to make it seem as if no other VPS can affect your performance.
"seem" is not the language that http://www.powervps.com/about.php uses.

But, there are cases in which there are flaws such as what you outlined above.
What flaw is that? I didn't see myself, or the support response outline any flaw, just that it was a 'normality' for a VPS to be grossly affected by other VPS's. Once again, not what http://www.powervps.com/about.php tells us is possible.

I'm sure a staff member would have a better explanation
Hence my posting

note that if ftpd was failiing--some of this may be on your end too.
Or just further illustrates the problem. The chkservd daemon that runs on our VPS's couldn't even get a successful FTP login prompt (likely due to timeout, such as I specified with HTTP and SSH), so attempted to restart the ftpd daemon and couldn't do so because it was already running .. hence the alert for failure to restart it.

Thanks for your constructive :rolleyes: post anyways

[netrider]

There are a few ways that other VPS can end up influencing your VPS, but not cripple it as you describe.
Then how was it able to do so?

Judging from your resource use today
Which is?, and viewable where/how?

I can only surmise that your VPS was only getting its guarantee at the time (or not much more that your guarnatee like you usually do), at which time your own resource use was high too, and thus you felt it so badly.
Virtuozzo or "top" do not reflect that at all.

Virtuozzo:
CPU Usage
Resource Capacity
CPU 0.0% The resource is in Green Zone
Load Average 1.42, 1.31, 1.44

System Usage
Resource Capacity
System 31.46% The resource is in Green Zone


So please don't lay all the blame on us or virtuozzo here. Nothing is perfect, but it does provide isolation and guarantees.
Until further explanation otherwise is given, that's exactly where it belongs and where all indicators point at. What more do you want when support explicitly says the problems is due to another VPS?

[netrider]

From WHM...


<< 08-01-2005 08-02-2005 08-03-2005 >>
Note: These figures are averages since 0000 hours today.
Note: This script will not able to track cgi cpu/memory usage if you do not have suexec installed.
Note: Percentages are based on one cpu. If you have 2 cpus divide the number in half to get the percentage of all cpu power used
User Domain %CPU %MEM Mysql Processes
netrider netrider.net.au 2.65 3.46 5.2
Top Process %CPU 6.0 httpd [www.netrider.net.au] [/forums/?PHPSESSIDc7a274323b110473e9333105de270881]
Top Process %CPU 4.8 httpd [www.netrider.net.au] [/forums/index.php]
Top Process %CPU 4.4 httpd [www.netrider.net.au] [/forums/?PHPSESSIDc7a274323b110473e9333105de270881]
root 0.58 0.88 2.0
Top Process %CPU 5.0 /usr/sbin/exim -Mc 1Dzvlk-0005Fe-6C
Top Process %CPU 4.0 /usr/bin/perl /scripts/restartsrv_spamd --check
Top Process %CPU 3.0 /usr/sbin/exim -Mc 1DzptS-0004Zj-N7
mailnull 0.01 0.00 0.0
Top Process %CPU 1.0 /usr/sbin/sendmail -bS
Top Process %CPU 0.3 /usr/sbin/sendmail -t -i
Top Process %CPU 0.2 /usr/sbin/sendmail -t -i
sshd 0.00 0.00 0.0
Top Process %CPU 1.0 sshd: [net]
mailman 0.00 0.00 0.0
Top Process %CPU 0.1 /usr/bin/python2 -S /usr/local/cpanel/3rdparty/mailman/cron/disabled
nobody 0.00 0.00 0.0
Top Process %CPU 6.0 /usr/local/apache/bin/httpd -DSSL
Top Process %CPU 4.8 /usr/local/apache/bin/httpd -DSSL
Top Process %CPU 4.4 /usr/local/apache/bin/httpd -DSSL
s 0.00 0.00 0.0
c 0.00 0.00 0.0
m 0.00 0.00 0.0
cpanel 0.00 0.00 0.0
n 0.00 0.00 0.0
r 0.00 0.00 0.0
mysql 0.00 49.78 0.0
named 0.00 0.60 0.0


Looks like very low resource/cpu usage by our VPS.

[elix]

Then how was it able to do so?


Which is?, and viewable where/how?


Virtuozzo or "top" do not reflect that at all.

Virtuozzo:
CPU Usage
Resource Capacity
CPU 0.0% The resource is in Green Zone
Load Average 1.42, 1.31, 1.44

System Usage
Resource Capacity
System 31.46% The resource is in Green Zone



Until further explanation otherwise is given, that's exactly where it belongs and where all indicators point at. What more do you want when support explicitly says the problems is due to another VPS?
What VPS are you using? If that's a Power-1 or even a Power-2 that's pretty high load...also there's a button called "extended" in VZPP please look at that and if you expand the boxes you can see oomguarpages--check if you're hitting that. I bet support said that only because you're bursting as it appears to me and if another VPS is bursting too, you can't burst. I bet you're definately getting your guaranteed resources, however.

[netrider]

And from WHM again, the average for the whole day yesterday to give an indication of what our VPS's normal/standard resource usage is like. Certainly does not look like our VPS is anywhere near reaching any allotted/allocated resource/cpu usage, let alone peaking.


<< 07-31-2005 08-01-2005 08-02-2005 >>
Note: These figures are averages since 0000 hours today.
Note: This script will not able to track cgi cpu/memory usage if you do not have suexec installed.
Note: Percentages are based on one cpu. If you have 2 cpus divide the number in half to get the percentage of all cpu power used
User Domain %CPU %MEM Mysql Processes
netrider netrider.net.au 2.30 3.05 5.1
Top Process %CPU 8.0 httpd [www.netrider.net.au] [/forums/?PHPSESSIDd325c5e5d4ed28466e20e982c844e7f7]
Top Process %CPU 3.7 httpd [www.netrider.net.au] [/forums/]
Top Process %CPU 3.0 httpd [www.netrider.net.au] [/forums/]
root 0.59 0.88 2.0
Top Process %CPU 3.0 /usr/sbin/exim -Mc 1DzRcl-00011U-DG
Top Process %CPU 2.0 /bin/sh -c /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1
Top Process %CPU 1.0 /bin/sh -c /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1
mysql 0.13 40.46 0.0
Top Process %CPU 0.2 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --pid-file/var/lib/mysql/host.netrider.net.au.pid --skip-locking
Top Process %CPU 0.1 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --pid-file/var/lib/mysql/host.netrider.net.au.pid --skip-locking
mailman 0.00 0.00 0.0
Top Process %CPU 0.1 /usr/bin/python2 -S /usr/local/cpanel/3rdparty/mailman/cron/checkdbs
nobody 0.00 0.00 0.0
Top Process %CPU 8.0 /usr/local/apache/bin/httpd -DSSL
Top Process %CPU 3.7 /usr/local/apache/bin/httpd -DSSL
Top Process %CPU 3.0 /usr/local/apache/bin/httpd -DSSL
s 0.00 0.00 0.0
mailnull 0.00 0.00 0.0
c 0.00 0.00 0.0
m 0.00 0.00 0.0
cpanel 0.00 0.00 0.0
n 0.00 0.00 0.0
r 0.00 0.00 0.0
sshd 0.00 0.00 0.0
named 0.00 0.60 0.0

[charles]

Run top. You have 74 instance of mysql alone running and 18 httpd right now. I monitored your CPU use earlier and you were in the top 5% consistantly.

But I am also not finding your site totally unresponsive as you said it was above (i wasn't around earlier - just telling you what I see now).

High I/O use is the only thing that we have found to have this effect, and its not something you can measure. Backups are still running on your server this morning, which are a contributing factor. Judging by the time of your tickets, your VPS was probably being backed up then.

charles

[Tony]

Heh, I just checked for myself, WHM info only shows hardware node memory, not CPU. For some reason, I was thinking otherwise at the time.

[elix]

As I can see now, You're definately bursting and you have a single site using 8% of the node which is quite a bit for a single site...don't blame PowerVPS..

[elix]

Appears that there's a WHT thread.

http://www.webhostingtalk.com/showthread.php?s=&threadid=430397

[netrider]

What VPS are you using? If that's a Power-1 or even a Power-2 that's pretty high load...
Exactly my point. "top" (when watching for around 10 minutes) shows CPU usage of no more than 10% and usually around approx 2%, yet the load averages are so high (they are normally around ~0.2 when looked under normal conditions and VPS response time). The WHM server status averages I posted also show we are well within resource/cpu usage metrics/indicators.

also there's a button called "extended" in VZPP please look at that and if you expand the boxes you can see oomguarpages--check if you're hitting that.
Would like too .. but 3 attempts to login/access Virtuozzo in the last 10 minutes and it just timeouts before the page even begins to load. Previous checks that I have done in the past under normal conditions and response times shows that we have been nowhere near any of these metrics/limits.

[charles]

Aren't we lucky. :)

Ron's VPS was completing a 4G *incremental* backup when I was alerted about the issue with this server this morning. Yes it took a day to move Ron's vps off the server that failed the other day. He has something very odd with his VPS (were looking into it, but my current guess is a very large number of files) that caused problems with the migrate, and it seems backups too, given todays fiasco. I can appreciate his concerns, but I personally think its a bit hasty to be running off to WHT asking if he should leave powervps.

Jason, your VPS is being migrated off this server. Power Panel access is restricted during migrates.

charles

[netrider]

Run top. You have 74 instance of mysql alone running and 18 httpd right now.


top - 00:26:54 up 4 days, 30 min, 1 user, load average: 0.12, 0.35, 1.21
Tasks: 116 total, 1 running, 115 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3% user, 0.7% system, 0.0% nice, 98.9% idle
Mem: 4122996k total, 4021032k used, 101964k free, 358732k buffers
Swap: 8193140k total, 3227244k used, 4965896k free, 1346368k cached


Many instances running doesn't mean high resource usage.

[netrider]

As I can see now, You're definately bursting
Based on what metric? And why?

"top" and virtuozzu both show very minimal CPU usage, and resource usage is all green. If we are burtsing, why is this not reflective in either of these? Why do we have only one active process, and the resting sleeping yet we are bursting?

[elix]

Based on what metric? And why?

"top" and virtuozzu both show very minimal CPU usage, and resource usage is all green. If we are burtsing, why is this not reflective in either of these? Why do we have only one active process, and the resting sleeping yet we are bursting?
Top will show the usage of the hardware node...why do you not understand that? You're looking at the incorrect place and you did not take my advice of looking at the Extended portion of Virtuozzo which may show that you're bursting over your memory limits. I certainly bet you are with all those MySQL processes.

[Robert]

Top is not fully accurate inside of a VPS. The individual processes and the overall load is not for the hardware node itself, but it does not reflect the amount of CPU you have available for your package.

That's where the PowerPanel comes in.

Either way... there is a support ticket open and our staff is working on it. I would ask that if you still need help, please use the ticket system. We're always happy to help.