Website Performance Metrics?

[netrider]

What metrics or methods are people using for monitoring their VPS performance in serving web pages?

Some users are complaining that it can occassionally take up to 20-30+ secs for a page load to occur, when it's fine at other times.

I would like to identify if these 'slow-downs' are related to traffic lag, or related to lag in apache server (or backend mysql) in generating and sending pages.

Ideally, I would like to see graphing or counters on the time it takes for the apache server to deliver a page and how much of that is waiting for the mysql query results (if any) when generating the page, or waiting on CPU time etc.

Do our VPS's have MRTG setup with these stats or something similiar? What do others use to monitor the performance of their VPS? (and ultimately their website)

[esc]

Hi netrider,

I can see those delays too on my cPanel Power-1 plan and have done some (still running) investigations to find out what causes them. I didn’t do a consequent monitoring of the server but just some experiments with a stop watch and some server tools.

I have several very low frequency project info sites on this VPS and some sites for development purposes. This is what I see:
The delays are most notable when a site is called first time after some pause. The pause to retrigger the delays seems to be about halve an hour and above of no Web traffic on the site. I didn’t investigate into exact timing of the lower limits.
This delay can be from 20 to 30 seconds (as test pages I use the login dialogue on several Typo3 intallations which are heavy PHP coded).
The delays are independent of calling via domain or IP. So they are not caused by my nameservers. This is also provable by calling the nameservers with dig which shows the direct DNS timing. This is about 100ms for a full recursion which is acceptable for me.
As the startup delay is independent of open SSH connections to the server it can not be an ARP cache timeout on a PowerVPS border router as was speculated some time ago on these forums.
I investigated with tcpdump running on the VPS while calling Apache for the first time and had to learn that the TCP handshake is more or less instantaneous. Than the Apache server starts answering continuous several times per second. All these hits to show my test page sum up to the long delay till the page starts loading in the browser. On a new call (with the browser cache flushed of course) the serving is much faster and the page displays within 2 to 3 seconds which is about the same time as on my two non-PowerVPS test accounts.
The problem is not Apache related. I run a Zope/Plesk CMS on one of my accounts which is bare bone Zope (I do not use Apache as a proxy) and I see the same timing. 20 to 30 seconds on the first load (after some longer delays above halve an hour) and about 2 seconds next time. Zope is her much faster than Apache/PHP. So I probably will install some PHP accelerator in a while.
I investigated with the WHT Unix benchmark if the delay depends on the total load of the node. But while I can see here the other VPSs on the node working, as my burst resources are sometimes noticeably lower (I see benchmark scores between 15 and 50, mostly 35 to 45) the startup delay seems more or less independent of the current total server load. That means I see high startup times paired with high and low benchmark scores.
All facts put together, I think this points to either to some configuration problem (Virtuozzo) with memory swapping between the different VPS on the server or perhaps a bottleneck in the storage system. Do we have SCSI raid arrays?

Just my 2 cents.

Erich

[charles]

All facts put together, I think this points to either to some configuration problem (Virtuozzo) with memory swapping between the different VPS on the server or perhaps a bottleneck in the storage system.


I'll talk to SWSoft about this. I don't believe there is any configuration problem. My guess is going to be that if anything, they will say that linux is doing its job which is swapping (probably more like paging) out idle processes and then swapping them back in when they are needed.

This issue has been brought to them several times, and this was never identified as a potential issue though.


Do we have SCSI raid arrays?

Just my 2 cents.

Erich

Yes, all are.

charles

[elix]

I believe this is an ARP issue where if you don't have a steady traffic flow processes need to be open back up which casues a delay--I'm not expert, though...Hopefully Charles will have a fix from SW-Soft.

[Simps]

Sites on my VPS also have the "first visit" lag symptom but apparently this is not on a per site basis. The first site I look up comes up after a delay, but if I then access another site from the server it comes up in a flash.
Also, I'm not quite sure about this yet, it seems that it might be a per IP symptom as if I then access a site on the same VPS but on a different IP it suffers the lag. Anybody else experienced this?

Simp's

[Simps]

Ideally, I would like to see graphing or counters on the time it takes for the apache server to deliver a page and how much of that is waiting for the mysql query results (if any) when generating the page, or waiting on CPU time etc.

ab gives you this :

$ ab -n 20 http://xxxxxxxx.com/
This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking xxxxxxxx.com (be patient).....done
Server Software: Apache
Server Hostname: xxxxxxx.com
Server Port: 80

Document Path: /
Document Length: 2363 bytes

Concurrency Level: 1
Time taken for tests: 12.295 seconds
Complete requests: 20
Failed requests: 0
Broken pipe errors: 0
Total transferred: 50080 bytes
HTML transferred: 47260 bytes
Requests per second: 1.63 [#/sec] (mean)
Time per request: 614.75 [ms] (mean)
Time per request: 614.75 [ms] (mean, across all concurrent requests)
Transfer rate: 4.07 [Kbytes/sec] received

Connnection Times (ms)
min mean[+/-sd] median max
Connect: 141 142 0.4 142 144
Processing: 199 472 1090.4 216 5096
Waiting: 195 472 1090.3 215 5095
Total: 338 615 1090.4 359 5238

Percentage of the requests served within a certain time (ms)
50% 359
66% 362
75% 370
80% 380
90% 668
95% 5238
98% 5238
99% 5238
100% 5238 (last request)

[falsedawn]

This issue is also something that I have been noticing as well.
I've even had support move my account to a different node to no avail. The problem was still there. As has been noticed, it is worse when the site has been idle for a while.
Delays of 2 or 3 seconds are not uncommon.
One thing that is weird is that I have one of my sites password-protected, and when I access it in my browser, the password dialog comes up pretty much immediately, but after entering my credentials, it _then_ takes 2 or 3 seconds to display the landing page.
I don't know why there should be such a delay going from the password dialog to the landing page, but there definitely is.

On a couple of my (non-live) php sites, I am using php's microtime function to both time my script and query execution.
I have noticed a marked deterioration in these times since I joined less than a month ago.
I'll admit I am anal about performance, but when my account was opened, I was getting page parse times around 4 or 5 ms commonly. Now the minimum I ever get is 7 or 8ms. On the same pages, obviously.

Granted, not a lot in absolute terms, but percentage wise it's a great deal. Also, there seems to be a lot less consistency in the page parse times.
Even with eaccelerator, and refreshing the same page, it can go from 0.01s to 0.5 secs and back again.
As my account has virtually no load at the moment, I can only assume that this is due to other VPSs on the same node affecting my VPS. Hence, the VPS is being badly affected by other accounts, and the "private" in VPS is not really living up to its name...


Also, my query execution times are sometimes all over the place. I have tried loads of different mysql settings, but my databases are pretty tiny and low-stress, so whatever settings I try have minimal effect.

It worries me when sometimes a query execute in <1ms, and then I do a page refresh and it takes 100ms to execute the same query next time.
Often, this is a simple static SELECT that would have been cached, too.

I think PVPS need to be more open about just how many accounts they have per node, and how they are monitoring performance, because if this is related to the number of accounts on a node, I won't be happy.

Anyway, I am continuining to monitor the performance of my sites...

[charles]

I think PVPS need to be more open about just how many accounts they have per node, and how they are monitoring performance, because if this is related to the number of accounts on a node, I won't be happy.

Anyway, I am continuining to monitor the performance of my sites...

It has nothing to do with the number of vps per node, i can assure you that. Your performance can be affected by other VPS to some extent though. Please submit a support ticket referencing this thread. I can't help you without knowing who you are and what node you are on. Submit a ticket and I'll have someone look at your node and vps.

charles

[falsedawn]

With all due respect, I have submitted tickets about this before, and all I get in response is
"the load on your server is minimal and we do not notice any performance issues at our end"
If the problem is intermittent, I don't see that a lot can be done about it.

Same with the initial delay that others have noticed. If it is not reproducible, or if it doesn't affect the network that you guys check the site from, then it is hopeless for you to try to troubleshoot it.

I understand that other VPSs may have a small affect on each other, but when it occassionally takes a simple script 2 seconds (or more) to execute, then I am really at a loss to understand why. Could it be a process on my VPS? Could a process on another VPS affect mine to this extent?
Could it be something within php/eaccelerator? Something else?
Maybe it is just inherent in all web based apps, and it's something I'll have to live with...
I don't know, and I doubt very much if it can be tracked down as it might happen once every 100 page loads or less.

[falsedawn]

And some evidence that my site is actually running slower now - I log all the page parse times and just ran a query, averaging them on a daily basis. Obviously, this was not a controlled test, but most days involved a fairly representative mix of pages.

Results here:
http://www.fractalhost.com/parse_times.html

[Fred]

Can i ask some questions for you...
First thing before i get some flames :) English is not my native language... i'm not a pro with traceroute stuff, but i know the concept basics. I know the basics of the main protocols(http, dns, dhcp, ftp, etc)... Well, I'm a technician since one year and i did 3 years of computer science ( half of programmation, half of network ) in a school after high school ( It's not university... it's between high school and university in my province, we have to do it before university, we enter university at 20 or 21 years old )

So, I have the same problem as stated here. They wasn't able to find why or notice anything... I do believe them... it's hard to know what are these problem's causes...

In my case, it's only on php sites... html sites loads fast... EVEN during a delay on another site. I explain myself.
I tried this(and i ask you to do the same): open two tabs in firefox... I tried to load a phpbb forum i host in the first tab, if i don't get a delay, i reload it 5 or 10 secs later to see if i get a delay... mostly, before 5 reloads, i get a delay. PAF! if i get a delay, Hurry up and go to the second tab and try to load one of the html sites you host...
I tried it many times and the html site loads fast ... without any delay.
The php site finally loads... 5 seconds later...

I did it many times... and i was looking at winmrt at the same time... and it didn't get higher then 40 during the delay... So, imo traceroute isn't a problem at all...

All my experimentation tends to say that the problem is related with php. I run php as cgi with phpsuexec... I will probably try to disable it and try my "test" with firefox, top, winmrt.

The other thing i will try ... downgrade for an hour or two to php 4.3.10... and try my "test" ...

This weekend i will not have a seconds to test it... but monday, you could be sure i will experiment a lot of stuff to locate and isolate the problem.

Can you tell me if you try my "test" and what are the results ? are you using phpsuexec too ?

[esc]

Hi Fred!

I don’t think that this delay problem is PHP or even Apache related. As I explaind in my first post on this thread, I also see it on Plone sites which run on a Zope server. Zope is programmed mostly in Python (with a little C) and there’s no PHP involved.

When looking with tcpdump (or similar tools) into the low-level net traffic, you can see that during the seemingly delay in the browser there’s some *steady* communication with the server going on, but just slower than normal (measured in connection per second not in the time the transmission (per junk) takes when it is initiated). – A HTML page is not transmitted in one large piece but in a lot of smaller parts which are labeled as hits in the statistics. The point when the browser starts to display the page depends on its structure, but normally this does not begin with the first received junk.

All this seems to indicate that the server cannot provide the data in this situation as fast as it should. Why? The problem does not seem to be load related. The machines are high end Opteron servers with 8GB of RAM (at least the node where my VPS is on, they have a very fast storage system (SCSI raids, as Charles pointed out) and they are not overbooked. So I think one should investigate into the paging performance of Virtuozzo, the hosting system of the VPSs. Probably swapping memory junks in GB sizes will take some time.

I’m no Linux expert and I don’t know how to benchmark and tune paging but perhaps someone in the know might have a look at this and comment on it. Do we have several concurrent swapping strategies which might interfere? Namely those of the hosting and of the virtualized hosted servers? What is the performance we should expect and what do we see? etc.

There are several forums of SW-Soft the owner of Virtuozzo. If the problem is Virtuozzo or VPS related, it might have been addressed here too. Perhaps someone with more time at hands than me might investigate.

Erich

[KARanden]

Just to put in my unscientific "test & comment" on this.

I have one pure html site http://kardekaler.com that I did a "test" on now, after not using the net for about 30 min.
This html page comes up in less than 2 seconds, nice and fast.

The other site I tested on, is a Mambo site http://modellnorge.net
This site takes about 20-30 seconds to load. And I see on my network connection that there is no in or out traffic for about 15-20 seconds after the first few seconds after I start loading the page.

So it looks to me, that there is something going on with the VPS, if the web site has not been visited for a while.
Just reloading the Mambo site, after just a few minutes, while writing this, takes about 15-20 seconds.

I'm in Norway, so I know I have longer ping times than US users, about 130 ms.
But my previous host, a shared host located in Texas did not show this type of "lag" loading any of my sites. And that host have even slower ping times that my VPS.

I have ADSL 2048/256 connection on the Mac I use.

[charles]

We do take these problems seriously and are looking into this. We haven't heard back from SWSoft about potential paging/swapping issues yet, and as you know this isn't easy to track down. All I ask is that you be patient while we look into this, and also submit a support ticket so we can track this. You have no idea how difficult it is to help you here. I don't know who you are, or what or where your VPS is etc. etc.

If you don't like the response you are getting from support, please politley ask for it to be escalated. :)

charles

[falsedawn]

No worries - I know exactly how hard it can be to pinpoint issues likes this.

I've been very happy with the support so far - I wasn't having a go, just observing that there's not a lot that can be done until a more thorough investigation has been carried out.

Keep us informed!

[Hvu]

KARanden, Your mambo site does seem to load slowly. It took about 5secs to load here. My question is are you running any php cache or that sort? Also would you mind posting your phpinfo too. Maybe i can help you with a few tweaks here and there.

Here's a mambo site I run on my VPS >> http://oginteract.info

[KARanden]

KARanden, Your mambo site does seem to load slowly. It took about 5secs to load here. My question is are you running any php cache or that sort? Also would you mind posting your phpinfo too. Maybe i can help you with a few tweaks here and there.

Here's a mambo site I run on my VPS >> http://oginteract.info

Thank you very much for you offer to help. I hope it's OK that I come back to you at a later time, becaus Tony (support) did some changes on my VPS last night (my time) that looks good so far. But I want to "test" for at least a day longer before I say it's a good tweak or not.
And I'm sure Tony will let you all know if what he is doing can fix the slownes/lag for other users.