Opinions Needed - Def. of Server Load

[Vin DSL]

I've been doing this stuff for quite a while, and I've come to the conclusion that nobody really knows what 'server load' means. That is, I've noticed that 'server load' means different things to different ppl, but mostly low is good, and high is bad. On that, I think we can all agree.

To my way of thinking, 'server load' is like a virtual checkout line at the store, with processes patiently waiting their turn. The longer the line, the larger the 'server load' number, expressed in a decimal. Still, 'server load' averages are an important indicator of what's going on inside 'our' servers.

I've been having a few sporadic, speed-related problems on 'my' server lately. Here's an example...



If you'll notice, I am monitoring several important things here. The one thing missing is 'server load'. The reason it's important is this...



The problem today was, the 'server load' was abnormally high!

So, I was sitting here tonight, thinking about all this - and thinking what a pain in the butt it is, having to shell into my account every time I want to check 'server load', blah, blah, blah, when it hit me. Why not add 'server load' to the rest of the things I monitor? This is what I came up with (idea taken from some Sun Solaris code)...



Here's where I need your opinions...

I don't want to have a 1 min, 5 min, and 15 minute average sitting on the bottom of every page, so I decided to take the 1 min average and express it in a percentage. That's where it gets a little tricky...

Personally, I think 'server load' should NOT go over 2.0 on a dual processor motherboard. Hence, I would judge 2.0 to be 100% 'server load' capacity on JagPC servers, even though I've seen 'server loads' as high as 60 here. Accordingly, I came up with a formula to make a 'server load' of 2.0 equal 100%, e.g. an 0.44 1 min average equals 22% 'server load'.

LoL! Does that make sense to anyone but me?

If you have a better idea on how to measure 'server load', let me know...

[Spathiphyllum]

Did a little research and found some nice work by Dr. Neil Gunther:

http://www.teamquest.com/resources/gunther/ldavg1.shtml
http://www.teamquest.com/resources/gunther/ldavg2.shtml

Excerpts:

The first point is that load in this context refers to run-queue length (i.e., the sum of the number of processes waiting in the run-queue plus the number currently executing). Therefore, the number is absolute (not relative) and thus it can be unbounded; unlike utilization (aka "load'' in queueing theory parlence).

...the magic number are really just exponential decay and rise constants expressed in fixed-point notation.

More formally, the UNIX load average is an exponentially smoothed moving average function.

So it is a smoothed moving average of the sum of all processes running and in queue with an unbounded maximum and a minimium of zero. It IS NOT a relative computation but IS a reflection of an absolute metric calculated by the kernel to reflect processes currently running and the excess requests waiting to be performed.

I don't know if trying to munge this into a percentage is really reflective of what is going on in the CPU. I had always associated load with 1.00 being the maximum load for a single CPU with the excesses above 1.00 being a multiple of maximum CPU computational ability waiting to be performed and beyond the capacity of the CPU to process. According to the good Dr., this is wrong.

[Vin DSL]

Yep, exactly!

To digress, it's like when I hear some whiney little girl or 'girlie-man' say, "That's not fair!"

The first thing that comes to my mind is, in order to say something isn't fair, you MUST establish a basis of comparison. In other words, if you wish to make a declarative statement that something is NOT fair, you should be prepared to show something that IS fair in comparison, no?

And, so it goes with 'server load'. In my estimation, I would judge 2.00 to be the maximum or 100% 'fair' amount, e.g. <2.00 is more than 'fair', and >2.00 is 'unfair.'

In the example above, it took almost 2 minutes to generate a page, and that's not 'fair'. Someone else, on 'my' shared server, was hogging the entire (acceptable) queue size, and then some, affecting the entire server, no doubt. Either that, or someone or something was attacking the server.

If I would have had that indicator in place, it would have shown 'server load' at 399%, and would have provided an immediate explanation as to WHY it took 2 minutes to generate a page that should have taken 2 seconds.

Anyway, thanks for the input. It confirms what I already believe to be real and true. I think I'll beta test it a while before releasing it to the community.

I've had a lot of requests for that footer script, but I was never satisfied with it. I always felt like there was something missing. Now, I know what is was...

[Ron]

A rule of thumb used for years is a run-queue depth of 3 for every processor... so on a dual CPU a non-maxed-out system would be under 6.

Rule of thumb. Take due notice thereof.

[Vin DSL]

Hrm... okay, I'll try that.

It yields a more realistic number on my page. That would make the event, above, 135% instead of 399%.

What thinks you?

[Vin DSL]

Here's a side-by-side with the reformulation...





That would have been 27% 'server load' before...

[Vin DSL]

Hrm... I was just surfing around, and I found this:

Another Rule of Thumb...

System Load Average (CPU)

> 10 - very bad

4-7 - fairly heavy

<3 - light load

Single user station <2

That's even more reasonable.

I noticed my web site starting to slow a few minutes ago, and my 'server time' indicator said 59%, so I guess it was up in the >3.00 range.

I think I'll recalculate the results so 4.00 equals 100%, and we'll see how that goes...

[gerilya]

I don't think showing 'uptime' on EVERY page would be a good idea.

Better write a script to be called from cron every few mins and have that script log a current load in file/db and send you an e-mail if the load reaches certain level.
Then generate the graphs from the gathered data and post it here

[Vin DSL]

I don't think showing 'uptime' on EVERY page would be a good idea.

Better write a script to be called from cron every few mins and have that script log a current load in file/db and send you an e-mail if the load reaches certain level.
Then generate the graphs from the gathered data and post it here

Hrm... yeah, that's a good idea too...

However, I'm doing this so I can see what's going on right now - this moment in time. If we had BASH access, then I wouldn't have to resort to parlor tricks, but that's the way it goes.

What I'm doing is using a BASH construct, via the proc file system, to gather this information. I'm not calling 'uptime'. LoL! Can you imagine parsing 'uptime' output and trying to make something useful out of it?

[Vin DSL]

Woo hoo!!!

Look what I ran across: http://www.tldp.org/LDP/abs/html/

That's a keeper!

[Ron]

Problems with rule of thumb for run-queue depth--

Short term spikes into "unhealthy" areas mean very little, other than the system was working hard for a moment. A consistently high number is more of a concern, and a slowly growing number indicates the machine is falling (further) behind. That's why they show the three different timeframes.

"System health" and load is better described by usage parameters (eg response time, wall-clock execution length, etc.)

memory utilization, paging system, cpu idle, and disk utilization are far superior numbers to look at rather than "load" . (I miss the ability to look at what is happening on the system, eversince jailshell... but I'm much more grateful for the security.)

Disk/memory bottleneck will raise run-queue in some systems while CPU will be quite idle. We've also seen components having diffculty (notably Apache) in turning around a request while server "load" is nil. Your server load climbing while your page generation is climbning suggests to me that your code/query is causing a large amount of work to be done by the SQL server, which is causing the "load" to increase on the machine.

This points me to a problem with SQL server, not with machine load. Either your query is causing problems intermitently, or there is a problem with mySQL. I am 50-50 on which I'd investigate first, since your page is so involved. Can you put some code in your page to dump out to a file when page generation > some time, say 2 seconds? Dump out the variables that will allow you to recreate the scenario.... # users, what type of user this is, is there a difficult rDNS lookup for your spybox, are there a ton of users logged in from the same IP (bots, etc.) causing a huge processes table, etc., etc., etc.
Perhaps a bot is trying to index your search page, doing search after search....
With that info, maybe someone who is as detail oriented as you are can actually nail down what the problem is on these machines.

[gerilya]

What I'm doing is using a BASH construct, via the proc file system, to gather this information. I'm not calling 'uptime'.

I didn't realize /proc is accessible from the jailshell, but either way you are spawing an extra process per page for each user. Why would you need BASH anyway? You can read files directly from PHP.

LoL! Can you imagine parsing 'uptime' output and trying to make something useful out of it?

That's actually not a big deal once you get familiar with Regular Expressions.

[Vin DSL]

My site doesn't get that much traffic. Look at the screenshots. 8 pages views in 5 minutes, 38 in 5, 34 in 5, and so forth There is NO reason for 117 second page generation times, or a 7.97 'server load'. Those things aren't being created by anything on my site.

The bots seem to be pretty well mannered also. Sometimes they'll hang around my site for 2-3 days, but that's because they're working DMOZ, not me.

As far as BASH is concerned, while I appreciate the added security (from buckwheats), I wish 'they' would realize that some ppl on JagPC know what they're doing, and we're NOT a threat. To the contrary, if we had access to PS and TOP, we would probably add to the security here. If you'll pardon the pun, we could be like Jailshell trustees or something.

Anyway, I don't mean to complain. I know if I want BASH, I should colo or whatever. I'm just trying to get some ideas on how others feel about 'server load'...

[Ron]

Ahhh well, I tried.

OK, one more try.
If you think it's has nothing to do with your own site, then why are you posting here instead of opening a ticket?

It may have nothing to do with traffic volume. It likely has to do with something either in your query, in your data, or a problem with mySQL. The fact is that you don't know whether the server load you are seeing is causing the slowdown, or the slowdown is causing the server load to go high.

You can continue to whine and bitch about page generation times (just like you did when your external feeds were the culprit) and play "guess the culprit" games, or you can try some sage advice on how to figure out what is going on.

Unless you'd like someone from the forum to hack into your site and take it over to do the job for you. Either do it yourself, or open a ticket. Or you can throw more hardware at it and buy a semi-ded account like FreeFall did without understanding the problem. He is apparently still seeing the same, and similar issue.

Good luck to ya-- I hope the same thing doesn't happen to me too.

[Vin DSL]

OK, one more try.
If you think it's has nothing to do with your own site, then why are you posting here instead of opening a ticket?...

Um...

- - - Tech - - - : Arun 2004-09-18 / 12:57:21
Hi,

With regard to the load being high on neutron,it was a temporary issue and we are keeping an eye on this.Hopefully,the server should continue with no problems.

If you have any further queries, please feel free to contact us.
-------------------
Regards,
Arun
Jaguar Technologies