Another server performance ticket

[Ron]

My server has had very high load, no idle time for an hour and 40 minutes, and now is virtually unresponsive; it took 10 minutes to edit my .htaccess file.

Ticket 13485427

[JPC-Zachary]

My server has had very high load, no idle time for an hour and 40 minutes, and now is virtually unresponsive; it took 10 minutes to edit my .htaccess file.

Ticket 13485427

Thank you for the heads up. I have handed this over to a member of our support team. They should respond back to your ticket shortly.

[Ron]

Thanks.

Took another 10 minutes+ to edit the "sorry" page I was trying to redirect people to instead of letting them try to use the forum.

After I got that done, I was able to get an uptime...

$ uptime
 15:33:21 up 145 days, 17:27,  3 users,  load average: 115.74, 166.08, 159.36

[Ron]

Support is saying that things are "under control" but I'm not seeing that at all:

2012-02-20/16:19:19
Hi;

We have killed some stuck processes on the server and monitoring the server for any further issues. The load is currently under control now. We will keep you updated in this ticket.

But 10 minutes later I can see that the load has not been under control, at least not for any length of time at all:

16:32:10 up 145 days, 18:25, 4 users, load average: 24.74, 13.66, 14.96

and another 15 minutes later

16:46:53 up 145 days, 18:40,  4 users,  load average: 22.84, 32.49, 26.88

[JPC-Zachary]

Support is saying that things are "under control" but I'm not seeing that at all:


But 10 minutes later I can see that the load has not been under control, at least not for any length of time at all:

16:32:10 up 145 days, 18:25, 4 users, load average: 24.74, 13.66, 14.96

and another 15 minutes later

16:46:53 up 145 days, 18:40,  4 users,  load average: 22.84, 32.49, 26.88

I'm being told that it is still be monitored and that it is still coming down. The last reading was at 6.

[JPC-Katrina]

Hi Ron,

What you are seeing is a raid rebuild. Please keep an eye on this notice for more info: Maintenance -- Manticore.nocdirect.com

[Ron]

"still coming down", is kind of an odd way of describing what I saw and what the stats show above over the course of 30 minutes, but once root stopped being idle (after I posted here and in a ticket) it has come down again.

EDIT Up again...

 17:26:06 up 145 days, 19:19,  2 users,  load average: 11.90, 10.07, 9.74

[Ron]

Hi Ron,

What you are seeing is a raid rebuild. Please keep an eye on this notice for more info: Maintenance -- Manticore.nocdirect.com

Ahhh, ok, that makes sense.

Thanks.

[Ron]

Ya know, if the people communicating with me and Zach knew that was the issue and not some "stuck processes" or "load is coming down and being monitored" and communicated that... I'd be happy and grateful for the technology instead of annoyed.

[JPC-Katrina]

Hi Ron,

The rebuild is not what started this. Our tech killed some stuck apache amd exim processes, removed frozen messgaes from exim queue, restarted the apache exim and mysql then it went into raid rebuild. Load started coming down after the previous stuff until it went into the rebuild.

[Ron]

Ahhhh, OK, my apologies.

21:02:12 up 145 days, 22:55,  1 user,  load average: 142.27, 97.44, 47.51

yeeee haw!

[Ron]

With performance like this:

 23:21:57 up 146 days,  1:15,  1 user,  load average: 120.75, 130.18, 112.81

Maybe just take it off line so the rebuild gets done?
It's worthless like this, and if it's slowing the rebuild it's counter-productive, IMnsHO

[Bell thorpe]

I was just about to open a ticket on this. Glad to know what's happening.

[Ron]

Hey guys, I don't wanna jinx anything, but manticore is running well now.

Spamassassin is behaving differently. it is now rewriting the subject line, prepending "***SPAM*** " on the subject line.

1st off, why the change?
2ndly, I thought that rewriting the subject line was impossible given the setup CPanel used?

Some of the changes in SpamAssassin as confirmed by clients:
[...]
2) Spam emails can no longer have their subjects rewritten. The "rewrite_header subject" configuration parameter in the spamassassin user_prefs file no longer works (either through CPanel or through editing the user_prefs file directly).

This is not the first time stuff has changed with SA, not the second time, either.

OK, WTH is going on now with spam emails???

Between 12:26 and 12:30 ET (specific enough???) the *** SPAM *** rewrite in the subject message went away after being here for 10 weeks, but the spam "Report" is still showing up on emails where the spam-flag=no.

Why are these changes being made? CPanel updates?

Or even the third time

Up until a week or so ago, it was working fine, and I had it set to rewrite the subject line.
[...]
then SA got upgraded and I lost that capability.

If the flow has changed, can This incoming email flow chart thread please be updated with the correct flow? Since I'm asking, can it be the flow for manticore?

Thanks.

EDIT: BTW, this changed sometime between midnight and 8:35AM this morning.
SA-spam-subj-rewrite.png

[Frank Broughton]

Hi Ron,

The rebuild is not what started this. Our tech killed some stuck apache amd exim processes, removed frozen messgaes from exim queue, restarted the apache exim and mysql then it went into raid rebuild. Load started coming down after the previous stuff until it went into the rebuild.

Huh? How can killing some processes, removing mail from the queue, restarting exim and mysql cause the op system to go into raid rebuild? I think you missed what they told you Katrina.