httpd causing cpu peaks

[Spice]

Hi!

I have a VPS account and I'm having problems with cpu peaks that sometimes put my websites unaccessible. It's not rare to find CPU usage around 80% - 90% and it sometimes gets to 100%.

I have 5 websites hosted there currently but only one of them is having regular visits (around 5k uniques/daily).

As I'm a complete newbie to VPS administration I'd like to know if anyone can please help me with this. Does anyone has any idea about what may be causing this to happen?

Also, what does that "user 48" that appears in the image mean? Who is user 48? (Please click on the image to amplify it)

Thank you in advance!

[jason]

I'm not sure about the CPU usage as I haven't done enough poking around the environment yet to have a good handle on how things work.

User 48 is most likely the UID of the Apache user. If you checked /etc/passwd you would probably see something like

apache:*:48...

--Jason

[Spice]

Please click the picrute for detailed view

Thanks for the answer, I learned something!

Some more questions...

I've been looking into my log files (please see picture) and I found that there is an IP address (69.73.157.4 - made a whois and discovered it is from JaguarPC and it has a domain hosted there - extremegoogle.com) that tries to run some cron jobs on my server, and I also discovered that my server is getting a lot of internal dummy connections (maybe caused by the script I'm using). Those are generating corresponding 404 and 403 erros (respectively) in the error log.

The questions I'd like to ask are:

Can this be causing the cpu peaks as it is occurring every few seconds?

How can I stop that IP (69.73.157.4) from hitting my server time and time again trying to run scripts that are not in there?

Thank you in advance!

[Connie]

Cant' you just block the IP using .htaccess? They will get a 403 error but I don't think that's a big deal as far a resources are concerned.

[jason]

I think I recall someone else wondering about the constant hits from localhost a couple months ago. I don't know if the reason was ever found and/or revealed, but my guess is that it is some kind of service availablility check that your control panel is doing to ensure everything is running. It is probably hitting other services, too, like email and ftp.

As for the other IP, I doubt it is really causing a problem since the hits are just resulting in 404 errors. It looks as though that IP is assigned to a VPS or dedicated server (rDNS doesn't return a nocdirect hostname). Perhaps that site once had your IP and a poorly written cron job is still trying to do things with it. Since JPC does have control of the server I'd report it to them. They can contact the site owner to alert him to the issue and have him correct it.

--Jason

[Spice]

Thanks for the replies!

I'm going to contact Jag support to see what they can do about that IP that's hitting my server but in the meantime I'd like to block it via .htaccess but, for some reason, I think I'm not being able to do it.

Shouldn't it be done like this and then uploaded to httpdocs folder for the domain in question?

order deny,allow
deny from 69.73.157.4
allow from all

The reason I say that I'm not being able to do it is because it still appears in the access logs trying to run the crons and getting 404's, is this normal even when it's blocked via .htaccess?

As for the dummy connections, I think that maybe caused by the script I'm using. I'm going to get in touch with the developers and make a clean reinstall soon and see if it keeps happening.

Thank you one more time for being so helpful, everybody!

[wuurp]

As for the dummy connections, I think that maybe caused by the script I'm using. I'm going to get in touch with the developers and make a clean reinstall soon and see if it keeps happening.

The dummy connections are part of apache 2.2+ and can usually be ignored.

http://wiki.apache.org/httpd/InternalDummyConnection

[jason]

What status code is being show with the more recent hits from that weird IP? I suspect they will still get logged, but should be shown as 401 (access denied) instead of 404 (not found) for that IP.

--Jason

[Spice]

Thanks Wuup! It maybe an Apache thing after all.

That IP that's hitting my server is still getting 404's for the pages that it tries to reach. I don't think it's blocked but it should be. I've created a rule on the firewall to drop connections from that IP so...

I also found something that maybe weird when looking to the error logs: it looks like there was some error that made it something restart by itself (please see picture) and generated some warnings (maybe some wrong configuration?)



Thanks once again for your help, everybody!

[jason]

Without seeing the line(s) above the restart message I can't say for certain why Apache restarted, but since it is a "graceful" restart that means that Apache was simply reloading its configuration and not really shutting down completely. Did you make any changes to your configuration (such as adding an account in Plesk or something like that)?

I'm not sure about the "Bad file descriptor: apr_socket_accept" messages. I tried googling on them and, while I found references to the message, I didn't find a reason and/or solution.

The RSA server certificate warnings are just saying that Plesk's generic certificate (used to allow you to log in to Plesk via HTTPS) wasn't issued to your actual hostname. This won't cause any performance issues (just browser warnings when you try to use the cert). The other SSL related warnings appear to be due to Plesk doing some non-standard things with SSL that Apache doesn't like, but again, it won't cause any performance issues.

The rest of the messages look perfectly normal. With the exception of bad file descriptor lines, I don't see anything wrong with Apache.

--Jason

[RickWeb]

Could it not be something as simple as a bad script running over time, or a bot stuck in a piece of coding ( i did that once caused alot of problems on draco) . What script are you running?

[Spice]

I'm running a blog script (Pivot 1.40.4 'Dreadwind'). I have thought about that and that's one of the main reasons I didn't contacted support yet. I want to reinstall it from scratch and redesign (recode) the template I'm using and see if this problem remains. I'm just waiting for an opportunity to do it since I've been having too many things to do lately. I may do it next week.

Thank you for all the replies!

[RickWeb]

They if that doesn't work slip a support ticket in, they mite be able to point you in the right direction.

[Spathiphyllum]

[QUOTE=Spice]...I also found something that maybe weird when looking to the error logs: it looks like there was some error that made it something restart by itself (please see picture) and generated some warnings (maybe some wrong configuration?)

[URL=http://img13.imagevenue.com/img.php?image=68702_error_log_04032008_1 22_859lo.jpg[/quote]
The thing that sticks out to me is the [error] notice you're getting:

Directory index forbidden by Options directive: /var/www/html/

Just prior to and just after the graceful restarts, your localhost [127.0.0.1] is making a GET request to the main server directory root, i.e. an attempt to read the directory and find a default index.html file to render. Your server is set to deny that request and throws an "error."

So, one question is whether there should be a default index file that should be getting returned from the general directory index request but isn't. Or should localhost be permitted to access directory indexing while everyone else is excluded. Or is your content management software, or blog, making repeated requests to the server root index and, being denied, repeatedly retrying after timing out only to eat up the secondary server processes as fast as the primary one can spawn off another process.

From the earlier posts, you notice how you have several httpd processes going, each using ~10% of the CPU time. Now, multiple http daemons is normal. The fact that they all have about the same amount of CPU time indicates to me that each child process is filling up with requests to their set max, then probably terminating only to be replaced by another child process until it too reaches its maximum set of connections. This cycling of the child processes because they are filling up too fast with improperly handled requests is proper from a server perspective, but from a serving resources perspective, it's incorrect. Your server, I'm concluding, is struggling to serve due to misconfiguration. It can serve HTTP, intermittently, but is flailing away at external requests because the localhost requests are chewing up the servers.

You may need to review your httpd.conf file and see if there is some error there. If you SSH into a terminal, try running:

service httpd restart

That will tell you if your configuration is syntactically correct. An 'OK' means all's fine, that at least the server will run with how it's set up. That doesn't mean the configuration is optimal, just that it's operational and serving. I'd probably check with the Options Directives first and confirm that they're acceptable with the CMS/Blog you're running. If that's OK, I'd start looking at the number of processes/connections/servers in httpd.conf to see if there's some tweaking I could do there to optimize your setup.

Those are for starters. The concerns over the external IP request (to be firewalled) seem trivial to me at the moment. The numerous dummy connections is abnormal, at least in my limited observations.

Good luck. If I have an Aha!, I'll update.

[Down2TheC]

My lord those numbers are nasty. You're not on fangorn are you? I've been chasing issues all week with my physical server getting it's proc util pegged while my VPS is taking 2-4%. Everything slows down including ssh login, CP startup, and all my domains.
Not saying you're the cause of my prob... but either way, I hope support can get your system leveled out.

Side question... output of the 'top' command shows the hardware proc util, right?