Huge loads time to time

[Ozz]

For 1 week there is a huge load on my vps time to time. Normally there is no problem vps got 70-100 processes and 0.50-1.00 load average.
But estimately 20 times in a day processes goes 200-30 and load average goes 80-150 for only 30 minutes. Then goes normal again.
ps -aux shows mostly Apache processes.
Is it a DDoS attack or something else?
How can I investigate it?
How can I get attacker IPs to ban them?
Is it a way to prevent attacks? Changing IP or stopping a service?
I searched and read about that attacks and no one could find a proper solution.
Thanks

[Milovan]

Hi Ozz,

This indeed sounds as some type of an attack? How many sites do you have on your VPS? If you would stop Apache (httpd), there is a good chance that your VPS would calm down.

What you could do is investigate which site is being hit and have it suspended. Which IP is getting attacked?

What do you get when you run:

netstat -nap | grep :80 | wc -l

by doing:

netstat -nap | grep :80

you'll see a list of IPs connected to your VPS, on which IP? From there, you could also have these IPs block (though it just might be that there is a huge number of IPs out there). Do you have a control panel?

[Ozz]

Only 1 site I got on my vps. It is big traffic site but there was no problem for 2 months.

netstat -nap | grep :80 | wc -l gives for 2 seconds interval: 526-740-809-1039

With "netstat -nap | grep :80" I got so many Ips. How can I decompose them? How do I get malicious Ips out? Conditions like "TIME_WAIT" or "ESTABLISHED" are important? Or maybe port numbers.

Also which log files are useful for this investigation?

Dehe support setup a software firewall (I think) called APF. Can I get any information from that?
Yes I have Cpanel with WHM and can block IPs there or I can use with .htaccess.

Sorry for bothering with so many questions but I want to get rid of this problem.

[RaZor Edge]

Hi Ozz,

On wich VPS are you: starter, business or enterprise?

I'm facing a situation pretty much like your's since the last two weeks. I have pass very much time investigate this and I (and support) have come to the conclusion that this was not an attack.

I would like if you can keep us updated if you fix your problem (or you can PM me if you want).

[Milovan]

Ozz: what is your VPS ID? Do you already have a ticket opened about this?

Razor Edge: As I said in the ticket, I am in the process of optimizing your VPS.

[Ozz]

My id is 270 and yes I opened some tickets but the main load problem mentioned above is going on.

[Andre]

You both are having similar load problems and they are not being caused by our services, they are being caused by what you run inside the VPS. If you try to make 3 cooks prepare dinner for 1000 guests, dinner isn't going to be served too fast.

RaZor Edge I already told you quite a lot of times before: you need a dedicated server, there is almost no way around that unless if you can find a way to lower your resource usage.

Ozz: Im personally not aware of what is causing the load on your VPS but Milovan will get this investigated shortly. What I can see on the node is that your VPS is using quite a lot of RAM, and also has a lot of CPU peaks. If you already have a ticket open about this please let us know the ID, and if you don't then please make sure to open a ticket.

[Ozz]

I want to ask about this TOP stats:

 04:37:07  up  5:52,  1 user,  load average: 92.22, 50.83, 37.95
220 processes: 212 sleeping, 1 running, 0 zombie, 7 stopped
CPU0 states:   0.1% user   0.1% system    0.0% nice  99.3% iowait   0.0% idle
CPU1 states:   0.2% user   4.4% system    0.0% nice  94.3% iowait   0.0% idle
Mem:   524288k av,  343424k used,  180864k free,       0k shrd,       0k buff
      1655160k active,             218280k inactive
Swap:       0k av, 18014394214524280k used, 4294957704k free  0k cached

Why CPUs are mostly in "iowait" condition?
Swap usage is interesting too.

[Andre]

About that iowait: that means your CPU time is waiting for disk output. Apparently your VPS is using a huge amount of disk reads and maxing out what's available. Keep in mind that the CPU usage of each VPS is individual, so while the IOwait of your VPS is high that doesn't necessarily mean that other VPS's on the same node have that problem too, it just means that your VPS is trying to read/write (IO) a lot of disk information.

About the swap: hm that's weird, that has to be an output error (it might even be caused by the huge load on your VPS). I would recommend to open a ticket about this then our techs can take a look at this for you.

[Ozz]

Yes I remember the swap file started looks like this with this load problem.
Ive just opened a ticket.
Thanks

[RaZor Edge]

On VPS 248, I've got the same Swap stats. We must be on the same node...

14:17:04 up 3:47, 1 user, load average: 0.19, 0.34, 0.72
53 processes: 52 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 0.1% user 0.1% system 0.0% nice 0.0% iowait 99.3% idle
CPU1 states: 0.3% user 0.2% system 0.0% nice 0.0% iowait 98.4% idle
Mem: 524288k av, 135752k used, 388536k free, 0k shrd, 0k buff
1474884k active, 336652k inactive
Swap: 0k av, 18014394214514988k used, 4294966996k free 0k cached

I also got more often high IOwait status...

[RaZor Edge]

Why CPUs are mostly in "iowait" condition?

Ozz, do you still have this IOwait problem? I hope this will be fix soon!!!

[Ozz]

Yes it is going on and also swap problem too. Dehe support team said Virtuozzo that is causing those swap numbers to appear like that and they would contact with Virtuozzo developers. And I am waiting for the solution. TOP stats same before like this:

 15:21:53  up 12:37,  1 user,  load average: 131.12, 114.27, 97.89
279 processes: 275 sleeping, 3 running, 1 zombie, 0 stopped
CPU0 states:   1.0% user   0.4% system    0.0% nice  98.0% iowait   0.0% idle
CPU1 states:  97.2% user   2.2% system    0.0% nice   0.0% iowait   0.0% idle
Mem:   524288k av,  322852k used,  201436k free,       0k shrd,       0k buff
      1231956k active,             623100k inactive
Swap:       0k av, 18014394214527840k used, 4294954144k free

[Andre]

Ozz: even though the swap usage is being displayed incorrectly, that doesn't relate to the load on your VPS. Your applications in your VPS are causing that. For any further information on this I would suggest discussing this with our support staff.

[RaZor Edge]

Hi Andre,

As I don't want to hijack this thread, I would appreciate if you or Milovan can give me an update about the problem we have discuss in troubleticket ACV-36672 (about strange high IOwait).