Argon?

[baldtechnologist]

What is up with Argon this morning? HTTP doesn't appear to be reliable at the moment, though other services are fine.

[Justin]

Hello:

Sorry this reply is a little late. The Argon server was having some major HTTP requests during the morning of the 21st which was crashing the Apache server. We are still monitoring the server to see if there is a responsible account or if we need to balance out the server by moving some accounts. The server is stable now but we are still in the process of fixing it for good.

[gerilya]

This server is having major issues with SMTP as well as WWW services lately.

If there is an HTTP attack in place that causes Apache to crash, could you elaborate what causing the second problem?

Thank you.

[Spathiphyllum]

Originally posted by gerilya
...major issues with SMTP as well as WWW services...
If there is an HTTP attack in place that causes Apache to crash, could you elaborate what causing the second problem?

I would venture to say the Apache app is getting overwhelmed with the DOS attack if one is indeed transpiring. As such it keeps spawning child processes to handle the requests and that eats up almost all available CPU time. So any other process, including SMTP, is not getting its share of normal CPU time either. Or a rogue script could be making too many GET requests from the sending end rather than the receiving end of the server and bogging down the system, too. Result: everything crawls to molasses and time dependent issues start compounding the problem.

Is this anything near the problem, Justin?

[gerilya]

Originally posted by Spathiphyllum
I would venture to say the Apache app is getting overwhelmed with the DOS attack if one is indeed transpiring. As such it keeps spawning child processes to handle the requests and that eats up almost all available CPU time. So any other process, including SMTP, is not getting its share of normal CPU time either. Or a rogue script could be making too many GET requests from the sending end rather than the receiving end of the server and bogging down the system, too. Result: everything crawls to molasses and time dependent issues start compounding the problem.

Is this anything near the problem, Justin?

I urge you to reconsider your post taking into account the following:
1. Apache has configurable limit of child processes to spawn.
2. If SMTP service is started before system resources are exhausted, it should still 'accept' incoming connection when the load gets high. The connections might time out later, but you wouldn't get "connection refused" error while trying to telnet to SMTP port.
3. The unusually large number of processes or connections is easy to troubleshoot. A 'rogue script' initiating too many connections from the same 'sending end' can and should be dealt with in a matter of minutes.

[Spathiphyllum]

Just scattershooting since we are diagnosing remotely with practically no information. But the exercise is useful.

The rogue script theory doesn't seem likely. Yes, finding its fingerprints would be pretty trivial assuming it didn't act up for a brief time and then go dormant. Even then it's probably not hard to spot and stop.

The "connection refused" error is new data... so from it some requests are certainly getting passed to the kernel. Maybe on a services restart during their diagnosis from original Apache travails other services were restarted and SMTP didn't get initiated because the HTTP flood was still happening. Nah, they would have spotted that. Hmmm.

How shared is this server? Maybe a configuration for some new accounts lacked Apache child server caps? Possible but again unlikely. Since they spoke of load balancing, maybe by coincidence a JAG netblock was being DOSed and a number of those targeted IPs were accounts coincidentally residing on the Argon server. But that wouldn't jive with the connection refused stuff unless JAG had temporarily shutdown some networking services for their diagnostics.

I guess the upshot is that I wasted time trying to figure this out so I'll quit while I'm marginally behind.

Have a nice weekend.

[gerilya]

We finally agree on something: I wish you a nice weekend too

[gerilya]

Just an update: the problem isn't resolved still.

As my request for more detailed explanation remains unanswered, my feeling that they just don't have any gets stronger.

This post was motivated by the recent load spike (which I posted in another thread, but I also want to get some attention to Argon):

-jailshell-2.05b$ uptime
19:29:39 up 14 days, 11:20, 2 users, load average: 137.55, 218.54, 222.83

[baldtechnologist]

HTTP on Argon is down yet again this morning... This really needs to be resolved.

[Joshua Clinard]

They're running a FSCK on Argon right now. The hard drive was just replaced like 3 weeks ago! This is getting out of hand. I am so glad I switched to another server. Jag's record has really gone down since the power switchover. I really hope it gets back to normal soon.

[baldtechnologist]

Yeah, I'm pondering a switch to a different server as well. Did you run into any problems or downtime as they moved the data?

[Joshua Clinard]

Actually, I wanted to reset my whole account, so I had them change my main account domain. Then I created a new Jag account with the original domain, re-uploaded all my files and databases, and then had them change my multi-hosted domains to point to the new server. I did this because I didn't trust the cpanel site mover, and for some other personal reasons. I did have to open a ticket to get them to move my mails from the old machine to the new one though. All went well, and there was very little downtime. I'm on Excelsior now.

But they are moving the hard drive to a brand new server tonight anyway, so I would just stay where you are if I were you.

[Justin]

Hey Alex:

The Argon server is now a new machine and seems to be running better but I still think some accounts will need to be moved off the server to lighten the load. If you do want to be moved please fill out a support ticket and the techs will move you to a stable server. There shouldn't be any downtime with the move, both server accounts will stay open until the DNS resolves to the new server. Thanks

[gerilya]

Originally posted by Justin
Hey Alex:

The Argon server is now a new machine and seems to be running better but I still think some accounts will need to be moved off the server to lighten the load. If you do want to be moved please fill out a support ticket and the techs will move you to a stable server. There shouldn't be any downtime with the move, both server accounts will stay open until the DNS resolves to the new server. Thanks

Justin,
we were interested in moving to another server, but we were advised that IP would have to be changed.

Unfortunately, it's not option for us at this time as we have IP address hardcoded into software licensing module. We are working on having all our clients upgraded, but for now we will have to leave everything unchanged unless there is way to keep IP address after the move.