extremely high IO wait spike on server daily; failed attempts to escalate issue

[signde]

I've had a support ticket (#13441660) ongoing for 18 days. I've clearly identified the issue as an iowait spike that occurs daily at 6pm. This in turn causes load on my VPS to spike as IO is a very slow operation relative to CPU. I've repeatedly presented the daily iowait times in my support ticket numerous times. I've checked everything on my VPS instance and as far as I know this IO spike is not being caused by my instance. My only conclusion is that this IO spike is the result of some other VPS instance or the server node itself.

Over the duration of the ticket I've received incorrect analysis of the root cause, reports of corrections that have nothing to do with the iowait issue, and on more than one occasion went days without a response. I have also made repeated attempts to escalate the issue, all of which have resulted in a repeat of the above failures.

My only conclusion is that support either doesn't understand iowait, hasn't taken the time to read my assessment, or is so overworked they simply don't have time to care.

Given the amount of time I've invested providing rebuttals to the support staff and reiterating the root issue I could have migrated my VPS at least five times. I've been a customer for a long time with great service and I do not want to switch hosts but this entire experience has rendered that history null and void. I've tried to remain logical and courteous even in the face of all the above frustrations but my patience has run out.

I feel I have run out of options. This is my last attempt to voice my frustration over this issue and the only remaining source where I feel I can raise awareness of this issue. Is there anything that can be done?

Regards,
Frustrated and loyal jaguarpc customer of over 4 years.

[gohighvoltage]

Hey Bud, iowait spikes can be extremely hard to diagnose/trace. The fact that you say it happens at the same exact time every day, makes me think this:

Alot of times, Spambots can be set on a schedule. It is not uncommon for a huge amount of traffic can happen all at once from a major spambot, etc.

The iowait spikes could very well be from a few different scenarios. The fact that it happens at the same time, makes me think it that it is not a hardware problem.

Have you looked at your awstats? What is the traffic like at 6pm when the spike happens?

Also, what are you running on your server? Just websites, or is there databases/forums/directories?

[signde]

That is a good theory but one I've already debunked. I also mentioned that in my support ticket as they asserted the same cause initially.

I've already examined my web stats. There is no increase in traffic during the specified time. In fact traffic is actually lower at that time than at other times during the day. I've also monitored netstat during this time. Again, there is nothing there.

I don't think it is a hardware problem but rather a scheduled job either on the server node itself or another instance on my node. Regardless, I don't have the ability to inspect this at any level higher than my VPS instance so I'm flying blind hence my reliance on JaguarPC support.

[gohighvoltage]

Do you run a forum or a large database?

[signde]

Yes. The only thing hosted on my VPS of any significance is an invision power board forum. Before jumping to any conclusions about that, know that the forum is private, only has 300 registered users, and of those only 100 or so are active in the past year. The most active at any time in a 15 minute interval is 10-15.

I've looked through scheduled jobs in MYSQL, cron for every user including root, and scheduled tasks in the forum. There is nothing scheduled around this time.

[gohighvoltage]

Ok, LOL, You knew where I was looking. Hmmm, the fact that it is at the same time, makes me definitely think it is some kind of cron job, or maintenance running from/to the database.

If you have nothing running, then maybe it is another node on the server running some crazy intense database operation.

[signde]

I truly appreciate your feedback and attempts to help. This has been the most intelligent conversation on this subject I have had since opening this issue. I honestly feel I have already covered all bases once, twice, and three times over from my end.

I 100% agree with your assessment but unfortunately I cannot get support to understand this and my options to troubleshoot end with my instance. I get continued response such as "we are looking at this" or "we have made optimizations, please check now". How am I to "check now" when I've made it painfully clear the issue only happens once a day at a specified time.

So frustrated.

[gohighvoltage]

Hey Bud, I feel for you. I can empathize with your frustration. I know from experience that iowait problems have been issues on servers for years. I remember years ago, there was issues within the kernel, then there was cpanel issues that can cause it, then even mysql had updates to address causing iowait issues. They can be a bear to diagnose.

Just to definitely rule out your forum. Are you able to turn it off for an hour, while you watch for spikes? I remember one time I came across an iowait problem, and it was a conflict with a forum talking to the database. A mysql update/reinstall corrected it.

If you did this already, then You definitely covered everything. Has this just started happening? Have you recently updated anything? Cpanel, etc?

I am sorry to hear of how frustrating it has been for you. Let me put some thought to all this.


Do you have any server backups configured?

[gohighvoltage]

If all this checks out, I would escalate your ticket and ask support to move you to another node.

There may be other nodes causing an issue, and moving you will just rule one this thing they have to check to locate the problem.

6pm, is a HIGH peak for lots of forums. I bet it may be other nodes with very big forum/database, and dinner time is a perfect time for an overload to happen.

[signde]

Hey Bud, I feel for you. I can empathize with your frustration. I know from experience that iowait problems have been issues on servers for years. I remember years ago, there was issues within the kernel, then there was cpanel issues that can cause it, then even mysql had updates to address causing iowait issues. They can be a bear to diagnose.

Just to definitely rule out your forum. Are you able to turn it off for an hour, while you watch for spikes? I remember one time I came across an iowait problem, and it was a conflict with a forum talking to the database. A mysql update/reinstall corrected it.

If you did this already, then You definitely covered everything. Has this just started happening? Have you recently updated anything? Cpanel, etc?

I am sorry to hear of how frustrating it has been for you. Let me put some thought to all this.


Do you have any server backups configured?

I made it clear early on in my support ticket I have not ruled out there is something going on with my instance but if that truly is it then it is beyond my (what I would like to consider above normal) comprehension.

You bring up a point I've already considered. I've toyed with the idea of suspending my main VPS user account via WHM during this time to rule out my instance as the cause. I've refrained because I have been in constant holding pattern with support because they informed me they were monitoring the issue.

The last time I forced an upgrade of cpanel, apache, mysql was well before this issue started occurring. I've checked for upgrades since then and found no updates. I feel that is a dead end.

You've given me some renewed patience with the issue. Tomorrow I believe I will kill my user accounts and see what happens with iowait. If you have any other suggestions. please post them here.

Thanks for your feedback. I truly appreciate it.

[signde]

If all this checks out, I would escalate your ticket and ask support to move you to another node.

There may be other nodes causing an issue, and moving you will just rule one this thing they have to check to locate the problem.

6pm, is a HIGH peak for lots of forums. I bet it may be other nodes with very big forum/database, and dinner time is a perfect time for an overload to happen.

I completely agree again, but as I stated in the subject, my attempts to escalate have fallen upon deaf ears. I get a new tech, go through the same feedback loop and rebuttals, then fast forward to a dead end. Rinse, repeat, fail.

[gohighvoltage]

Hey Bud, I would disable your forum/database tomorrow at 5:30 and wait it out till 6:30. If no spikes occur, then it could be an issue with how your forum is talking to the database. Sometimes mysql can hang with connections open, etc.

If the spike happens even though your forum is off, then you can rule out most of what could be a problem on your end.

Some other thoughts, do you have persistent database connections turned on in IPB? If you do, turn it off.


Also, when the iowait spike occurs, how long does it last? What does the memory usage look like?

[gohighvoltage]

Also, what is your max connections set to?

Not that I think this is an issue. Most likely not. Cause you would get some kind of error message I would assume.

[Ron]

OK, this conversation has been nice and friendly and all, but nobody has enough info to diagnose anything yet.

You need to narrow down what the problem is for support (or move to another server) or else this could take forever to resolve. JAG support does not have the necessary monitoring tools to diagnose many of these kinds of issues.

You say load goes high on your server. I imagine you're talking about the numbers you see on top or on uptime. Is that correct? What are the actual numbers that you are seeing?

Can you post the command and results of a command like

$ sar 5 12
Linux 2.6.9-101.ELsmp (myserver.nocdirect.com)         10/20/2011

11:33:49 PM       CPU     %user     %nice   %system   %iowait     %idle
11:33:54 PM       all     32.30      0.00     10.90      8.10     48.70
11:33:59 PM       all     39.80      0.00     15.60     28.90     15.70
11:34:04 PM       all     29.57      0.00     11.19      5.39     53.85
11:34:09 PM       all     23.60      0.00     11.80      4.20     60.40
11:34:14 PM       all     20.30      0.00      7.30      1.40     71.00
11:34:19 PM       all     44.10      0.00     13.20      1.80     40.90
11:34:24 PM       all     60.40      0.00     25.70      0.20     13.70
11:34:29 PM       all     29.94      0.00      8.88     10.48     50.70

11:34:29 PM       CPU     %user     %nice   %system   %iowait     %idle
11:34:34 PM       all     26.75      0.00      9.02     15.23     49.00
11:34:39 PM       all     46.17      0.00     33.93      2.09     17.81
11:34:44 PM       all     31.20      0.00     15.70      3.50     49.60
11:34:49 PM       all     21.12      0.00      6.71      2.50     69.67
Average:          all     33.78      0.00     14.17      6.98     45.07

How is your real-world performance being affected?
Is your server responding slowly to http requests?
Is it responding slowly to forum requests?
How about flat file requests?
How about php instantiation cpu intensive processes requests?
How is php being executed, as a module or CGI?
How about non-db but still disk intensive processes?
How about memory intensive processes?
How about CPU intensive processes?
Are you running DNS on your server? Can you exercise the DNS resolver?

Let's start with those kinds of questions, and we can go more in depth as we narrow down your problem instead of throwing darts at the most obvious targets.

Also.... you could install some monitoring tools on your own, but they may be limited by the VPS container. I've never personally tried to nail down a performance issue on a VPS when the issue was/could be outside of my environment.

[gohighvoltage]

Ya have some good questions Ron. I didn't want to dive in to deep with the technicals, cause most users may not know how to do all that, or what it all means. I was leaving that for support. But, if he knows all that, this would be a great start.