Raid 5 servers

[Ron]

Hi,
This isn't ticket level stuff, so I was hoping that JAG would get around to answering this when he has some spare time....

My accounts are on a new RAID 5 server, and that array has also once gone down and had to be restored from backup.

I was concerned about it right away (after all, RAID is all about safing data, and if all it is going to do is to corrupt everything whenever a single disk goes down we're just decreasing MTBF by adding spindles) but now things have been quite stable.

It's possible that this class of PC hardware is not yet as reliable as midrange versions (like Sun, DEC, HP, IBM)... and embedded software (firmware) is very complex for RAID. Are there higher quality controllers available?

I will be taking backups every day or two now (it's 1.4 GB now for my main site, though I can probably whittle that down to once per week and take an incremental every day) ... but I don't have any software to manage backups, I just have tarballs. Is there any software that can enable image and incremental backups and manage it remotely, not involving a hard tie into hardware (ie creates a backup tarball for download).

Can I also ask what other hardware solutions under consideration, and if all customers on RAID are being considered for migration?


Sooooo to summarize the questions/issues;
1) Higher quality controllers
2) Backup management software for self-maintained remote storage
3) Migration of current RAID environments

Thanks. I know you're busy, so please... only when you have time.

Of course, I'm all ears from other clients about backup management software.

[Gwaihir]

You'll want to check out rsync I think it was originally written to keep remote mirrors in sync with minimal bandwith use. It is also a great tool for backups and there's plenty of script examples floating around the net showing you exactly how to set up whatever scheme suits you best.

[JPC-Greg]

Hi,
This isn't ticket level stuff, so I was hoping that JAG would get around to answering this when he has some spare time....

My accounts are on a new RAID 5 server, and that array has also once gone down and had to be restored from backup.

I was concerned about it right away (after all, RAID is all about safing data, and if all it is going to do is to corrupt everything whenever a single disk goes down we're just decreasing MTBF by adding spindles) but now things have been quite stable.

It's possible that this class of PC hardware is not yet as reliable as midrange versions (like Sun, DEC, HP, IBM)... and embedded software (firmware) is very complex for RAID. Are there higher quality controllers available?

I will be taking backups every day or two now (it's 1.4 GB now for my main site, though I can probably whittle that down to once per week and take an incremental every day) ... but I don't have any software to manage backups, I just have tarballs. Is there any software that can enable image and incremental backups and manage it remotely, not involving a hard tie into hardware (ie creates a backup tarball for download).

Can I also ask what other hardware solutions under consideration, and if all customers on RAID are being considered for migration?


Sooooo to summarize the questions/issues;
1) Higher quality controllers
2) Backup management software for self-maintained remote storage
3) Migration of current RAID environments

Thanks. I know you're busy, so please... only when you have time.

Of course, I'm all ears from other clients about backup management software.

The controllers we use now are the best in the industry and so even if throwing more money at something were a fix there simply isnt anything higher to throw money at in this case

Eric is working on a backup script for distribution to our clients that will help you manage backups.

We have only one active raid5 server lesft, Zeus. We dont trust raid5 to work for our needs anymore and we will be migrating zeus to a raid10 machine, it should be seemless. It should improve I/O as well.

[Ron]

That's great to know, all of it. Thanks for taking the time to respond.. I know things are busy.

Yes, I/O is certainly the bottleneck on zeus when server loads are high (I'm on zeus). Today, for instance, server load was around 10, with about 50% CPU wait I/O.

Response time was still very good, but just a hint less than it's usual instantaneous response profile. Even a savvy first time visitor would have never thought anything other than "This is a snappy webserver."

I am surprised that RAID 5 isn't suitable for I/O needs of a web/MySQL server, but if reliability is the issue, I am *ALL* for a switch. I remember being offline for a day or so after the RAID5 array got corrupted a little while back.

My main complaint when that happened was that support didn't keep me apprised of what was going on. It has to be the most frustrating thing in the world to be down and in the dark.

My problem occurred over a weekend. Due to the lack of response from support, EVEN THOUGH YOU WERE WORKING ON THE ISSUE OVER THE WEEKEND, I truly believed that the machine was sitting there, dead, waiting for Monday morning.

Perhaps a list of email addies (alternate site emails, of course) for clients on a particular machine, or a list of emails from the ticket system for anyone who has opened up a ticket for a major event could be used to dash off a quick "Hey, this is a widely distributed note, please don't respond to this email: The issue is still being worked on, we have an eta of blah (or we still have no eta, but we have 3 techs working this issue). We don't have any more info that that at this point, but we WILL keep you posted as we move along to fix this service outage"

That would have gone a LOOOOONG way towards making me happy.

Oh yeah, my other complaints had to do with the rebuilt server's configuration, but that's just life, I guess. You guys had full backups (the system went down about 23 hours after the last backup, so nearly a full day was lost, but them's the breaks) and really pulled my fat out of the fire. I highly recommend JAG's managed services, in case anyone is wondering

[Vin DSL]

You'll want to check out rsync

Every file transfer program I use has some sort of 'sync' utility in it. I've never used any of them, but ppl swear by this 'sync' function. However, from my understanding of the situation, and what Ron wants to accomplish, I doubt this is going to be robust enough. (Heh, thinking out loud here...)

Hrm...

I just Google'd this subject and struck a mother-lode of possibilities:

http://www.google.com/search?sourcei...p+script+linux

I have a meeting in two hours, but I think I'll check this out when I get back...

[Ron]

What I am *thinking* I'd ideally like to be able to do is have a utility where I bring up a filename, and it shows me what versions are available.

I found a page on using rsync and I skimmed it, actually reading the first part, but I have not yet studied the sperate machine ideas... I think they all revolve around NFS mounts, but like I said, I haven't had the time to read it yet.

http://www.mikerubel.org/computers/rsync_snapshots/

[the_ancient]

Perhaps a list of email addies (alternate site emails, of course) for clients on a particular machine, or a list of emails from the ticket system for anyone who has opened up a ticket for a major event could be used to dash off a quick "Hey, this is a widely distributed note, please don't respond to this email: The issue is still being worked on, we have an eta of blah (or we still have no eta, but we have 3 techs working this issue). We don't have any more info that that at this point, but we WILL keep you posted as we move along to fix this service outage"

I was thinking the same thing, I have been lucky with servers of late, but that is probally because "life" has taken over "living" and I have not had the time to "play"

but the ticket system I am sure already stores the Server data (as it stores all the URL's) it should not take much to program a function to send out Mass updates on Box's that are down.

and to go a Long way for the people "in the dark"

[Gwaihir]

Every file transfer program I use has some sort of 'sync' utility in it. I've never used any of them, but ppl swear by this 'sync' function. However, from my understanding of the situation, and what Ron wants to accomplish, I doubt this is going to be robust enough. (Heh, thinking out loud here...)

Surprised you never heard of it!

Those are generally quite different, as they still allways need to transfer complete files. The beauty of rsync is that it only needs to transfer the changes, so if you make a minor change to a 10MB file, it will transfer just the little bit.

Another thing is that you can tell it to keep the earlier version too, yet still maintain a full backup. I.e. it can first make a copy of your earlier backup and then sync that copy with your site in order to have a new full backup without overwriting the old one, while still having to transfer only the changes.

This stuff sure is robust enough. It isn't a sync function thrown on top of some FTP or SCP utility, that cuts out when it gets more than x folders / files to deal with a once, but a program specifically designed to compare large datastructures and then move over only the changes.


The only downside I have found in this thing is that, being a truly powerful open source linux / unix tool that has been around for years, it is generally controlled from a command line script that, no matter how many examples you read, you'll have to fine tune to your own situation yourself. I.e. it isn't click-and-go; you can actually screw it up making it do entirely different things then you thought (or nothing at all).

That said: I have the scripts running fully automated on four tasks for nearly a year now: backup of my site from here, backup of a hobby site from elsewhere, backup of my home data to elsewhere, backup of office data to my home. The logs show it missed a scheduled backup every now and then when my internet connection was down, but other than that it has worked without a hitch.

[Vin DSL]

Surprised you never heard of it!

When it comes to backups, I'm a trogladite. I've been doing this forever. Before I came here, I just made mirror backups of my web site to my PC. Pre JaguarPC, my web site was only a few megs -- no big deal, even on dialup. But, now, it's pushing a gig, so it's a whole different ballgame.

[Ron]

I dunno what a trogladite is. Is it anyuthing like a trilobite (or one of my fave intentional typoes (that's another)) "Trilobyte"

That a really really old diskpack, the trilobyte. Or a game comapny, Trilobyte. Very cutely named, that company.

[Vin DSL]

I dunno what a trogladite is...

Here's a picture of one...

[Vin DSL]

Oops, sorry!

My Bad! That's a trogla-dyke...

[Vin DSL]

Here'a a troglodyte...

[Vin DSL]

The controllers we use now are the best in the industry and so even if throwing more money at something were a fix there simply isnt anything higher to throw money at in this case

Eric is working on a backup script for distribution to our clients that will help you manage backups.

We have only one active raid5 server lesft, Zeus. We dont trust raid5 to work for our needs anymore and we will be migrating zeus to a raid10 machine, it should be seemless. It should improve I/O as well.

You might be interested in this, Chief...

http://www.eweek.com/article2/0,1895,1866790,00.asp

[Gwaihir]

What I am *thinking* I'd ideally like to be able to do is have a utility where I bring up a filename, and it shows me what versions are available.

I'm not sure what you're after here. Shouldn't a backup generally be "most recent version before disaster struck"? If you're after true version management of documents, point your google to subversion (http://subversion.tigris.org is the main page.)