martin update please?

[btimms]

Could we get an update on martin? Downtime is close to 21 hours now.

[btimms]

bump

[stuff]

bump

Its really frustrating
Any news?
About accounts? will they be ok? Or will there be data lost?

[btimms]

It doesn't look promising at all as to the last post in the martin emergency thread. Guess we'll just have to see whats left that can be restored. I'm loosing the faith.

[Ron]

I'm just wondering: Has there EVER been a completely successful retrieval of current backup data when an array has failed? I'm not talking about the RAID array saving the day, I'm talking about when the RAID array has failed, like here.

I don't think I have ever seen one. I know that when I needed (and paid $15 for) some backup data, it was 2 weeks old. I know that when I had a different server array failure that the backup data in the db partition was 2 weeks old.

For much more than a year, a more robust backup process with integrity checks has been promised. Perhaps it's time for a real and complete overhaul of the approach that you're taking to backups?

[JPC-Greg]

I'm just wondering: Has there EVER been a completely successful retrieval of current backup data when an array has failed? I'm not talking about the RAID array saving the day, I'm talking about when the RAID array has failed, like here.

I don't think I have ever seen one. I know that when I needed (and paid $15 for) some backup data, it was 2 weeks old. I know that when I had a different server array failure that the backup data in the db partition was 2 weeks old.

For much more than a year, a more robust backup process with integrity checks has been promised. Perhaps it's time for a real and complete overhaul of the approach that you're taking to backups?

Yes there has, you just dont hear about them as much nor do they catch anyones eye. Granted though this does show some weaknesses in our backups that we have struggled with. In dec ironically enough, we began researching a new enterprise backup solution which just recently passed our tests. We are deploying that this month and hopefully it will put an end to lost data.

As for martin, we definitely have all the current mysql data backed up , which tends to be the biggest issue for people. We do have most the home dir but its unclear what, if anything, is missing beyond that. Recovery on a new node is already taking place.

[stuff]

Yes there has, you just dont hear about them as much nor do they catch anyones eye. Granted though this does show some weaknesses in our backups that we have struggled with. In dec ironically enough, we began researching a new enterprise backup solution which just recently passed our tests. We are deploying that this month and hopefully it will put an end to lost data.

As for martin, we definitely have all the current mysql data backed up , which tends to be the biggest issue for people. We do have most the home dir but its unclear what, if anything, is missing beyond that. Recovery on a new node is already taking place.

I just left maybe a month ago my last hosting provider with problems like that. and now again. This is amazing. I had no MYSQl, well only one domain had. Other sites had over 1000 pages each and all my modifications where on this servers. Amazing. So much work and looks like gone. This is not good. And Your service came highly recommended.

[btimms]

I'm just kind of sitting here devastated. When I think how much time may be involved to fix this and the time on current projects that has to go on hold it makes me want to cry. Damn.

[Ron]

Thanks for the answer Jag.

For stuff and btimms, I'm very sorry you have had this problem. I know EXACTLY how you feel as I lost 2 weeks of data from my MySQL database once, I can't imagine what it would be if I had lost more.

All I can say is to take this as a very painful learning experience and backup your own stuff to your PC every night or at least every week. I was dumping my database and tarballing up my site and downloading the whole thing daily, then weekly as my site grew and the tarball size got huge.

Then I started using WinSCP and its synchronize function, and after the first initial download (which took about 6 hours if I recall correctly) now I get "incremental" daily backups -- only files that are new or have changed since the last backup are downloaded. I still manually dump my databases into flat files every day, then I start the WinSCP sync. The whole process takes less than 10 minutes, most of that time is: About 1 minute to dump and compress the db, about 4 minutes for WinSCP to scan the thousand directories I have on the site, and 2-4 minutes to actually download the necessary files for backup, including the 40MB compressed db dump. I do have a very fast connection.

I am concerned that the backup isn't robust because the amount of time that would be required for me to upload 10 gigabytes of data as everyone else is trying to do the same thing would be incredibly painfully slow, but I would have all the data. I already have in place a partial restore process that would bring my database and my forums back online first, followed by pictures and other large files as bandwidth became available.

Please do CHECK GOOGLE'S CACHE FOR YOUR SITE, I was able to retrieve many of the lost database entries from Google's Cache.

[btimms]

Well fortunately most of my sites are mostly html currently and the php sites were just starting up so the db didn't have much - and at this time it looks like most of dbs may be OK. This is definitely a wake up call. I put my 1st site up in '96 and have had quite a few since and several different host - and have never experienced anything quite like this. I guess its easy to get complacent. My biggest issue is one of the main reasons I came with JPC was imap - and I tend to leave emails on the server too long - so all of that appears lost. Lesson learned.

[JPC-Greg]

I just left maybe a month ago my last hosting provider with problems like that. and now again. This is amazing. I had no MYSQl, well only one domain had. Other sites had over 1000 pages each and all my modifications where on this servers. Amazing. So much work and looks like gone. This is not good. And Your service came highly recommended.

We're recommend because of the great team and efforts we went through. Just a simple replace one disk in a raid10 of 4 disks went terribly wrong. We suspect the raid card itself which if so, would be the first case we've had of this particular card failing. Im sorry if we have let you down, no matter how good the recommendation or reputation, you just cant escape hardware issues. The problem has got to be our backup methods which as stated above are going south. Our previous plans to expand the existing platform and modify it were nullified after testing a new enterprise class system we came by. Trying to get it in place asap.

Our NAS devices are still excellent, its the underlying software performing the backups that we are replacing. What many might see from this is an unexpected increase in servers performances due to the way the new backups pull data off causing less demand on the system as a whole.

We'll announce more about it soon.

[Ron]

In enterprise systems that I've seen, the system is quiesced, the mirrors are broken, and the system is allowed to resume in a matter of seconds. The backup data is pulled from the inactive mirror and when the back is completed the system is quiesced, the mirroring restarted and the backup disk is brought back up to date.

If I recall correctly. Besides, this was getting close to 10 years ago, I'm sure lots has changed.

[Vin DSL]

An observation, maybe...

Is it just me, or is there some sort of correlation between these events and a general slowdown of the entire network?

That is, I've been noticing that from time-to-time, my site runs 'slow'. It'll do this for 1-3 days, then all of a sudden it's faster than greased lightning again.

I noticed, when I come to the forums, after this resurgence in speed, this or that server is 'down' with problems. When 'the problem server' is fixed, and put back online, everything remains speedy for a while. Then, the cycle repeats itself.

I guess what I'm saying is, it seems like these unexplained periods of slowness always precede some sort of catastrophic event on one of the servers.

Put another way, is it possible that a 'server going bad', if you will, can slow down the entire network?

And, if so, couldn't this be used as a harbinger of things to come -- a signal to check into things you normally wouldn't, and possibly head off an impending crisis?

No answer is necessary! Just a thought...

[JPC-Greg]

I hadnt noticed any slowdowns and nothing seems to hiccup or pause. If or when you see this again, try bringing up the slowness to me right away so i can look at some charts and things. Theres more than enough capacity so it shouldnt be that. but if theres some device in the link thats taking a pounding and needs some attention we would like to catch that before something stupid happens.

[Vin DSL]

Will do, Chief!

Thanks...