Emergency Maintenance: argus.nocdirect.com

[mlittle]

Any ideas on when the server might come back to life? Thanks.

[Hershey]

We are contacting you to inform you about some

Emergency maintenance.



The server on which your semidedicated hosting

package is located experienced hardware failure

today resulting in total data loss on its hard

drive's RAID array. We do have backup systems in

place however and your sites are being restored at

this time.



Unfortunately the backups of your server were

affected by a bug in our new backup system. A

recent bug was detected that allowed the daily

incremental backups, under very rare conditions,

to be corrupted. The fix for this was an update

which we applied and if we located any backups

with the corruption detected we had to recreate

them. Unfortunately the corruption wasn't detected

so this morning we've been working to identify the

problem which of course turns out to be a result

of the aforementioned bug.



The only backup of the servers data we have which

is not affected by the bug was the inital backup

taken Apri 25th 2007. This backup is being

restored at this time. I sincerely apologize for

the trouble and inconvenience this may cause you.

If you maintain your own backups separately and

you would like our technicians to restore that

data for you we will happily do so. Simply open a

support ticket and once our initial restore is

complete of the April data we can begin to restore

your more recent data.




======================================== =================


I CAN'T BELIEVE THIS. WHAT KIND OF SERVICE IS THIS? HAVE WE NOT ENTRUSTED THEM WITH OUR DATA? And how is it that there is a total data loss? Even if some corruption happened, unless overwritten by other data it should b mostly recoverable!!

I just find it difficult to buy this story. They're being dishonest and I don't appreciate that.

[JPC-Les]

The data restoration is currently at 89%. Looks like maybe 1-2 more hours. I apologize for this delay.

[Hershey]

The data restoration is currently at 89%. Looks like maybe 1-2 more hours. I apologize for this delay.

So the earliest date it can be restored to is April 25th?!!!

How come?

Why not attempt to recover the data from either the original or the backup HardDrive? This should be possible even if it was formatted, let alone something as simple as some file corruption.

[JPC-Greg]

We not a bunch of kids that get their kicks off by lying to our clients. The software is r1soft and this is a verified bug, contact them about it you like. We all feel pretty bad about this as it is. Our new fancy stupid system we paid so much for and trusted turns us out a flop of a backup. The backups are like bare metal backups, we cant restore corrupted ones. Backups dont work that way where they tell you joe bobs file 123 is bad, but his abc is good. It sees the image as either good or bad I beleive.

Again we are all sorry for this and feel pretty bad about it. If theres anything we can do to make it right by you just let me know. Im sure we can credit you in some way in the least.

[JPC-Greg]

So the earliest date it can be restored to is April 25th?!!!

How come?

Why not attempt to recover the data from either the original or the backup HardDrive? This should be possible even if it was formatted, let alone something as simple as some file corruption.

It just doesnt work that way. We dont have the sophisticated hardware that professional data recovery businesses do. Their whole business is recovering data and their techniques, besides costs thousands or tens of thousands, arent exactly something any host has. not a single host .

[Gwaihir]

Then I guess it wasn't something as simple as some file corruption..

I can't see what's being dishonest about this. They're admitting a major screw up. What do you think they might be hiding behind that admission?


What difference does it make whether its April 25 or a couple of days later? The backup system f#-ed up, it's more than a couple of days old, so you'll be uploading your own backups anyway as soon as your account is restored, no?

[Gear]

man my last backup was on the 29th and i was just thinking last night how im overdue and was about to do a backup when i got sidetracked by a script problem

[Hershey]

We not a bunch of kids that get their kicks off by lying to our clients. The software is r1soft and this is a verified bug, contact them about it you like. We all feel pretty bad about this as it is. Our new fancy stupid system we paid so much for and trusted turns us out a flop of a backup. The backups are like bare metal backups, we cant restore corrupted ones. Backups dont work that way where they tell you joe bobs file 123 is bad, but his abc is good. It sees the image as either good or bad I beleive.

Again we are all sorry for this and feel pretty bad about it. If theres anything we can do to make it right by you just let me know. Im sure we can credit you in some way in the least.

alright. I'm just really frustrated with what has happened. My site has an active forum with members in the thousands and as you can understand this timewarp that will obviously take place will not be taken lightly by the members....and the fact I lost a month of data when I accidently dropped the msql database about a month ago only makes it worse....this will be de ja vu once again and i don't think this time it will be easy to explain.

In addition, I have personally created more content which is of course lost as well. So My question is: how can I know this won't happen again? Obviously you guys had no idea and therefore didn't let this happen intentionally. So with that respect i'd assume it would be hard for you to say whether or not we can be confident our data is secure.

[JPC-Masood]

Anyone interested can find the details of the bug here: http://forum.r1soft.com/showthread.php?t=270

[JPC-Greg]

alright. I'm just really frustrated with what has happened. My site has an active forum with members in the thousands and as you can understand this timewarp that will obviously take place will not be taken lightly by the members....and the fact I lost a month of data when I accidently dropped the msql database about a month ago only makes it worse....this will be de ja vu once again and i don't think this time it will be easy to explain.

In addition, I have personally created more content which is of course lost as well. So My question is: how can I know this won't happen again? Obviously you guys had no idea and therefore didn't let this happen intentionally. So with that respect i'd assume it would be hard for you to say whether or not we can be confident our data is secure.

I know and understand, a month is a lot of work and info to lose especially to a forum. Your question of how to be certain it wont happen again, honestly nothing. There are no certainties when dealing with hardware and software. You can increase your chances of not being affected by backing up data in more than one area and more than one time. I honestly cant count the number of backups being done to this site. Aside from copies all over my machines at home, the server, the backups, the backups of the backups. I dred loss but I also dont discount that it can happen to us too. Just takes a lot more to go wrong when you duplicate it.

We can say that r1soft is aware of their bug and going to fix it. But we cant promise there wont ever be another one. There shouldnt be, but then again this one shouldnt have been.

its all just mind numbing

[Ron]

Sorry you guys are going through this.

If you have a MySQL database or databases, you should be doing a dump periodically (daily?) and download that dump to your own PC or to another offsite (off of JaguarPC's site that is) for saftey. At a minimum, you should subscribe to their backup service and send the dump over the internal network so it is on a different machine at least.

I don't know that JAG has an offsite location for backups. If the datacenter were to catch fire, you would lose everything. From now on, resolve to take your MySQL dumps daily.

I also learned the hard way with the time warp. However, I was able to capture about 75% of the data in Google's page cache, and grabbed it, but my users were so understanding. I polled them and not a single user said that I should bother trying to restore all of the lost threads. I had 2 (count'em, 2) users who asked if I was able to find two threads, which I restored for them. One was a personal heartfelt tribute, the other was some research results or some such.

Good luck folks. Sorry you had such a rough day, clients and staff too.

[mlittle]

This is pretty annoying. Blog posts and comments I can recover, but losing mail back to the end of April is very frustrating indeed.

Whilst r1soft are saying they have a potential problem, it has been known about for some time (5th April or 4th May, depending how they state their dates). Are JaguarPC saying that there is no verification of the backups being taken? This seems like a lesson needing to be learned from this mess.

[Gear]

I'm quite surprised that we were charged for this month, especially concidering how many additional times the server has been down since.

[Gear]

...and I found the server down again right after making that post