Intrepid Down for Hours

[Tazmania]

If you are interested in a professional backup and recovery software contact me via PM.

It's annoying to see that the server still is nor running

[Tazmania]

Do you know if there is any loss of data or mails?

[Fotiman]

Do you know if there is any loss of data or mails?

I received an email this morning that said:

"I tried sending this last night, but it kept bouncing back for some reason."

So yes, there was loss of mail at the very least.

[Fotiman]

Total down time was somewhere around 56.5 hours or more. You guys really need to re-evaluate your backup/restore strategy.

[twt]

I received an email this morning that said:

"I tried sending this last night, but it kept bouncing back for some reason."

So yes, there was loss of mail at the very least.

Email systems are configured to try only a certain amount of time before failing. Some systems are configured to keep trying for three days, but the system that attempted to send your message was probably configured to give up after only a few hours, maybe a day. On the plus side, email systems SHOULD report back that the sending attempt failed, so the sender will usually have a way to know to try again.

The timestamps on my email imply that messages apparently started to come through to my account by around 4am today, CST. [When the server first came up but before the RAID array was rebuilt.] I don't have any indication of missing data, but I wouldn't expect to see any. YMMV.

P.S.: I believe it's ALWAYS a good idea to regularly assess your business expectations and whether a "low cost" shared host provider can give you the level of service you need for your business. I realize it's up to me to keep my OWN local backup so I can "failover" to another host or a different hosting plan if necessary. We now know how long it takes to do a Bare Metal Restore on Intrepid, and it would be nice to know if JaguarPC will be able to address this "serious but not worst case"* scenario by, for example, reducing the number of affected accounts on a single server or whatever.

*P.P.S.: This was NOT a worst case because JaguarPC restored from a recent backup and got the new server online. If the backup had been damaged and/or JaguarPC's data center went offline due to some sort of calamity we would all be MUCH worse off.

[JPC-David]

Hello residents of Intrepid,

Let me just once again say how truly sorry we are about this situation and that we will make it right.

Please email me at david@jaguarpc.com with your domain name and I will take care of you.

[JPC-Sabrina]

Thank you all for enduring the Intrepid outage. If you have any additional difficulties please contact technical support or customer service. Our sincerest apologies go out to you.

[Ron]

The client who said the email "kept bouncing back" may have misinterpreted delay notices (from 1 or from multiple email s/he may have sent during the outage) as failures. Or it indeed could have been a bounce.

The lengthy amount of time that a bare metal restore from R1 requires has been an issue for perhaps four or five years; I can't recall exactly how long ago they partnered with R1. It used to take 3 days to do the restore, so 2 days is quite an improvement.

Will the cloud have a better backup solution guys? Or will TBytes have to be restored?

I'm glad you guys are back online with minimal problems. Good luck.

[Tazmania]

The lengthy amount of time that a bare metal restore from R1 requires has been an issue for perhaps four or five years; I can't recall exactly how long ago they partnered with R1. It used to take 3 days to do the restore, so 2 days is quite an improvement.

Improvement? With the right software that is done within 30 minutes.

[JPC-Les]

Improvement? With the right software that is done within 30 minutes.

That's possible if the data was stored on a massive SAN with SAS drives in a stripped array over a fiber channel. However the cost of such a backup system wouldn't work for the price point your shared hosting accounts is at. No we are not satisfied with the current time required for backup restores. And It's not to say we cant make more improvements which we are working on but don't forget that the shared plan is a budget solution and it's not feasible to have such a backup system in place for them.

[Ron]

On a 1Gbit line getting throughput of 100MByte per second, a restore of 500Gig might take a couple of hours. Getting 15MBps would take maybe 12.

Les, what IS the problem with the speed of R1?
Is it you can't allow the restore to swamp the network?
Is it something with the R1 server? Reconstructing incrementals?
Is it the target server's performance?

Some combination?

[JPC-Les]

Speeds of the backup servers and network speeds are the main roadblocks. Once we get our network upgrade done and have backups 100% on a private network this will improve greatly.

[Ron]

Excellent.

Thanks!