Restore times - Rivan

[saracen]

Can somebody explain how long restores take?

There have been complaints recently about not being kept updated on progress when disk arrays crash and restores are done, and this time (the Ewok/Rivan crashes) we are given periodic updates in percentage terms.

I don't want to seem ungrateful, but percentages don't actually seem to tell us much.

For instance, Rivan :-

1:41 AM problem announced in forum
3:51 AM Rivan 82% restored.

Great ... from problem to restore and four fifths of the restore are completed, two hours. Shouldn't be much longer now.

7:15AM Rivan 84% complete.

Hold on ..... three and a half hours to do 2%? That's 1.75% per hour.

9:38 AM Rivan 85%.

Two and a half hours for 1%.

Good job it was well underway before the restore slowed down, because the whole thing would take a week and a half at that rate.

3:17 PM Rivan 86% complete.

Now we're at five and a half hours to do 1%. At that rate, a full restore would take three weeks and two days. We're heading for a world record here, surely.

5:34 PM Rivan 87% complete.

Okay, two and a quarter hours for 1%. That's better. I suppose. At least I won't be drawing my pension before it finishes. Not quite, anyway.

8:52 PM. Rivan 90% complete.

Wow - 3% in 3 and a half hours. Heading for 1% an hour. So only another 10 hours of downtime, on top of that we've already had. Crack out the Champagne.

I just feel sorry for the poor souls on Ewok. Theirs just restarted. Ouch!

10:49 PM Rivan 91% complete.

Uh oh. Looks like I relaxed too soon. Back to two hours for 1%, so most of another day to complete, it seems.




As I said, we've moaned about not being kept updated, and this time, there have been regular progress reports. I would rather question quite how useful a percentage figure is as a progress report when it seems so hugely inconsistent and unpredictable, but I appreciate there may not be any better you can give us.


But I also wonder why it takes so insanely long to restore?


And finally, having had my VPS being repeatedly killed and dropping to "mounted" state, and/or various services including named (taking out hosted sites, obviously) dying (or being killed) on a frequent basis, I've not had a properly usable VPS for quite a while, and certainly since 23rd of last month when this was raised as a ticket. This, on top, is really very, VERY irritating.

[TahirMeer]

any one got any news... at

7:46AM 98% restore was completed

its now

11:46 AM and VPS aren't yet online?

[saracen]

Well, restore is now complete. Great.

But fsck is running, and that can take quite a while to run.

And during the process, we STILL aren't being given any projection or estimate of when the servers will be up and running. It's like watching paint dry.

[saracen]

It's back up.

[TahirMeer]

It's back up.

Sadly I and my customer lost 2 weeks data... last info is from 26th oct.

Thanks the services is back up... ashame JPC didnt take any backup in last week...

[JPC-Masood]

But I also wonder why it takes so insanely long to restore?

Because of huge data that is on the vps node, usually 200~250GB that is compressed on the backup node.

The %age reporting is not properly coded in this backup software, we will need to get that fixed. It shows the %age based on the partitions (i.e. if there are 4 partitions of different sizes and 3 are restored, it shows 75% restored) The main partition on a VZ node is the last one to restore and this is where 99% of the data resides.

[saracen]

Thanks for that, Masood. It certainly explains the screwy percentages.

It still seems to be very slow, though. I have servers here (in-house, not net-facing) that backup and restore at MUCH higher speeds than that. I could perhaps understand it if what was going on was a RAID rebuild from remaining parts of the array and parity data, but I understood this to be a restore - presumably from an image on a network node - not a rebuild??

[JPC-Masood]

Yes, it's a compressed disk image that is restored from the backup node over local network, not a rebuild.

[Gwaihir]

So.. what's the bottleneck? It DOES seem slow . Do you have a couple of gigabit network ports available for such cases? (I presume these reasonably new servers actually have onboard Gigabit NICs, despite the 100Mbps switch-ports they probably spend most of their lives connected to?)