Back Ups

[wasabinut]

Is it true that at any given time JaguarPC only maintains one back up of a server and should that one back up be corrupt your clients are out of luck?

I am reffering to the total failure of andromeda.nocdirect.com

I received this email early this morning:

" We are contacting you to inform you about a critical issue regarding the hosting of your VPS hosted on the server Andromeda.nocdirect.com. I'm sure you are aware of the recent hardware issue on the server which we believed to be resolved.

Today the server went down and despite our efforts the data on the server is unrecoverable.

Compounding the issue we were at the same time addressing an issue with our enterprise backup
system which had stored corrupted backups for this very node leaving us empty handed.

We will be creating new VPS accounts for you using the same IP addresses you had previously and logins will be sent to you.

We know this comes as terrible news and will be a nightmare to say the least. We will work with all affected users to the extent of our capability to make things right for each of you."

Now the server was up and running after the fsck for a while before crashing. How does a user gain access to those files? Also I did generate backups however I generated backups on the server, is their any way of accessing them?

Another thing.... How long will it take sales to get new VPS's set up? Are all the people on the failed server screwed till Monday?

Lastly, Will there be a refund for this colossal collapse/failure?

[AncientLady]

I have been in contact with the help dudes.

Any backups done using the VZPP interface are lost.

They said that a full backup using cPanel was done, you can restore from that. I guess that assumes the backup must be done to some location other than on a Jag system.

I have already informed my clients that it will take some time to restore their sites. I keep multiple backups on different machines to make sure I don't lose site content. I try to keep databases as current as possible locally. Hopefully my clients will be patient and understanding.

Jag apparently doesn't validate the backup process. Otherwise they would have discovered that the backups were corrupt before they started the upgrade process on andromeda. Then they might have been able to create good ones before the upgrade.

There should be some sort of compensation. The server (and all the sites) has been down for full 24 hours. And they haven't given us any information about how long it will take before they create the new accounts.

I recently recommended Jag to a new customer looking for a new hosting company. I really hope that I haven't steered them wrong. I've only had one other problem (that was not of my own making) in the 16 months I've had my VPS. Other than that, the service and help has been great.

[EuroNut]

Is it true that at any given time JaguarPC only maintains one back up of a server and should that one back up be corrupt your clients are out of luck?

Please don't hate me for saying this, but the integrity of your own backup system should be classed at the same highest priority as it is to have your web site up and running.

No hosting company I know of guarantees to have full backups to be able to restore all sites on a server in case of failure - Of course Jag does run a backup system, and are trying to upgrade it, but that is no replacement for your own, carefully planned, scheduled backup routine.

I deeply sympathise with your current predicament, and all those customers hosted on Andromeda, but if you ain't got your own backup ......

I wish you the very best of luck with this one.

[AncientLady]

Restoring a basic site's files from my own backup is one thing. Having to recreate all the domain accounts, make sure that nameserver has the right configurations, recreate all the e-mail accounts, recreate sub-domains and redo my Apache configuration to have it the way I want, is another thing.

All those things were covered on the VZPP backup. Part of the VZPP backup process had appeared to be validation of the backup (at least that is one of the steps listed.) I had also assumed (dangerous, I know) that that backup was going to a machine other than the server my VPS was on. This is apparently not the case, otherwise that backup would not be gone.

Once my stuff is all back, I'll look into the cPanel backup to see if it will keep the configuration information and save it on my machines here.

I sent an e-mail to the help guys asking when the VPS would be rebuilt and got a funny reply from the sales people:

I am forwarding your ticket to concerned department so that they can rebuild your VPS.

I was sort of assuming that this would be done for all the people who had VPS's on andromeda without having to ask.

I'm hoping the response (from sales?) was a misunderstanding.

[EuroNut]

Once my stuff is all back, I'll look into the cPanel backup to see if it will keep the configuration information and save it on my machines here.

The WHM/cPanel backup is quite thorough. It does back up each individual account into its own file, and includes stuff like email accounts etc. I used those files to restore a bunch of my accounts after that pesky hacker's visit - It was quicker than looking for the edited files (and risking missing one).

I sent an e-mail to the help guys asking when the VPS would be rebuilt and got a funny reply from the sales people:

That sounds very much like a support agent who's not actually working on site, and answering tickets remotely to ease the strain on the guys and gals at Atlanta/Houston. The info as to who has an account on Andromeda is all stored in Nixcore (and probably many other places) so I'm sure the techs already have a big list of who needs what VPS setting up.

Hang in there - I feel your pain.

[AncientLady]

I don't normally bitch in public, but I am very annoyed.

Well, late Saturday afternoon Jag created my 'new' VPS on saphyre. (I had been on the disasterously failed andromeda with no backup, right?)

I spent till late on Saturday and all Sunday morning getting things configured and reloaded.

On andromeda, my linux OS was Fedora, with Apache 2.0. When they made the new account, it is with CentOS5 and Apache 1.3.

When I tried to get the sites up that are registered on my own nameserver, they would not work. The support folks helped and and fixed the 'known' problem with CentOS5 and DNS. (I wonder what other 'known' problems I'll have.)

I tried to use the WHM to update my Apache. It failed 5 times . So I gave up on that.

Everything was back up and e-mails and sites were all working by about noon. Then it all died Sunday evening (between 5:30 and 10:00pm while I took time out for dinner, laundry, cutting my husband's hair and a critically ill cat). Since then, nothing works and the VPS is unreachable by anything (FTP, ping, web, etc.).

I had requested help about 5pm to change to Apache 2.2. When I got back to my computer at 10pm, there was a message asking for info about server, account, etc. (all of which was in the various messages in the support ticket). I sent the information they requested and informed them that the VPS was down again.

The response I found this morning just referred me to the new forum thread about disk problems on andromeda.

Assuming that the person who responded (from sales again) just looked at the title on the ticket and didn't look at the history on the ticket, I responded with a fuller explanation of what had happened since Friday.

This was their response (from sales again):

I believe that your VPS was resetup on Andromeda that got drive failure today again. I'm forwarding your ticket to the attention of the concerned department so that they can setup your VPS on a new Node.

Total confusion. I asked for information/clarification:

WHAT?!

Why was my account moved back to something that was known to have a major problem? Why was it moved without anyone informing me or asking me?

I spent hours rebuilding my stuff. Was any sort of backup made before you moved the account? Was that backup verified? Will I have to rebuild it all again?

I am really quite upset about this.

In the meantime, my clients are quite annoyed and I do not know how much e-mail is being missed (going through my main domain). (I do have non-jag e-mail hosts and try to use those, but don't know who has what at the moment.)

I just received an e-mail while composing this. They have rebuilt my VPS on ulysses with CentOS4. AND EVERYTHING IS GONE AGAIN!!!

Okay, time to start over again and see what happens.

[AncientLady]

Thanks to great help from the support dudes, all is good.

Everything is back and running on ulysses (whose upgrades went much smoother than andromeda).

They even just finished upgrading my Apache to Apache 2.2!

I'm a happy camper now (and my clients are understanding and not particularly annoyed )

The support staff really does rock here.