Total chaos

[counterpoint]

I am appalled by several features of the failure that has affected many Jaguar servers in the last 24 hours. This has involved many people and is dragging on for a very long time. Those unfortunates, like myself, who are on skywalker look likely to suffer at least 24 hours of down time.

Given the seriousness of the incident, Jaguar owes its clients some explanations, and so far none has been forthcoming.

Although it is clear that efforts need to be directed to fixing problems, there is no excuse for the lack of a clear statement to say truthfully exactly what was the nature of this incident. How was it caused and how could it have had such disastrous effects despite claims of 100% uptime, UPS protection, generators, and so on. Vague statements are not good enough.

The level of information provided has been extremely poor. At no time has enough been said to allow clients to make informed decisions about how to cope with the problem. Never has any estimate been given of time to fix. Sometimes hours have gone by with no information whatsoever. Even now, there is the barest indication of how long skywalker users will continue to suffer.

I appreciate that it is hard to make predictions, and that they may go wrong. But that is no excuse for the total failure to provide any guidance.

Now that it is apparent that skywalker will be down for some time, I have made some limited improvement by reconfiguring my DNS (which is thankfully not at Jaguar) so that people looking for web sites that are down will at least see an apology and explanation, rather than just a blank screen.

It is extremely annoying that there was no warning yesterday of the possible extent of the down time. The implementation of an apology message would have been done 15 hours ago had there been any clue that there was a risk of the failure duration extending this far. As it was, given that it takes time for DNS changes to propagate, it did not make sense to redirect if the server was likely to be up again within an hour or two.

And of course that still leaves the question of why Jaguar is unable to restore a server within a reasonable time. If the data centre is vulnerable to major failures, then far more rapid restoration of backups is essential. Why is this such a major problem, and why was it not planned for?

So far the answers given have been sketchy or non-existent. Jaguar owes it to its clients to be more open and to provide far better information.

[Julian Muñoz]

Now I will need to explain to a customer why his website has been down for over 24 hours now. But the worst part is that I still don't know how to explain that everything he did on March 11th is lost because for some unknown reason, theoden is being restored from a backup made on March 11th at 02:00:35 EST. The power outage was around March 12th 09:45 AM. 32 hours of work lost.

Can please somebody explain to me:


1. What really happened yesterday. Several times I have read on this forums that JaguarPC has UPS, power generators and some great equipment to avoid interruption of service. None of these worked, neither the backup system.

2. What is JPC doing to prevent a power outage of this nature from happening again?.

3. Where is the theoden backup supposedly made on March 12th?.

Thank you.

[counterpoint]

None of this is helped by the terrible turnround on Emergency support tickets. I have one that has been outstanding for over four hours, and hasn't even been allocated. Others have taken just as long.

If much fuller information had been provided voluntarily by Jaguar, I wouldn't be raising support tickets. As it is, almost every bit of information has to be dragged out - and many questions still remain unanswered.

If there is nothing happening in relation to yesterday's incident apart from backups being restored, there is no excuse for such slow response to Emergency support tickets.

[JPC-Bilal]

We understand your concern guys and can understand the frustration been caused due to this event. We always try to provide as much information as available to clients and have been putting efforts to restore the service as quickly as possible.

Regarding the power issue, we have backups and arrangements in order to tackle situations like these but currently we do not have information on how this happened and why the servers didn't shift to the backup. The issue is being investigated and we will provide information as soon as we get it from engineers. Currently our focus is on restoration of the service and bring the clients back up.

As I said earlier that we do have arrangements and have backup but unfortunately it didn't work. Once we get the information we will take measures to prevent situations like these to happen again.

3. Where is the theoden backup supposedly made on March 12th?.

Although we had 12th March backup for Theoden but the server went down during the backup was taken and it resulted in backup corruption so we had to use the 11th March backup to restore the server.

The ticket load has been high which is causing delay but we are working on the issues and trying our level best to answer Emergency tickets as quickly as possible.

Thank you.

[counterpoint]

It appears that the backup of skywalker is also from 11th March, presumably for the same reasons.

If you published all this information promptly (the restore of this backup has now been running for 15 hours, yet it was only a few minutes ago as a result of raising a ticket that I could find out exactly what backup was being restored) you would get fewer tickets.

We cannot make proper plans if you do not tell us anything, and it is no good making excuses. As regards the 15 hours that have elapsed so far on the restore, I have an offsite backup that is as new as what you are restoring. The only reason I have delayed putting up sites on other hosting is because I expected you to restore a backup from 12 March, since the failure did not occur until 9 am or thereabouts.

That is 15 hours of unnecessary delay, caused by your failure to provide full information about what you are doing.

If you do not know the cause of the original problem, you should have said so publicly instead of failing to say anything. Although it remains a matter of concern that you do not know enough about the Jaguar installation to know why fundamental services do not work when called on. The more information you give out, the fewer tickets will be raised.

And how do we know that you really do have the facilities you claim to have? Will you allow an independent auditor to review your installation?

[JPC-Bilal]

We do mention about the date of backup being restored but unfortunately it was missed this time. The Skywalker thread has been updated regarding this.

[counterpoint]

When do you expect to have a report on the reason for whole servers being down for more than a day and a half?

Can you explain exactly what your guarantee of 100% uptime means? Does it mean anything, or is it just promotional guff?

[JPC-Jim]

We will have a report by the end of the day, once management meets, and goes through all of the facts.

[Julian Muñoz]

We will have a report by the end of the day, once management meets, and goes through all of the facts.

Thank you Jim. This is important to us!

[wcselby]

Where will we find the report?

[irdogg]

Where will we find the report?

yes where

[JPC-Jim]

The electricians and engineers were still going through all of the equipment late yesterday afternoon, as soon as we get a final word we will post in the forums.

[Vin DSL]

Here's an idea, Jim:

http://www.wikihow.com/Use-a-Toyota-...ckup-Generator [How to Use a Toyota Prius As a Backup Generator]

Just don't try driving it... Bwahahahaha!

[Ron]

The servers powered by a prius might crash, but even that might not stop them...

[Vin DSL]

The servers powered by a prius might crash, but even that might not stop them...

Toyota lover, eh?

Here you go, Ron...

http://cgi.ebay.co.uk/TOYOTA-375KVA-...1123002r4 446

58,000 pounds sterling.