titanium - restore and outage

[Sara]

Supposedly both the filesystemp check and restoring of accounts has been going on since sometime last night. So how much time should I expect for it to take before accounts are restored?

Have never been through this kind of crash and burn and have no clue what to expect.

[tstahmer]

I'm also disturbed by the lack of any information about the status of this problem. Just a few vague statements and apologies. And no response to email.

So much for the promise of "%100 Uptime Guarantee".

[JPC-Greg]

The network has been up, the server hasnt. We'll give you guys some specials and credit for the problem. This places new emphasis on us to resort to a high availbility clustered solution of some kind, even if we have to custom code a complete system ourselves. These long restores are just too much of a killer on client sites, our man power, revenues (fallout from cancellations) , and so on.

Theres a heap of data to restore and depending on where your data sits in that stack , your site is either up now or pending as the data continues to move.

[Connie]

My e-mail is back, but still no site. Greg as you know, this is hell on a e-com site. Not bitching. it's been a couple of years since we (or I ) have had any problems.

I will be glad when this is over.

[Geoff]

Despite promises in the past to keep clients updated during a major outage communication about Titanium's current outage is lacking.

It took hours for the first post to the forums.
There have been a number of hours between subsequent posts.
The last update was hours ago.
The server has been down for almost 24 hours now.

In contrast another host I deal with in this situation will post updates every 20 minutes even if it is to say "Still FSCKing".

This is the second incident like this (24 hours downtime on a server) that I have seen at Jag in the last month. This seems too long for a professional hosting company to sort issues like this out. 24 hours and counting without the website is perhaps excusable - but try telling my resold clients that I have no idea when their email is coming back up.

A little regular communicaton from Jag would go a long way.

[JPC-Greg]

I know and it taxes our entire crew and systems heavily going through these types of problems. Trust me when I say jaguarpc is giong to change for the better drastically over the next 6-12 months. I would like to take everything into the clustered format even vps, ded servers, shared, resellers, everything. Its a big undertaking but I think the world has enough solo server hosting companies out there already, time to take it up a notch.

[Sara]

Thanks Greg. I suspect mine is probably towards the bottom of the pile. Thankfully I have only minor twitches from waiting :P

On a serious note though, if you're seriously thinking about setting up a shared cluster with a non CPanel control panel let me now and I'd love to play guinea pig.

[Geoff]

Your response doesnt really address my concern about lack of communication.

Surely you have some way of saying "After 4 hours 200 out of 1000 accounts have been restored" or "After 6 hours 30% of accounts have been restored". At least then we have an idea of progress.

Service has improved at Jag over the years Greg, but I still see the same problems in communication in situations like this. I know there are better ways to handle communication, because I see it with another company I deal with.

[JPC-Greg]

Despite promises in the past to keep clients updated during a major outage communication about Titanium's current outage is lacking.

It took hours for the first post to the forums.
There have been a number of hours between subsequent posts.
The last update was hours ago.
The server has been down for almost 24 hours now.

In contrast another host I deal with in this situation will post updates every 20 minutes even if it is to say "Still FSCKing".

A little regular communicaton from Jag would go a long way.

Your absolutely right, i dont see that any staff member sent a mailing on this one. Completely our fault, we need to add more instruments that prevents this step from being missed in the future.

I hate to update every 20mins with "nothing to report" type notices, but suppose it beats none. gosh, erm.. I am sorry for the comms on this one.

[JPC-Greg]

Your response doesnt really address my concern about lack of communication.

Surely you have some way of saying "After 4 hours 200 out of 1000 accounts have been restored" or "After 6 hours 30% of accounts have been restored". At least then we have an idea of progress.

Service has improved at Jag over the years Greg, but I still see the same problems in communication in situations like this. I know there are better ways to handle communication, because I see it with another company I deal with.

i was typing the mailing and quoting your message , had just merged the threads , i didnt see yours when i typed that... can only work so fast

[Geoff]

Thanks Greg.

I know the "nothing to report" after 20 minutes might not feel like much to you - but it would have cut out me posting replies to my support ticket every few hours.

I suspect you could come up with some stuff to report though if you are restoring a server. Is it actually possible now to know the percentage of data/accounts that have been restored?

[the_ancient]

I know and it taxes our entire crew and systems heavily going through these types of problems. Trust me when I say jaguarpc is giong to change for the better drastically over the next 6-12 months. I would like to take everything into the clustered format even vps, ded servers, shared, resellers, everything. Its a big undertaking but I think the world has enough solo server hosting companies out there already, time to take it up a notch.

HSphere?

Sounds Interesting, I cant wait to see what you have up your sleeve

[Spathiphyllum]

I know the "nothing to report" after 20 minutes might not feel like much to you - but it would have cut out me posting replies to my support ticket every few hours.

It's not much, but it's something. I just check it every hour to see if anything has changed. Clogging up the server with another site reload or cPanel request or introducing yet another ticket for the queue just bogs the QC systems down. Yes, it's disappointing to see this downtime for so long and what seems to be an eternity even though it isn't. I noticed the poor server response late last night and submitted my only ticket then. The server was subsequently rebooted. The rest is history. Reboot "failed", FSCK began, rebuild ensued.

If another architectural design will minimize such delays (and hassles for your staff, too) when the inevitable occurs, by all means Greg, switch... even if there are some inconveniences introduced by the lack of the traditional cPanel GUI. There's downtime and then there's downtime. We are approaching downtime territory.

[the_ancient]

some inconveniences introduced by the lack of the traditional cPanel GUI. .

you say that, like it would be bad thing to lose cpanel.....

[Spathiphyllum]

you say that, like it would be bad thing to lose cpanel...

Heh. Well, I'm not wed to it if that's what you're implying.

I use some of the tools just because they're there. If another GUI were supplied, I'd adopt it instead. If there were none, I'd go CLI. Hey, I'm adaptable. Recall I still like static xHTML, Perl, and Notepad-like editors, so don't think I can't handle pain.