Demetrius Down

**at0mic** · 08-08-2011, 01:44 PM

Can we get a restart on Demetrius? I can't tell which services beyond apache aren't working. The network status page hasn't refreshed itself in over 12 hours.

**JPC-Sajjad** · 08-08-2011, 01:52 PM

We have noticed that there is some high load on the server. We are investigating this and it will be back to normal soon.

**boyandin** · 08-24-2011, 09:12 PM

It is said some of RAID drives failed and will be replaced.

Could you provide any estimation on the replacemenet/recovery/etsting/whatever? Bad news (say, 24 hours there will be no Demetrius) are better than no news at all.

Please keep us updated.

**JPC-Bilal** · 08-24-2011, 09:33 PM

Our Admins are working on the server but unfortunately it is not possible to provide any ETA at this time. For updates and details please visit this thread. Thank you.

**Ron** · 08-24-2011, 09:55 PM

Bilal, the forum stylesheet is broken; links don't show as links, they show as regular text. Unless a person knows that, your sentence above seems to ask the user to follow the current thread, the one that they are reading now, because they don't see that the word thread above is a link.

Boyandin, Bilal wants you to follow this link: Maintenance -- demetrius.nocdirect.com

**boyandin** · 08-24-2011, 09:59 PM

Originally Posted by Ron

Boyandin, Bilal wants you to follow this link: Maintenance -- demetrius.nocdirect.com

Thanks, but his link looked completely normal (nothing broken, as you've mentioned).

**Ron** · 08-24-2011, 11:50 PM

You see it ask a link? Underlined or a different color?

To me (and everyone else who has spoken up) it looks like plain, non-link text.

If you ARE seeing it like a link, can you tell me what browser you're using?
Thanks.

**boyandin** · 08-25-2011, 12:12 AM

Originally Posted by Ron

You see it ask a link? Underlined or a different color?

To me (and everyone else who has spoken up) it looks like plain, non-link text.

If you ARE seeing it like a link, can you tell me what browser you're using?
Thanks.

Ah, now I see what you mean. No, the link looks like an ordinary text.

To system administrators: out of pure curiosity, what is the total amount of data to restore from backups?
(to make rough assessment and calm down the customers). At this time I estimate the total downtime to be at least 24 hours (optimistic value).

Thanks.

**johnonjaguar** · 08-25-2011, 05:20 AM

The last update from the official announcement forum (Maintenance -- demetrius.nocdirect.com) states that they, as of 3:39am, were still copying files over to a USB drive. That's more than 3 hours after they say they started copying files.

Can I just point out what a poor strategy and lack of planning JPC has in place. Why was there not already a backed up copy of our files? Had that been in place, you could have run a differential backup to grab the few files that might have changed since the last backup.

We JPC customers need to learn from this that JPC does not have a back-up strategy in place for our files and that we either need to be doing this ourselves or finding a more responsible host.

**boyandin** · 08-25-2011, 08:54 AM

Looks like 48 hours is better estimation.
What I really admire this time, compared with December 2007, is quietude.

Also, out of curiosity, how often do other servers suffer failures of this scale?

**JPC-Veena** · 08-25-2011, 09:03 AM

johnonjaguar,

We do have a backup strategy in place, we use R1soft backups. We do have couple of daily and monthly backups for all servers including demetrius.
In this case, along with filesystem corruption and some data loss, the drives were also found failing in the raid array. We chose the fastest method to get the server up and running.
We are doing our best, we will keep you updated.

**boyandin** · 08-25-2011, 10:42 AM

Originally Posted by JPC-Veena

We do have a backup strategy in place, we use R1soft backups. We do have couple of daily and monthly backups for all servers including demetrius.

I do not doubt you do all your best to restore server back to working state. What's bothering me is timeframe. I do not know the exact Demetrius disks capacity, nor other specs, but rough estimation based upon my own system administrator's experience and comments from your Techs tell me it can take 2-3 days more, to restore everything back.

Judging by December 2007 thread, and taking into account you could probably increase disk array, the more powerful becomes server, the more downtime we will have in such cases.

A logical suggestion would be, for users that have their own backups of their sites, to offer, in cases of similar disasters, a temporary hosting on another, alive, server.

I know it isn't possible to restore/copy all the data by casting a spell, nor I wish to accuse your redundancy plan of its inefficiency, but if a downtime lasts for more than, say, 12 hours and it IS known the server is literally built from the scratch, I suppose I could request from the hoster temporary Web space, if I can restore the site myself.

Whom shall I contact to suggest such a service (I don't mind to pay extra for it)? I have a spare reseller plan at another hoster for such a case, but surely you could do your loyal customers a favor by providing their sites a temporary, even one-page home, if they choose to pay for that.

May I ask how large are disks of Demetrius now, and how large they were in December, 2007? Just to estimate how much longer shall we wait this time.

I am with JaguarPC for more than 5 years, but as soon as disks grow so big that it will take 5 or more days to restore a server, I will certainly reconsider staying.

Good luck in recovering. Thanks.