Bulk Reseller Server - Down

[eHits]

.
Bulk Reseller Server (cressida.nocdirect.com) is down (12 hrs)

Hi,

About the above reseller server outage.... I'm hosting a law firm website
and they are desparate for email. One attorney needs email within the
next couple hours, for a time-sensitive matter.

The server has been down for over 12 hours.
And 5 hours ago JPC said they cannot suggest any ETA.
So I think it's fair to say that after that length of time
you owe us some sort of ETA, and frequent updates on the restore status.

Would you kindly update the thread for server/network status
for Bulk Reseller Server cressida.nocdirect.com:
www.jaguarpc.com/forums/showthread.php?t=19851

Thank you
Mark B

.. > Yesterday 05:38 PM
.. > Topic: Emergency Maintenance:cressida.nocdirect.com
.. >
.. > www.jaguarpc.com/forums/showthread.php?t=19851
.. >
.. > The bulk reseller server cressida.nocdirect.com is
.. > currently down.The techs at our data center are working on it.
.. > Please wait we will update this thread as soon as
.. > the server will be back online.Thanks for your patience.


.. > Yesterday, 07:38 PM
.. > Our data center techs rebooted the server but it did not come up,
.. > upon further investigations, we came to know that there is a
.. > kind of RAID failure. Currently techs at data center are working
.. > on the server. Hopefully it will be back up soon. We will keep
.. > you updated and we appreciate your patience.


.. > Yesterday, 09:56 PM
.. > Unfortunately, there is RAID card failure on cressida, so now
.. > we have replaced the RAID card and mounted new drives on the server,
.. > currently we are restoring data to these drives from the backups.
.. > We are really sorry for the trouble you have to face, but we do
.. > appreciate your patience. We will keep you updated with the restore process.


.. > Today, 01:42 AM
.. > Restore process is in progress, we can not suggest any ETA
.. > at the moment. We will keep you updated.

[eHits]

.
ps --

I tried posting this under the server/network section but it does not allow
postings, hence my post in this section.

Also, I am posting here because over an hour ago
I opened support ticket 12689485 marked as Emergency.
It's been over an hour so far, and no reply from JPC.

[the_ancient]

.
Bulk Reseller Server (cressida.nocdirect.com) is down (12 hrs)

Hi,

About the above reseller server outage.... I'm hosting a law firm website
and they are desparate for email. One attorney needs email within the
next couple hours, for a time-sensitive matter.

If they dont have a Email Fail-over plan, and email is that critical to thier business they need one...

You can not expect everything to work 100% of the time, you have a have a Failure plan, and then plan on your failure plan failing.

Only then can you be somewhat prepared.

JPC is updating you from the looks of thing, your talking about Restoring an array of data probably well over a Terrabyte, that does not happen instantly, Even though I wish it could.

You has a web host should have a plan in place for this, and you has a web host should be offering directly or indirectly failover services to your mission critical clients,

[eHits]

You has a web host should have a plan in place for this, ,

ancient,

With all due respect, your comment is WAY off base. Every time a sever goes down,
do you immediately start going to your backup archives and doing restores
onto another server for your customers? NO WAY! You do not. (well at least I don't think you do)
What you do is wait, and wait, hoping that the server will come up any time now.

and you has a web host should be offering directly or indirectly failover services to your mission critical clients,

The law firm, and everyone else doing business online, claims that their email is VERY important
to their business. They say their business cannot run without email. Virtually every business says that.
Everyone's email is mission-critical (your term) to their business.
We do regular backups, just like you probably do.

But that's not the issue here.

The issue is that the server has been down for 15 hours, and it's been over 7 hours
since jpc provided an update of the status. At this point JPC owes it to their customers
to provide a status update. Do you not agree with that?

Quit trying to brownnose JPC and get real man!

Mark

[JPC-Tracie]

I've moved this to the reseller forums as it's not a pre-sales or sales question.

It also looks like an update has been posted.

[JPC-Rizwan]

Update has been posted on the forum, the data restore is still in progress and will take some time.

[eHits]

.
Rizwan, thanks for updating that thread. Can you also provide an ETA?
Ballpark is fine... approx 1-2 hours, 5 hours, etc

(i expect that with as many servers as jpc has, and the number of times
you've done this type of restoral, you must have things pretty down pat...
and by looking at the size of the backup data, you should be able to give a fair
guestimate of the restoral time)

[JPC-Tracie]

It's not that simple. What's on every server is different. Originally the restore was thought to take no more than 5-6 hours. It's obviously been longer than that. I'm not tech support, however, just letting you know that ETA's are hard to give and we are usually reluctant to give them. The actual process of doing these restores is down pat of course, but each one is different and there's no way to know really how long it will take.

[jason]

While I realize that you are commenting on the lack of updates about the downtime, the_ancient makes some valid points. If your client cannot live with email being down for several hours then they should be willing to pay for a redundancy plan.

Mission critical is all in the eye of the beholder. For my part-time business, although much of my business comes from people "randomly" finding my site in search results, a few hours of downtime probably isn't going to kill me. I don't have constant enough stream of customers coming to me to notice losses from downtime. I make my money from long-term customer relationships, not quick turnaround sales. My site is critical to my mission, and while I would hate to miss an opportunity because of downtime, the likelyhood that I will is pretty slim.

Mission critical has another meaning at my full-time employer, a university with 16,000 students and close to 3,000 faculty and staff. If, for example, our website or email were to go down for even a few minutes it would disrupt the activities of thousands of people. Imagine the final exam in our online course management system that failed to be submitted because of a glitch, or the online tuition payment that got deducted twice from one checking account because of a failure somewhere in the middle of the transaction... To prevent these kinds of things from happening we have multiple redundancy built in to our most critical systems. Sure, individual servers go down from time to time, but we don't wait to see if it comes back up, we immediately and automatically cut over to another already running server to avoid even a minute of downtime. Because of the foresight that our central IT staff puts into these systems most people never know of the problem. To them, everything is always up. All of this does cost a fortune, but our needs justify the cost.

There are failover options out there for businesses that have a critical need to be connected at all times and email is probably the easiest to implement. Consider offing it, at an extra charge of course, to clients that decide they really need their email at all costs.

--Jason

[the_ancient]

ancient,

With all due respect, your comment is WAY off base. Every time a sever goes down,
do you immediately start going to your backup archives and doing restores
onto another server for your customers? NO WAY! You do not. (well at least I don't think you do)
What you do is wait, and wait, hoping that the server will come up any time now.

I am not a reseller, I stopped doing that because there is no money in reselling.

Part of my FT job is making sure the data infrastructure of my company is up and running all of the time,

Guess what I don't pull data from backups on mission critical item, there are ALWAYS at minmum 2 servers just waiting for 1 to fail so they can do something, data is sync'ed to them in real time

The law firm, and everyone else doing business online, claims that their email is VERY important
to their business. They say their business cannot run without email. Virtually every business says that.
Everyone's email is mission-critical (your term) to their business.
We do regular backups, just like you probably do.

I dont, my email is meaningless to me, I do not rely on email for anything important. Email is the most unreliable, unsecure communications protocall on the face of the planet


My companies Database is Mission Critical, if it goes down we are dead in the water, So we have plans in place, if Server A goes down, Server B is instantly Active with real time data, if Both went down in some fashion, I can have Server C online in a mater of hours.

But that's not the issue here.

The issue is that the server has been down for 15 hours, and it's been over 7 hours
since jpc provided an update of the status. At this point JPC owes it to their customers
to provide a status update. Do you not agree with that?

Quit trying to brownnose JPC and get real man!

Mark

Me Brown nose? ROFL you apparently have not been around here long, I have regularly complained to the very owner of this place. I have him on speed dial

The fact of the matter is, they appear to be updating you when the ir is something to update, Restoring form Backups take time, that is the curse of large data arrays

[the_ancient]

the_ancient makes some valid points.

SOME!!!!!!

Everything that come out of my mouth if the valid point

[Ron]

eHits,
I'm sorry that you're having problems. I don't want to kick you while you are down, and he is usually so far off base on most things, this pains me to say it, but the_ancient is right on target with this one. I would think that a law firm needs a MUCH more robust system than a plain bulk reseller cpanel plan.

I hope your ToS covers you...

Tell your client to open a GMail account to get the files they need in this backup situation, and to CALL their correspondent.

I hope it comes up quickly for you.

[eHits]

While I realize that you are commenting on the lack of updates about the downtime, the_ancient makes some valid points. If your client cannot live with email being down for several hours then they should be willing to pay for a redundancy plan.

Jason and everyone who commented,

Thanks for your candid replies. My problem with what I am hearing is you are saying
that for all of our hosting clients, (remember, earlier I said that virtually every business
claims their email and website are critical) we should have a fail-over system with
each business client's website/db/email mirrored on a server#2 and server#3.

Think about what you are saying.

--> You are saying that JPC, or any provider, is so unreliable
--> that it's necessary to have such a fail-over plan for all your business customers

Sounds kinda crazy. And the opposite of what you'd think after viewing any
provider's website, which touts their uptime, reliability and the like.
(don't get me wrong, I do understand that JPC or any provider will have downtime,
and I'm 100% fine with that. It's part of the biz)

MY ORIGINAL POINT....

The only point in my original post was that it was 5 hours since the last update
about the server/network, and I wanted to see another update.
That's it. Really. scroll up and re-read the post.

.. > The server has been down for over 12 hours.
.. > And 5 hours ago JPC said they cannot suggest any ETA.
.. > So I think it's fair to say that after that length of time
.. > you owe us some sort of ETA, and frequent updates on the restore status.

Thanks again ancient, Jason and others
who gave their candid comments.

[Ron]

In the past, some reseller servers have been down for 3 days for a full backup restore.

Yes, I think that you are correct that for the majority of businesses that require their website or email as a mission critical tool, a plain vanilla reseller plan is not suitable. JAG is/will be offering a failover system on their SDX plans, and perhaps you could have a backup mailserver somewhere, or some sort of a failover plan.

I can live with my server gone for a day or three (I would NOT be happy, but I'd get over it) but if I couldn't, I'd be looking at other more robust alternatives. If the website/email is truly mission critical and an outage might cause irreparable harm, then an investment of more than $20 or $30 a month is required to keep it running. It is just a completely different class of operation.

[eHits]

Can anyone recommend some fail-over systems?
thx! Mark