Is that all there is?

[Automile]

DNS issues have been fixed so all domains should be correctly resolving at this time.

To quote Peggy Lee, is that all there is?

Almost every site at Aletia was down ALL DAY today and this is all we get?

I'd like to know what the resolution to this problem was. I'd also like to know what steps Aletia intends to take IMMEDIATELY to prevent down time like this from happening again.

In the last seven days, my site has spent a total of over 24 hours unavailable to my visitors. Over the past month, it has been unavailable repeatedly. I feel an explanation and a plan of action is called for.

Today was a wake up call.

Andy

[timechange]

Short answer: yes.

[Terence]

It seems like the nameserver problems had something to do with some sort of control panel system upgrade.

Annette of Hosting Matters first mentioned at WebHostingTalk.com, regarding the nameserver problems:

Probably related to the BIND updates in last night's cPanel updates...

The admin of Superior Host, which uses PWebTech like Aletia, had this to say regarding the DNS failures:

The servers went down for several hours today due to a control panel system upgrade. This caused over 700 web servers to go off line.

Until the problem was fixed, there was nothing we could do about it from our end. It basically took a couple of files out that needed to be there to make web sites come up by their name.

It did however take several hours for the data center to get to our servers, as the files had to be manually added to each web server, then run a new control panel update to fix the other internal issues it caused.

...

I am sorry for the mishap, but there was nothing we could do for hours, and wanted you all to know that there was nothing physically wrong with the servers, but they were all off line by name for several hours.

[timechange]

CPanel = evil

[megmond]

So,

basically what I read here between the lines, is that in order to know what the problem is on 'our' aletia-servers, we have to hear from other hosting companies?

That's a bit bizarre...

I'll be looking for a new host though. My own company will pretty soon depend on more uptime than I get here so I can not afford to take the risk for much longer... :sick:

[Automile]

So, basically what I read here between the lines, is that in order to know what the problem is on 'our' aletia-servers, we have to hear from other hosting companies?

That is pretty bad, and the other hosting company said they were unavailable for several hours. Aletia was down ALL DAY!

So, was Aletia looking for a name server problem, when in fact they should have been adding files to the server, related to the Cpanel update?

This whole situation leaves me with a bad taste in my mouth, due to:

- Limited information during the downtime, which should have been updated at least every couple of hours.

- Inaccurate information as to what the real problem was; or at best the information wasn't complete.

- Ongoing issues with high server loads, downtime, and poor communication between support and customers.

I have no way of knowing how much money I lost yesterday, and I know that Aletia doesn't care. After all, it wasn't their fault, was it?

It seems to me that a post yesterday telling us this was a Cpanel problem due to an update, as well as possibly something else related to the update, we would have at least known they weren't negligent in setting up the redundancy of the service.

Aletia needs to dramatically improve communication and server uptime between now and the time my one year contract is up, or I'll have to go elsewhere. The repeated downtime and slowness of the servers due to high loads of the last few months cannot be allowed to continue.

I have no choice; I'm being forced to leave. I don't know how many customers Aletia lost yesterday, and will lose because of yesterday, but this is a cumulative problem that continues to worsen with each unscheduled server reboot.

Aletia's continued silence regarding issues only says one thing to me: We don't care about you as a customer, we don't care if your site works properly, we don't care if you lose money because your site isn't available.

As a customer, what options does that leave me? None. I have to go elsewhere.

As I said earlier, I hope this is a wake up call for Aletia.

Andy

[Terence]

I came across more detailed information about the nameserver outage:

As most of you are aware, every night the control panel runs an automated update. The update is to incorporate new features, fix existing issues, and so on. In general, we have experienced very few issues related to the updates, and Saturday was no exception.

BIND, for those who are asking, is simply the name services that run on the machines. Name services, for those who are now telling me not to be such a geek, are what allow you to type in domain.com and reach your site (or someone else's) instead of having to remember a series of IPs to move from site to site. It provides the translation necessary for the server to understand that a name (domain.com) is actually a number (1.2.3.4) - the language of machines.

One of the symptoms of BIND failures, assuming no other failures, is that sites can be reached by IP address but not by name. For some hosts Saturday morning, this was the extent of the problem. Because the syntax used in the latest versions of BIND is different than in previous versions, in many cases it was simply a matter of adjusting one of the configuration files and restarting. We encountered a couple of errors ourselves, but corrected them as soon as they popped up - in addition to cleaning up some unneeded entries, as that seemed an opportune time for it. Some of the hosts to whom we spoke during the wee hours Saturday had more or less the same problem.

Some hosts, however, were not so lucky. Several wound up with entire libraries corrupted - one to the point where their main machine could not be booted at all, requiring a complete rebuild. Some of these problems appear to be the result of nonstandard configurations at several hosts, and at others were due to the update itself overrunning certain installation tasks - that is, trying to move on to another piece of the update before a task on which that next piece was dependent had completed properly. In both cases, the most likely resolution was a rebuild of BIND from source - not a pleasant task at 3 AM. In several cases, it was a simpler fix of just copying the needed libraries from a known good system and moving from there.

[Not4You]

That looks familiar.

[jkeller]

Automile, thanks for that information. It's great to have a detailed explanation.

May I ask what source you quoted? Not out of doubt, just because I'm curious that way.

[Automile]

jkeller,

Automile, thanks for that information. It's great to have a detailed explanation.

May I ask what source you quoted? Not out of doubt, just because I'm curious that way.

Do you mean terence? My post mostly dealt with my frustration, terence's had the notice about downtime.

Andy

[jkeller]

Originally posted by Automile
Do you mean terence? My post mostly dealt with my frustration, terence's had the notice about downtime.

Ooops! Right you are. I meant Terence. Sorry about that.

[timechange]

Maybe then the CPanel "wizards" should be made to pay about this freaking mess!

[jkeller]

That's be nice, but I doubt it's possible. However, it isn't the first time that it's happened. I remember a previous outage from a CPanel screwup. I think it messed up something to do with the mail configuration -- back in May or so. It was the first time Cobra had gone down for any length of time since I'd gotten here.

It sounds like we'll be moved to a HostGUI setup when we're transfered, barring any changes in business decisions. Since it looks like JaguarPC is part of the same company as HostGUI, that means just a smidge better accountability. Money reparations matter less than a chain of command/communication -- stop it from happening, rather than ripostes after the outage.

I'm looking more and more forward to this changeover.