Some Servers down?

[mfoxglove]

I was just in the middle of tinkering with something on my ftp when the server suddenly went down! What happened? I hope it gets fixed up soon. Of all the times to go down, it went down right in the middle of something, lol. Couldn't it have waited for another hour, until I went to bed? (lol )

[prattboy]

Most likely due to scheduled maintenance: http://jaguarpc.com/forums/showthrea...threadid=10741

[ChanceR]

Since around 2 AM this morning Carbon, Iron, Gold, Plutonium, Xenon and Lead have been 100% dead.

The only mention of a scheduled maintenece I could find is in this post:

http://jaguarpc.com/forums/showthrea...threadid=10741

which conveniently has no mention of any server name and there's a mention of Brief outages. Well, I don't classify a 5 hour outage as a brief outage.

Any news as what the heck is going on and how long it's going to take? Aren't these things supposed to be announced in advance?

[Stargazer]

Nobody seems to be aware...

Maybe there's nobody monitoring it? We're not supposed to put in tickets when the server status shows them down, but this is getting to the point where you have to wonder if anyone knows they are down...



Shouldn't there have been an update to the 'scheduled maintenence' topic by now? It's been more than 8 hours since the it was scheduled to have been taken down.

[JPC-Greg]

I merged a couple threads on this subject and am checking into this now to see what my staff is or isnt doing.

[GivenRandy]

Yep, my Xenon is dead. Wasn't maintenance scheduled for yesterday afternoon and off-hours?

[siceman105]

Carbons been dead for a while now

[Stargazer]

Hi,

This was the last update:

We will also be adding a redundant layer 3 switch to the network tonight around 11 P.M. CST so you may notice outages at that time as well.

That was about 9 hours ago..

Jag is working on it now, so expect an update soon

Cheers

[Stargazer]

looks to be back up now...

[lookout]

Yes, the couple of servers that I was watching just came back to life too.

[lookout]

FWIW, I just checked, and the network status page is showing all clear again too. Even the php errors that had been showing up there last night (from some servers not responding) have cleared up. Odd that gold and carbon showed all red lights last night (now green) and the other downed servers were just showing up as a php error on line 54. I wonder why the network monitoring was showing up differently on them.

[JPC-Greg]

We've got a new version of our monitoring system coming out soon when we push the release of the other updates we've been talking about. Just about there, if should give everyone a really nice, well informed, and integrated experience here.

I did not work on any server issues but I will be sure to pass on that information when I have a clear and concise answer.

[prattboy]

There was quite a bit of downtime last night (5 or 6 hours, maybe? I'll have to double check); my site on genesis was unreachable. It turns out IPaliases went down on the server and it had to be rebooted. I'm not sure how this related to the scheduled maintenance and why it wasn't caught sooner. If an IP becomes unreachable, shouldn't that set off red flags for the server admins?

I'm curious to find out!

[hughnet]

a pity that the people answering the support tickets using a standard "outage script" couldn't have updated the forum post to indicate that the outage was more than "brief outages" but had turned into something major on a few machines (including Lead that I am on)

[jason]

NEarly every site that JPC hosts has a unique IP. They only wa to know for sure they are all up is to ping every single one of them which would create a lot of unecessary traffic in the NOC and slow the whole network down. Generally, if the server is up every site on it is reachable unless some sort of routing failure occurs, as seems to be the case here. Those types of things are rare and usually easy to repair if you report it to support.

--Jason