dione is down.

[Phaedrus]

dione.nocdirect.com is down.

Is somebody working on it?

Thanks...

[JPC-Anoop]

Server is back online and all services are running fine. Please open a ticket with support, if you're facing any issue.

[Gwaihir]

I had a note about maintenance in my mail. Perhaps that was it?

Bit weird though: the point of a RAID array is that it can all continue if you replace a drive, so why announce downtime for that (without further info)? On the other hand, looking at the other thread, it looks like the opportunity was NOT taken to do the kernel upgrade at this time.

[JPC-Rizwan]

The servers which do not have hot swap bay ( mostly the older ones ) do require a shutdown/reboot for drive replacements.

If you are also referring to dione server, then it still has some open issue as drive replacement was not successful and is being investigated as mentioned in related forum thread as well.

For Kernel , it was upgraded on Dione during that reboot or any other servers which have been rebooted for disk or other issues in last few days.

[Gwaihir]

The servers which do not have hot swap bay ( mostly the older ones ) do require a shutdown/reboot for drive replacements.

Ah, right. Thanks for the info.

If you are also referring to dione server, then it still has some open issue as drive replacement was not successful and is being investigated as mentioned in related forum thread as well.

Yes, dione, of course, as this thread is about dione. I noticed and read the info thread a moment later, but thanks for the pointer. Hope you'll get it fixed smoothly .

For Kernel , it was upgraded on Dione during that reboot or any other servers which have been rebooted for disk or other issues in last few days.

I went by the info in the kernel upgrade thread. I see you've posted it there now too .

[Gwaihir]

Quick question for JagPC about the server switch: will it actually have a new name, or can that be avoided?

The info thread mentions newdione.nocdirect.com, but I hope that's just till you're ready to switch that to "current" dione.nocdirect.com (and perhaps the other to olddione for however long it will still be needed). It would save me, and I guess all of it's users, some updating of bookmarks, scripts, email settings and such.

[JPC-Rizwan]

The new hardware name will be newdione.nocdirect.com , however all IPs from old server will be moved to new server so hostname dione.nocdirect.com will also be pointing to new server and will continue to work.

[Gwaihir]

Any ETA on the move? You aren't forgetting us Dione users, I hope? It's been syncing for over a week now.

With that crippled array, it is not a happy machine. Throughout the day it often just can't keep up. I trust what you guys told me in my ticket: that it's a direct result of the reduced I/O performance. That means that this fix is not something to do "oh, some day".

Some stats from today, to illustrate the point:

00:35:16           load average: 204.61,    110.36,     48.31
03:02:57           load average: 169.52,     74.39,     31.10
05:50:24           load average: 135.49,    135.96,     66.12
11:30:08           load average:  66.54,    188.23,    125.43
14:23:13           load average: 175.32,     73.13,     30.57

I'm amazed at what a machine like this manages to handle, but this is out of its league. The scripts behind your status page may have enough patience to make for good looking stats percentages still, but from a user point of view the sites on dione are effectively out of operation for an hour or two each day and have been that way since September 12 at least. (The 12th is when I myself first noticed and sent in a ticket.)

[JPC-Rizwan]

Gwaihir,

We are sorry for the continued problem on this server. No we have not forget this and still working on it, unfortunately the new hardware faced problems as well during the rsync process which itself has been very slow due to old server's disk issues. I will update the original thread as well in a min.

[Gwaihir]

Eh.. are you doing the initial filling of the new machine by rsync from the crippled machine? I'd think that under these circumstances you'd start with most the recent backup and only start to rsync after that's on the new machine, or is even that taking so long? If the latter is the case, are the daily backups still being made at all, or is that too impossible? I'd hate to think this machine is running on a no longer fault tolerant RAID array without backups for three weeks now..

Thanks for the update in the info thread, but it doesn't mean much without an estimate of how long this will still take, as the usual two or three days tops clearly doesn't apply. So I can only repeat my question: what can we expect? What is the ETA on a complete solution? What service level (availability and backups) is offered in the meantime?

[JPC-Awais]

Yes we are doing initial data migration using rsync. Once it is complete we will do a final sync with all services stopped. More than half of the data have been copied so far on new machine. The migration will complete in next 24 hours.

[Gwaihir]

Post migration, SSL certificate for dione is no longer working , neither web nor email.

SSL for my own domain doesn't even connect, nor does anonymous FTP.

I guess I'll go post this in a ticket now. I did not receive an email message about this maintenance either. As you clearly need clients to check things themselves now, that would have been in order.

[Gwaihir]

Tried by the new name as well: https://newdione.nocdirect.com:2083/ that doesn't connect to anything at all. Meanwhile it seems that much of my email doesn't get through, so I REALLY want to get to that control panel on the double! (I fear updating my ticket (#13184977), as that is already gathering dust and I've heard adding info only bumps it back down the line.)

[JPC-Jawad]

Gwaihir, your tickets are being checked as I am writing this. For control panel access, you can temporarily use your domain name like https://yoursite.com:2083 and ignore the SSL warning. Hostname SSL issue will be resolved shortly and you will be updated in the ticket. We apologize for the inconvenience faced so far.