Files reverting to earlier states after UL8 Maintenance

[heavyharmonies]

I'm getting stonewalled by support on this, even after I requested that the issue be escalated to a senior tech or management. (Ticket 12450727)

There was a major outage to server UL8 on April 17. The server was down for several hours. What "maintenance" was done, I have no idea.

I am now discovering that ALL files that were either added or modified between April 12 and April 17 reverted to their state as of April 12 (either rolled back, or nonexistent if the files were added to the server within that time period), and then were modified going forward on April 17.

This is not isolated to a single domain. This is across all domains on my VPS.

Files added and modifications made post-April 17 are there. It's as if the VPS was brought back up using a backup from April 12. Web stats under cpanel on all domains support this. There's a huge gap showing ZERO visitors from April 13 through April 18, yet the web sites were live over that entire period.

I have to ask what the heck happened?

If it was necessary to restore files due to a server crash, I can understand that, BUT:

1. Why was I not informed that it would be necessary to use an old backup and that I would lose 5 days worth of data. At the very least I could have been proactive about this.

2. I'm supposed to have NIGHTLY off-server backups. It was configured while I was with DEHE, and WHM is configured to perform these backups. Why then, was it necessary to revert to an April 12 backup of the VPS?

I and others have now lost many man hours of work because of this.

I realize that I probably should have noticed this 2 days ago, but in all candor I never anticipated this scenario. The difficulty is trying to now sort out all of the changes that were incorrectly done to thousands of files after April 17, supposedly to current files, that were in fact reverted files as of April 12. How the heck do I somehow insert 5 days of work in to the middle?

It's a giant cluster****.

More importantly, how do I assure that this NEVER EVER happens again?

[Connie]

I can only say make your own backups to back up JagPC for files that are critical.

[Gwaihir]

Yeah, but if you're not told that you need to put them back, even that is of limited use..

[heavyharmonies]

Yeah, something has definitely happened.

I just went to do some tweaking on a new domain that I had registered and configured on April 17, and it's gone. No trace of it at all in WHM.

So much for the backup service...

[heavyharmonies]

Just wanted to follow up on this.

As it turns out I was neither insane nor imagining things.

For whatever reason, there were two installations of my VPS on the network, one on server UL8 and one on server UL10, and periodically DNS would change from one to the other, thus giving the impression that the VPS had "rolled back" to whatever point the other VPS was last accessed.

Talk about a data headache...

However all is well now.

-Dan

[thisisit3]

If you are interested to do proper backups for your VPS then read my signature