Configuring incremental backups?

[heavyharmonies]

I'm tired of dealing with terse support personnel, so I'm hoping someone here can help with some answers in lay terms.

I'm sure the tech support folks are probably proficient, but they seemingly have no patience for, or inclination to, explain things to people who aren't experienced server admins. DEHE support was at least forthcoming with suggestions, options, and assistance. With Jag support, it's like pulling teeth: it's painful, and it's pissing me off.

I've been told that I should get a remote backup:

You need backup space. Here are some of the backup plans we offer:

http://www.jaguarpc.com/services/backups.php

Once you have purchased the backup space, we can configure your vps to do backups from WHM.

to which I responded:

1. What backup method does that use? From what I've been able to glean from reading at WHT, people generally recommend an rsync backup, as that is the most bandwidth-effective method for doing incremental backups.

2. How much backup space will I require? I'm only concerned with the /public_html hierarchies of all the domains (not logs, mail files, etc.). I just went through all my domains and that is approximately 4.5GB at the moment.

3. How long can backups be kept? I would want at least a week's worth of nightly backups retained at any given time. Obviously, doing full backups nightly, this becomes cost prohibitive, but using rsync should use considerably less, correct?

4. Assuming at some point I need to have a file or directory restored from backup, how do I do this? Do I simply file a support ticket with the domain name, directory, filename, and the date I want the files restored from?

The response I got (I'm sorry but these terse partial replies are unacceptable IMO):

1) It can use ftp or rsync. Cpanel only supports ftp.

2) The space requirements depends on how you configure your backup in WHM.

3) Cpanel allows you keep daily, weekly and monthly backup.

4) You can download the backup using ftp yourself.

to which I responded:

I'm sorry, but I do not understand your short replies.

First you say I need a a WHM backup, but in your last message you say cpanel backup. Which is it? I asked about rsync, but you say cpanel does not support it. How would downloading a backup using ftp restore a certain file or directory?

Is there anyone there that could please answer the questions in layman's terms? It's making no sense to me at all right now.

and then I received:

"First you say I need a a WHM backup, but in your last message you say cpanel backup. Which is it?"

Cpanel and WHM (Web Host Manager) is one product with two different levels of access. I hope you have used WHM? The backups you will configure from your WHM are called cpanel backups as they are backups of individual accounts.

"I asked about rsync, but you say cpanel does not support it."

Yes, it can backup using ftp only.

"How would downloading a backup using ftp restore a certain file or directory?"

You can not access certain file or directory. You will need to download backup of the account to extract the required file/folder.

to which I replied:

So how can I do incremental backups then? You say I have to use backup by ftp, but WHM says that incremental backups cannot be done by ftp.

and received back:

Yes, cpanel does not support remote incremental backups. If you want to do remote incremental backups, you will need to write custom code to do that. Or you can create incremental backups on your vps itself.

I'm tired of going around in circles. Can someone explain in English what the options are to try and do remote incremental backups of portions of multiple domains on a VPS, either with WHM or Cpanel or FTP or rsync or whatever?

If I'm understanding any of this correctly, the only backups JaguarPC will help configure are WHM backups, which are nonincremental FTP backups, meaning that in order to retain 7 days worth of nightly backup, I would need a backup space at least 7x the space of my entire VPS (since I cannot just choose to archive the public_html hierarchy for each domain; I have to backup the whole thing). That plus the bandwidth costs would be ungodly expensive. Then if I ever needed to restore a single file or directory I would first need to download the ENTIRE backup to my local machine, extract the files I need, and then manually upload them to the VPS.

People talk about backups being efficient and cost effective... this sure as heck ain't it.

Should I be asking these questions of the sales department rather than support? Or as Jag support seems to imply, I'm on my own in figuring out what my options are?

P.S. Can anyone recommend a VPS host that provides support in English to the lay person? I had it with DEHE, but it's gone now... and I've reached my frustration limit with JaguarPC.

[thisisit3]

I'm also looking into incemental backups for VPS and dedicated servers.

Their replies don't help much, thats what you get for talking to level 1 support, they can't handle too many questions at once and they won't respond with long explanations.

Anyway, back to the issue at hand. Ofcourse rsync is the only way to go, so ignore the ftp methods. From what i understand, cPanel/WHM can do individual backups of your domain accounts and settings. I'm personaly looking at something that can backup the entire server and keep an rsync'ed copy at a remote location, so the only solution is to create my own script.

I'll look around at other forums and see what other people are using.

[Ron]

rsync is the most basic type of backup you can do. There are much more robust backup solutions available, or I shoudl say used to be available when I was into such things. Things like incremental backups with versioning to arbitrary points in time.

JAG keeps saying that they have a much more robust backup system coming online, but we never hear any details about it. Perhaps when they have this system in place for their shared servers they will make it available as part of their backup space service.

A basic rsync will keep 1 copy of your server up to date.

I strongly recommend that you not use rsync to backup your database files, especially if you have smallish DBs. I suggest you dump the databases and then the rsync will backup the dumpfile. I have had issues with trying to restore rsync'd DBs, including tables corrupted and tables out of sync with other tables.

I use WinSCP to keep my sites backed up to my PC.

It will be good when JAG or a customer *cough*thisisit3*cough* comes up with and publishes a great solution.

[thisisit3]

yeah it would be good when *someone* comes up with such a solution

[heavyharmonies]

Thanks for the replies. I believe I do need some sort of versioned backup, or at the very least, the ability to keep a week's worth of separate backups, hence my contention that WHM backups are completely too cost prohibitive.

The problem with a system that keeps a single backup up to date, is that (by my understanding) if you don't catch the problem before the next iteration of the backup, you're screwed. The error will have carried through to the backup.

I'm trying to find a way to guard against the 5-day rollback caused by the UL8 server outage (the rollback that support claims never happened ), meaning that if it's not caught until 2-3 days after the event, I need a way to be able to go to a backup earlier than 1 day.

A bigger problem, is that from what I can tell by the WHM backup options, I cannot choose *portions* of an account to backup, I need to take the whole thing, log files and temp files included, meaning the cost is going to be prohibitive.

Right now my VPS is using 9GB of disk space. Of that, approximately 50% is data I need to have archived. If I have to archive the full 9GB en masse, and want to keep 7 days of nightly backups, that's 63GB... since apparently I have to do a full backup nightly.

This doesn't even get into how doing backups of 15 or so domains every night is going to effect the performance of the VPS. Then there's the issue of trying to restore a single file or directory from the backup. For example, my largest domain has about 4GB of disk space used. From what I'm understanding, I'd have to download the full 4GB backup to my local workstation, uncompress it, find the files I need and manually upload them back to the VPS. This just strikes me as clumsy and time consuming.

Or am I misunderstanding?

[thisisit3]

You are not.

After doing my research, i believe that these are our options:

- cpanel does backups per domain only
- whm does full WHM/cPanel backups for all domains
- virtuozzo does complete VPS backups (for VPS only, not dedicated)
- custom made backup solution

cPanel and WHM are out of the question, virtuozzo is only for VPS servers and doesn't allow incremental backups.

i believe our only solution is a custom make backup system, either a custom made script or one of the available Linux backup applications floating around (some are commercial).

[Gwaihir]

Note that rsync itself is mainly a very efficient mechanism to sync files, not a "gui" or "backup program" around it. Most people automate that part via a simple script. There are plenty of example scripts on how to make rotating backups with rsync .

Rsync is much more effective at syncing than for example WinSCP. (I tried using that for syncing once, after Ron suggested it last time. I broke it off after 20 minutes at which point it was still comparing folders. I then ran my rsync based script and that was done in 40 seconds.) The ways it achieves this efficiency is by transfering only the changes in files (almost any other solution transfers changed files completely) and apparently by a rather efficient way of comparing folders for updates too. (I doubt that part is unique, I suspect it's just not one of WinSCP's strenghts.)

[AlexKall]

I tried to do a automated backup, everything worked great besides the fact that you could not pass on username and password over SSH using Rsync on JAG backup so it failed

[JPC-Masood]

I tried to do a automated backup, everything worked great besides the fact that you could not pass on username and password over SSH using Rsync on JAG backup so it failed

You can get in touch with support to setup ssh keys from your vps/server to backup account.

[AlexKall]

Really? I got the answer of it being impossbile from support, was quite some time ago though so I guess it changed?

Anyways glad to hear that masood!

I guess I might have to get my backup space again hehe

[Gwaihir]

Indeed you can't pass on username and password over SSH, but you can set up a key instead. There's a good tutorial here. You could wonder if you need SSH when transferring data inside the JagPC network only (or are the backup servers not in the same datacenter cage?)

[Vin DSL]

Oh, my, this is interesting... read the whole thing!

I don't understand how backups can be so difficult....

I just TAR the whole mess (using BZ2) and download it to my local PC. Plus the obligatory MySQL dump[s], of course. Simple pimple!

All this incremental stuff is what's causing the confusion, I think!

Just do full backups! Forget all that Rsync, Robocopy, WinSCP, WHM, cPanel, FTP crap...

Who wants to spend hours or days restoring incremantal backups, in case of emergency?

Not me... give me the shotgun every time!

[AlexKall]

Indeed you can't pass on username and password over SSH, but you can set up a key instead. There's a good tutorial here. You could wonder if you need SSH when transferring data inside the JagPC network only (or are the backup servers not in the same datacenter cage?)

Yeah that was what I ment, the keys. It was a long time ago so i can't remember all the fancy wording for it. I didn't really mean passing passwords I ment communicating but that got lost in the translation

Oh, my, this is interesting... read the whole thing!

I don't understand how backups can be so difficult....

I just TAR the whole mess (using BZ2) and download it to my local PC. Plus the obligatory MySQL dump[s], of course. Simple pimple!

All this incremental stuff is what's causing the confusion, I think!

Just do full backups! Forget all that Rsync, Robocopy, WinSCP, WHM, cPanel, FTP horsesh!t...

Who wants to spend hours or days restoring incremantal backups, in case of emergency?

Not me... give me the shotgun every time!

Thats not automatic, as an Automationtechnician I need to try to make everything automatic

[Vin DSL]

I'm tired of going around in circles. Can someone explain in English what the options are to try and do remote incremental backups of portions of multiple domains on a VPS, either with WHM or Cpanel or FTP or rsync or whatever?

JAG keeps saying that they have a much more robust backup system coming online, but we never hear any details about it. Perhaps when they have this system in place for their shared servers they will make it available as part of their backup space service.

I read something, a long while back, about the Vietnam War -- said it took, like, 10 ppl in the rear to support 1 soldier in the field...

This reminds me of backups -- and the reason nobody does them... 1 GB production web site takes 10 GB of backups, incrementally backed up daily/hourly, yada, yada... Pretty soon the whole war becomes supply, support, and backup -- and most of the time, none of it works anyway -- Murphy's Law.

In simple English... keep it simple!

When you *think* about it, ALL backups are incremental...

Personally, the way I keep the size/bandwidth down is by NOT backing up cache files. This makes a huge difference. By excluding them, my entire account compresses down to about 700MB -- download it to my local PC -- run an integrity check -- I'm all set!

The devil is in details. The more details involved, the more devilish the task becomes, you know?

[Vin DSL]

Here's an interesting post, IMHO...

Disaster Recovery / Business Resumption plan

Looked like Houston was going to disappear, and take everyones' web site with it -- no time to argue over backup scheduling and augmentation...

Doesn't get any more simple than this!

Here's how I deal with cache files, and other crap I don't care about backing up -- one-liner...

tar -cvpjf Lenon.com_`date +%m-%d-%y`_Backup.tbz /home/<account name> --exclude=/home/<account name>/etc-bkp --exclude=/home/<account name>/mail-bkp --exclude=/home/<account name>/tmp/analog/cache --exclude=/home/<account name>/.trash/cache --exclude=/home/<account name>/g2data/cache --exclude=/home/<account name>/public_html/e107/e107_files/cache --exclude=/home/<account name>/public_html/info/wp-content/cache --exclude=/home/<account name>/public_html/joomla/cache --exclude=/home/<account name>/public_html/joomla/includes/Cache --exclude=/home/<account name>/public_html/joomla/includes/patTemplate/patTemplate/TemplateCache --exclude=/home/<account name>/public_html/modules/ODP/cache --exclude=/home/<account name>/public_html/phpMyAdmin/save --exclude=/home/<account name>/public_html/pilot/modules/ODP/cache --exclude=/home/<account name>/public_html_old --exclude=/home/<account name>/www