Backup Recommendation

[ptmuldoon]

I'm looking for some advice on the best way to set up some automated site and database backups on a routine basis.

Essentially, I like to create a backup of my sql databases probably every 30 minutes, and then have that backup zip (or gzip) and then ftp'd to my own ftp server. The backups would than old keep the last 12 or so backups with the older one's being removed automatically.

If possible, I'd like to have the all the site (php scripts, images, etc) also be backed up and ftp'd as well maybe oncee a day or so as well.

I think some of this is doable with a cron job, but I'm not sure about the ftp'ing of the data some where else.

I would prefer to simply find a piece of software to install on a home pc that could simply connect into my site and databases and then back up the data on the home pc.

Anyone have any suggestions?

[JPC-Masood]

This is what I'll recommend:

1. Setup a Linux Machine or Virtual Machine (using vmware) at your home/office OR Signup for a Backup Space through JaguarPC OR Signup for another hosting account to use its space as Backup.

2. Setup ssh keys between #1 and your hosting account

3. Use mysqldump to dump dbs on your hosting account via cron (do not overkill by doing it too often, 30 minutes looks too much especially if you have large database)

4. rsync all your data

5. Generate a Full Backup from cpanel every week or month. (which will get copied to backup space in step #4)

6. Verify that backup process is working and document disaster recovery (how you will restore your data in case of emergency).

There are lots of tutorials available with step by step instructions on how to do that if you google. Some steps are already in our KB as well.

[Gwaihir]

1. My home machine runs Windows XP. Using cgwin, that works fine too. Cygwin's installer installed rsync and OpenSSH for me (and helps me update those occassionally). The tasks are run from Windows own built in task planner.

5. I must admit that I don't frequently do this step, which means I'd probably have to redo a number of settings, should disaster strike.

The main reason is that this is a very clumsy step:
a) I do not want a Full Backup, as files and databases are already taken care of. I want a backup of only the settings (for FTP, e-mail and such), rather than waste time, space and bandwith on making and syncing a backup that again includes the files and databases.
b) I can't automate this step, or can I?
Can't JagPC offer some way around this? Either press CPanel to build it (yeah, right), or write a tool yourselves?

[the_ancient]

1. My home machine runs Windows XP. Using cgwin, that works fine too. Cygwin's installer installed rsync and OpenSSH for me (and helps me update those occassionally). The tasks are run from Windows own built in task planner.

5. I must admit that I don't frequently do this step, which means I'd probably have to redo a number of settings, should disaster strike.

The main reason is that this is a very clumsy step:
a) I do not want a Full Backup, as files and databases are already taken care of. I want a backup of only the settings (for FTP, e-mail and such), rather than waste time, space and bandwith on making and syncing a backup that again includes the files and databases.
b) I can't automate this step, or can I?
Can't JagPC offer some way around this? Either press CPanel to build it (yeah, right), or write a tool yourselves?

you can never have too many backups.......... So even though you have the data backed up I would still do a FULL back up

[ptmuldoon]

Thanks for the tips guys.

I already have an new ftp setup at home to use to back things up. I'm just trying to determine the best way to approach it.

I'll likely try to setup a cron job to dump and gzip the the sql databases, keeping only the last 6 to 8 on the site.

I've used to synback in the past, and maybe that will be the easiest approach. I'l' just have to fine a good script to use for the cron job. I use to have one, and I'll have to see if I can still find it.

[JPC-Masood]

Here is a snippet of code I use personally:

#!/bin/sh
LOCAL=my.local.machine.ip.or.hostname
/usr/bin/rsync -zaHlv --delete --progress -e /usr/bin/ssh /home/username $LOCAL:/backup/mydomain.com/

Please edit it appropriately for your use.

[wgiese]

I follow Massod's steps almost identically - works like a champ. Not to belabor a recent problem I had on my SDX server, but I just had reason to perform disaster recovery, and can vouch for how quickly this method can get you back online. Even if Jag can only restore portions of your data, or what they do have is old, reversing the rsync from your local Linux box back to your server gets you online very quickly, with minimal data loss.

Some comments/clarifications as it relates to my setup:

3. Use mysqldump to dump dbs on your hosting account via cron (do not overkill by doing it too often, 30 minutes looks too much especially if you have large database)

I use local cron on my Linux box, and execute this step over ssh just a few minutes before my local rsync cron job. It's more secure, as your passwords for mysql remain on your local box (presumably behind a firewall), not on Jag's public servers. Also, pipe your mysqldump into gzip, it will reduce your export file size tremendously, particularly if you have large forums. Doing so also reduces disk usage on your server, and makes the rsync process go much more quickly.

5. Generate a Full Backup from cpanel every week or month. (which will get copied to backup space in step #4)

I handle this differently.

On my local Linux box, I've got another cron job that tars/gzips the current local copy of my site, and then I use logrotate to keep as many iterations of this full backup as I want. It's more flexible and automatic, and saves the transfer of a weekly/monthly full backup across the wire.

The process outline by Masood definitely takes some time and effort to setup properly, but it's well worth the effort.

[ptmuldoon]

Ok, I'm working on trying to get a cron job to run on my site, and using a php script/file.

Yet, I can't seem to get the script to run. Is the below correct to run the file? I thought I had everything set correctly, and I have the file permission yet at 755.

I've place the file outside of the public_html path to keep it better secured.

/usr/local/bin/php /home/username/backups/fullbackup.php

All I can think of is not having the correct time set. I'm unsure how to verify the server time, but I believe all of jaguars server on in the U.S. Central Time zone.

[jason]

There are two ways to execute PHP scripts on the command line. The first is to make the script file executable. In this case you need to chmod the file to a minimum of 700 (755 will work, but means that others can still read and execute the code if they can get into your directory). 700 gives you full control but doen't let anyone else do anything with the file. You also need to set up a "shebang" on the first line of the file. If you've ever done any work in Perl, you'll recognize the shebang:

#!/usr/local/bin/php (or #!/usr/local/bin/php5)

This needs to be the very first line of the file, before the opening PHP tag (<?php).

The other way of running the script is to pass the path of the script to the PHP interpreter. In this case you don't need to set permissions (beyond having the file readable to you) or add the shebang. Just list the path of the PHP interpreter followed by the path of your script on the command line:

/usr/local/bin/php /home/username/your/path/here.php

If you are having trouble getting a cron job to work, try running the command on the command line in an SSH session and see if you get any result.

Also note that you may need to use 127.0.0.1 in place of localhost for your MySQL host name.

Hope this helps.

--Jason

[Gwaihir]

I handle this differently.

On my local Linux box, I've got another cron job that tars/gzips the current local copy of my site, and then I use logrotate to keep as many iterations of this full backup as I want. It's more flexible and automatic, and saves the transfer of a weekly/monthly full backup across the wire.

But it also serves a slightly different purpose. If I understand correctly, the cPanel full backup is supposed to include all types of settings made in your account. Stuff like e-mail account names, settings and passwords, ftp accounts, subdomain settings, the works, that isn't included in your sync of files + database dumps.

[wgiese]

But it also serves a slightly different purpose. If I understand correctly, the cPanel full backup is supposed to include all types of settings made in your account. Stuff like e-mail account names, settings and passwords, ftp accounts, subdomain settings, the works, that isn't included in your sync of files + database dumps.

Wasn't aware of that...thanks for the heads-up. That gives cause for concern, beyond my missing these settings.

Can portions of the CPanel backup data be restored individually, i.e. ONLY the settings I'm not grabbing via rsync? If not, that defeats one of the primary benefits of an rsync backup - the ability to quickly restore a site that needs only a partial restore. In my case, uploading a full CPanel site backup to the server would take forever...we've got tons of data due to photo albums.

A better solution than a full CPanel backup (for me, anyway), would be having access to all CPanel data in my home directory, where rsync would pick it up.

Masood - is this possible? Is the extra CPanel backup data stored in files outside of our home directories, that could perhaps be linked or synched to our home directories? Or if it's stored in a db somewhere, would it be possible to have nightly cron job on the Jag server export this data to a home directory?

[JPC-Masood]

cpanel full backup will not work on large accounts (more than a few GBs). CPanel has no solution for it at this time. Some of the settings like email forwarders are available in partial backups from cpanel interface.

The missing things, I can think of, that you do not have access and can only come through full backup are:

(A) subdomains and addon domain names [you can list them down easily somewhere in your documents and re-create them easily]

(B) virtual ftp accounts [again not a big deal as they can be recreated]

(C) mailman mailing lists with their data [no solution for this]

(D) mysql users and privileges [can be easily re-created]

(E) horde (webmail) settings and addressbooks [for this you can install your own horde in your account if you depend on it a lot]

Typically the disaster recovery, without cpanel full backup, will be like this:

1. setup fresh account
2. rsync home directory
3. import mysql dbs and create mysql users from cpanel
4. restore email forwarders and addon/subdomains
5. re-create subdomains/addon domains, and virtual ftp accounts (if any)

[Gwaihir]

CPanel has no solution for it at this time.

[..]

The missing things, I can think of, that you do not have access and can only come through full backup are:

So.. wondering again: can't JagPC offer a way around this?

I've got no actual knowledge of how this WHM/cPanel is built internally, but my outside impression is that its a big bunch of scripts in something like bash or perl (i.e. non compiled, very Linux-geek readable). If that's the case, it can't be that hard for someone like you (Masood) to open up the relevant script, read what it actually does, and custom built a script that takes care of all the same steps, except for the backing up of the space, time and bandwith eating databases and files. In case of a full restore, you'd restore that file first, using cPanel's normal restore, and then refill the account with data from the seperate data backup discussed above.

If that's WAY off, please tell me and I won't bug you about it any further. But if it's kinda like how it works, please do give it some thought.

[JPC-Masood]

Their backup script is closed source. The idea of our own custom backup script is not a bad one but currently we are working on another backup system server-wide and getting ready to deploy it.

[Frank Broughton]

Their backup script is closed source. The idea of our own custom backup script is not a bad one but currently we are working on another backup system server-wide and getting ready to deploy it.

Most excellent.... please do let us know when it will be available for VPS users...