1 time full site backup

[titanspot.com]

Ok guys, I have 2 sites, an IPB forum and a phpbb forum, running on an SDX plan. I'm using 11-12 gigs of space currently (lots of attachments over the past 2 years). I'm performing nightly DB dump backups via cron for all of my databases. Right now, these backups are stored in the same account as the sites. This won't do me a whole lot of good in case of a catastrophic failure. I don't want to rely on Jag's backups.

I want to do 2 things. First, I want to do a 1 time full account backup and download it to my PC. The site is too big to do a CPanel backup (it errors out). I thought I could use Vin's method of accessing the account via shell/puTTy and issuing the following command: tar -cvpjf archive.tbz /home/<account name> . However, it errored out as well. I really don't know how long it ran...I went to sleep. The error it gave was tar: Error exit delayed from previous errors . That doesn't tell me much. I really only need to get this full backup once for now. Although, in the future I'd like to set up incremental backups (rsync maybe?) but I'm just unfamiliar with doing that.

Second, and this is very important, as simple as it may seem to some of you, I don't know how to go about setting up a way to download my nightly DB backups to my home machine. I'm running Windows XP at home and I'm sure there is probably some way I can set this up. I'm just drawing a blank.

Any ideas on either would be appreciated. I really don't want to pay extra for backup space right now when I'm sure there is a way I can set something up between my local PC and my hosting account. I just don't know how to go about it.

[titanspot.com]

You know what would be ideal? Something like robocopy (windows script) that would run from my local pc and nightly sync up my local copy with the copy of the site on the server. It would only copy changed and/or new files. I don't suppose something this nice exists that can be setup on a home Windows pc and sync with a linux SDX account......????.......

[Doug Nelson]

Can't help you on the first item, but as I see it you have two options for the second:

Run an FTP server on your home machine (requires an open port in your firewall, and problematic if you have a dynamic IP, but there are workarounds for that)
Schedule a nightly FTP download (I use FTP Voyager which has a builtin scheduler, but most FTP apps should have a way to do this)

Of these two options (and there are probably many more), I'd go with option two.

[titanspot.com]

I could probably do that....I do have a dynamic IP right now but could probably get a static one.....but....the command to tar and bzip my site errored out. Without being able to actually create the backup, I have nothing to download.

[Doug Nelson]

I had the same problem with backing up my forums. It turned out a lot of the backup was my search index, which doesn't really need to be backed up. I removed it from the backup and now I can backup every night without timing out.

Are your attachments in your db? If so you can move them to a folder and that will also help.

[wuurp]

On the tar problem, maybe try a smaller set of your data, like /home/<account name>/somesubdirectory and see if that works. If that does, then maybe tarring and bziping 11-12 gigs is using too much resources and it's getting killed.

If you're thinking of using rsync anyway in the future, I'd suggest doing it that way. That's how I back up all my sites. One benefit is that if it fails at some point, like your internet goes down, or computer crashes, you just start it again later, and it basically picks up where it left off. You also have a backup where you can easily browse through and find individual files if you need to restore them. Save that full back off somewhere, then periodically run rsync again to keep a current full backup.

It would also work fine for daily backups of your database backups with a scheduled event.

However, I have no experience with rsync on windows, but it works great on linux.

[titanspot.com]

I just wish there was some sort of program that worked like Robocopy....only between a *nix server and a windows pc! Wishful thinking huh?!? I use robocopy at work to sync certain directories on one server (windows) to those corresponding directories on the other(windows). After the initial sync, it only copies over new/changed files from those directories to the other server. It does this once nightly.

From what I understand, rsync doesn't work this way. My understanding is that it creates a tarballed copy of only the changes made each day. While I guess I could figure out a way to make that work, the whole *nix to windows thing still has me stumped. Has anyone seen a howto or guide written to specifically do this? I couldn't be the first person who has wanted to do this....

Plus..all of this is from an SDX account.....

[Ron]

WinSCP has a sync feature and also has scripting abilities.

I use it daily to backup to my PC; it takes about 6 minutes to scan all of my 50,000+ files in 800+ directories consuming 11GB, and downloads the 100MB in db dumps and changed files in just a couple of minutes more (but I'm on a FiOS fiber optic connection.)

[Vin DSL]

The site is too big to do a CPanel backup (it errors out). I thought I could use Vin's method of accessing the account via shell/puTTy and issuing the following command: tar -cvpjf archive.tbz /home/<account name> . However, it errored out as well...

Hrm...

I'm on SDX also. Let's do a few backups from a PuTTY window and see if they fail...

tar -cvpjf Lenon.com_`date +%m-%d-%y`_Backup.tbz /home/my-account --exclude=/home/my-account/*.tbz --exclude=/home/my-account/.trash --exclude=/home/my-account/etc-bkp --exclude=/home/my-account/mail-bkp --exclude=/home/my-account/public_html_old --exclude=/home/my-account/g2data/cache --exclude=/home/my-account/public_html/e107/e107_files/cache --exclude=/home/my-account/public_html/info/wp-content/cache --exclude=/home/my-account/public_html/joomla/cache --exclude=/home/my-account/public_html/modules/ODP/cache --exclude=/home/my-account/public_html/pilot/modules/ODP/cache --exclude=/home/my-account/tmp/analog/cache --exclude=/home/my-account/tmp/webalizer/dns_cache.db --exclude=/home/my-account/www

Okay, I dumped all my MySQL dbs to the SDX server, then did 3 complete backups (except a few caches and some old backup folders) - took about 12 minutes - final size 7071 MB uncompressed --> 403 MB compressed (bz2) including the dumps...

Now I'll download the backup to my local PC and check it for integrity...

Tick, tick, tick, tick, tick, tick, tick, tick....

Okay, an hour later, the backup is on my machine and PKZIP reports that it's error-free!

Soooo, you might want to check out the above command, use it as a template, and adapt it to your situation...

BTW, I'm NOT a fan of incremental backups. If I was you, I'd stick with complete backups!

Also, I use Robocopy all the time. It works great across a LAN, but forget about using it across the web...

[mj_pilot]

I have encountered the same problems Titan. The only copy I keep on my local computer is downloaded once every now and again via ftp as a new backup.

I have recently asked JPC if it was possible to purchase a gigadeal account (located on another machine) and have all my data from my SDX account backed up automatically to the gigadeal account, they said YES.

I think this is a great idea, as you are not wasting your bandwidth on backups and your main site is always backed up including files, databases and settings. All this while you are sleeping

[the_ancient]

WinSCP has a sync feature and also has scripting abilities.

I use it daily to backup to my PC; it takes about 6 minutes to scan all of my 50,000+ files in 800+ directories consuming 11GB, and downloads the 100MB in db dumps and changed files in just a couple of minutes more (but I'm on a FiOS fiber optic connection.)

This is what I use, although I may have to stop now that Ron uses it



//FiOS Sucks

[wuurp]

I just wish there was some sort of program that worked like Robocopy....only between a *nix server and a windows pc! Wishful thinking huh?!? I use robocopy at work to sync certain directories on one server (windows) to those corresponding directories on the other(windows). After the initial sync, it only copies over new/changed files from those directories to the other server. It does this once nightly.

From what I understand, rsync doesn't work this way. My understanding is that it creates a tarballed copy of only the changes made each day. While I guess I could figure out a way to make that work, the whole *nix to windows thing still has me stumped. Has anyone seen a howto or guide written to specifically do this? I couldn't be the first person who has wanted to do this....

No, rsync does just what you say robocopy does, at least the way I use it. Just new files are copied. On changed files, only the differences are copied. So if you have a 1GB log file, it only transfers the changes at the end, not the whole file again.

[Vin DSL]

This is what I use, although I may have to stop now that Ron uses it

Yeah, to each his own...

I cannot mirror my site to local. I mean, I could, but I have around a half-million files in one of my directories, at any given time, so we're talking a day or two to mirror my account across a web connection using WinSCP or whatever. TAR and transfer works a LOT better for me!

I dunno, maybe I could script it out, but why - you know?

[Vin DSL]

BTW, let me tell you why I don't like differential/selective/incremental backups...

Let's use an overly simplistic example:

Let's say you do a complete backup every week and incremental backups thereafter.

On Day-1 your backup will contain everything you need to restore your site[s].

Day-2 will only contain things that have changed...

Day-3 will contain more changes, but nothing from Day-1 and Day-2 that have NOT changed.

Dittos for Day-4, Day-5, Day-6, yada, yada...

Now let's say your account crashes on Day-7. Hello!?!?! You've got 6 restores to do and most likely you'll still be missing whtever changed on Day-7 unless your doing backups every few minutes...

Plus, finding specific files amid a collection of incremental backups can be challenging. That's why JagPC charges like $100/hr or $15 per file (or whatever the charge now) for doing restores...

Incremental backups create an unacceptable layer of complexity, IMHO. The last thing you want to do, when your site crashes, is attempt to put Humpty Dumpty back together again from multiple backups!

Mirroring a site is probably a good alternative to complete backups, if you archive them daily/weekly. Hell, I still have every backup I've done since the last century on my HD! Can you imagine the mess I'd have, if I was doing incremental backups?

[Ron]

Vin,

With WinSCP I'd recommend tarballing and transfer and untarballing for an initial copy, then syncing... the local directory is kept current with the original -- any changed files (and only changed files) are copied across the network. That includes DB dumps of course.

I wind up transferring about 150MB a day for any new/changed files and my daily compressed DB dumps.

I keep a couple of tarballs too... for a fast restore I'd upload a single tarball with the essential files (program/html files) and I'd manually upload and restore the latest DB dumps, then allow WinSCP to run for a few days restoring all of the gigs of pics.

@mj_pilot; I am not sure how backing up to a gigadeal account saves bandwidth? FTP traffic counts towards the limit, doesn't it?