Multiple !@#$% Backup Failures

**Vin DSL** · 05-31-2004, 04:04 AM

I haven't read any complaints about this lately, but it took me several tries to backup my site...[list=1][*] Used cPanel. 130 MB GZ file was corrupt (trailing garbage). Hrm...
[*] Dumped my cache and tried it again with the same result. Oh, well! HTTP transfers suck! I'm surprised it ever works. I've never had cPanel fail twice, but there's a first time for everything!
[*] Tried SSH next. 360 MB straight up. I started the download and went to bed. Sometime during the night, my Internet connection barfed. It was a stupid idea anyway!
[*] This morning I tried WS_FTP Pro. Surprise, surprise! SFTP finally works. That's the first time since I've been a client here. I went ahead and downloaded all 360 MB. This time the connection held.[/list=1]Half a day later, I finally had a good backup, but I decided to do it one more time, the good ol' way. I shelled into my account, TAR'ed my site and downloaded the 125 MB file to my local machine using SFTP. Went smooth as silk, and the file tested good for integrity.

Now, I'm confused...

When did SFTP start working?

Is anyone else having problems using cPanel to backup their site[s]? If so, what do you use?

When I TAR'ed my site, it took a minute or two. When cPanel does it, it's instantaneous. How can this be? Maybe that's why the files are corrupt...

Any thoughts on all this?

**Spathiphyllum** · 05-31-2004, 08:39 AM

Originally posted by Vin DSL
Is anyone else having problems using cPanel to backup their site[s]? If so, what do you use?

When I TAR'ed my site, it took a minute or two. When cPanel does it, it's instantaneous. How can this be?

cPanel working fine for me. I have had WS_FTP botch a batch download from another webhost before because it (WS_FTP) didn't know how to handle softlinks - it just stopped in mid-download, ergo failed download. Note the download was not an archive but straight binary and ascii formatted files.

As far as instantaneous vs delayed, maybe the archive is not created on demand but on a scheduled interval. My other host archives every 15" for their backup - they do not make the tar archive available for my download at any time; consequently, I use WS_FTP. More speculation is that maybe the archive script performs on-demand archiving until a certain amount of data has accumulated; then it resorts to timed, incremental archiving which is then stored for download. In this case your stored volume has exceeded the threshold and a "stale" archive sits their on the server waiting for you to download. Just speculation that might fit the pattern you are observing.

**Vin DSL** · 05-31-2004, 02:26 PM

Interesting theory!

Well, I think I'm going back to the old way of doing things - TAR via SFTP (now that it's working)...

**Vin DSL** · 05-31-2004, 06:52 PM

Originally posted by Spathiphyllum
As far as instantaneous vs delayed, maybe the archive is not created on demand but on a scheduled interval...

You know, the more I think about this, the more sense it makes. I'll bet the file was bad from the get-go, and I simply downloaded the same bad file twice...

**Connie** · 05-31-2004, 07:06 PM

Hate to expose my ignorance again. This is something I have always wandered about.

1: How can the small file you get when you back up from CP replace your website?
I can replace either of my websites but it would take hours. I don't understand
how such a small file could replace an entire website.

2: How do you verify the integrity of the file after it is downloaded?

**Spathiphyllum** · 05-31-2004, 08:48 PM

Originally posted by Vin DSL
I'll bet the file was bad from the get-go, and I simply downloaded the same bad file twice...

Seems more reasonable since there was minimal delay between the two failed archives and the next (much delayed) attempt was successful. Or your corrupted initial archive was stuck in a cache somewhere between the server and your harddrive - seems pretty unlikely though that the latter scenario would happen since the file size was so large.

**jason** · 05-31-2004, 08:53 PM

Connie,

1. The backup file is a .tar.gz file, that is a GZiped Tape ARchive file. TAR files go back to early days of Unix when backups were typically made to tape. Since, on tape, files are written linerally and can't be accessed one at a time the tar program in unix is used for adding or extracting files from tapes.

Tar files are essentially the contents of a tape, but they just aren't written to tape. They are big files that contain the contents of many individual files. There is no compression built in to tar, so it is common to use GZip to compress tar archives to make them take up less disk space.

That is what you get when you download your backups--a compressd archive of all of the files in your account. You can open the archive in programs such as WinZip or WinRAR and access the individual files that they contain.

2. As far as verifying the integrity of the files, generally speaking, if you can open the .tar.gz file and you don't get any errors you can pretty safely assume that the archive is OK. You can, of course, compare the files one at a time to make sure they are intact, but generally if the archive works the files will all be there.

--Jason

**Spathiphyllum** · 05-31-2004, 08:58 PM

Originally posted by clssam - quotes in bold

1: How can the small file you get when you back up from CP replace your website?...I don't understand how such a small file could replace an entire website.
Depends how small you're talking about.

If it's a meg or two, I'm not so sure your .tar or .gz has not been corrupted - unless your content is just miniscule. But tar and gzip are pretty good at compressing things. You'll have to do a Google search to really get down to their efficiency rate.

2: How do you verify the integrity of the file after it is downloaded?
Zipping/unzipping software allows you to check archive integrity. The best windows archiving utility that I know of is WinZip. Amazing little tool. On *nix just use the tar and gzip utilities.

**Vin DSL** · 06-01-2004, 04:44 AM

Originally posted by clssam
Hate to expose my ignorance again. This is something I have always wandered about.

1: How can the small file you get when you back up from CP replace your website?
I can replace either of my websites but it would take hours. I don't understand
how such a small file could replace an entire website.

2: How do you verify the integrity of the file after it is downloaded?

I think 'they' answered your first question... If not, try this:

http://computer.howstuffworks.com/fi....htm/printable

Answer 2: Every archive utility I've ever used has a test feature. All you have to do is click 'test' and the program will check the file for integrity.

**Vin DSL** · 06-01-2004, 04:46 AM

Originally posted by Spathiphyllum
Seems more reasonable since there was minimal delay between the two failed archives and the next (much delayed) attempt was successful...

They both failed at exactly the same spot too - 92% complete. That's too much of a coincidence for me...

**Vin DSL** · 06-01-2004, 04:51 AM

Originally posted by jason
The backup file is a .tar.gz file, that is a GZiped Tape ARchive file...There is no compression built in to tar, so it is common to use GZip to compress tar archives to make them take up less disk space...

LoL! I knew I forgot something...

I did it again using the GZip filter and shrunk the backup file to 59 MB. I think I'll try it with BZip2 next.

**Vin DSL** · 06-01-2004, 05:59 AM

Originally posted by Vin DSL
...I did it again using the GZip filter and shrunk the backup file to 59 MB. I think I'll try it with BZip2 next.

Forget that! The BZip2 was 1 MB bigger - 60 MB instead of 59 MB.

Here's another weird one for you! I went back and checked a backup I did on 5-8-2004 using cPanel. It was bad too! Trailing garbage at exactly the same spot. So, I went back even further, into April, but they were okay.

It looks like all the cPanel backups of my site have been corrupt for the last month. However, the sql db backups are fine. Hrm...

So, what have I learned?

I guess the cPanel backups cannot be counted on. They are stale and corrupt, or so it would appear. I think the best procedure is to use cPanel for the MySQL backups, but make my own tarballs for the rest of the site.

Sound like a winner to you?

**Vin DSL** · 06-01-2004, 06:18 AM

Originally posted by Vin DSL
[list=1][*] Used cPanel. 130 MB GZ file was corrupt (trailing garbage). Hrm...[/list=1]

Here's yet another oddity!

I was rereading the thread, and I thought I might have gotten the size wrong. How can the cPanel backup be twice the size of MY backup. So, I checked. The cPanel tarball was 128,634,880 bytes. How can that be?

Evidently, when cPanel backs up your site, it TAR's the files twice - once in 'public_html' and once again under the 'www' symlink.

Sounds like another good reason to 'roll your own', no?

**Galen** · 06-01-2004, 02:45 PM

Vin's talking to himself again....

**Oldfrog** · 06-01-2004, 02:56 PM

I do that on occasion when I need immediate intelligent conversation. If the need is not immediate then I talk to the dog. If the need is neither immediate nor intelligent I call my ex-wife.