3:14 out of 3:27

[Ron]

Yup.

That's when mercury went down, after 3 hours and 14 minutes of a three hour and 27 minute upload (according to winscp).

Anybody know a way to fetch a humongous file through the http protocol directly to one of our servers?

Specifically, I am trying to get a 300MB .gz file that is only available through http onto mercury. What I did was download it to my machine via IE (which exploded the file from 300MB to 1.7GB) then I zipped it back up, and was in the process of uploading when mercury went down.

Any way to just fetch via http through a comand line interface so I can do it at server to server speed?

Thanks in advance...

[jason]

The easiest way would probably be to use lynx via SSH.

lynx http://www.whatever.com/whatever

--Jason

[Vin DSL]

When I do backups, I TAR my site using WinSCP. Then, I download it via DS3 at work. Finally, I save the backup file to my JumpDrive.

http://www.mobiledriven.com/25juspusb20d.html

Did it yesterday - 136MB - no problem!

The curious thing is, while this connection does 7mb up/down (I've tested it), 'my' server at JagPC limits the transfers to around 240kb. Been that way for a long time...

[Ron]

Originally posted by jason
The easiest way would probably be to use lynx via SSH.

lynx http://www.whatever.com/whatever

--Jason

Looks like exactly what I was looking for, Jason, but it complains...

Configuration file (null) is not available.

Any idea where to point me for a config file, or what should be in it?
Thanks again

[jason]

A less interactive mehod is to use curl to save the file instead of lynx. With curl, however, you'll need to specify the exact URL of the file you're downloading. Use -o to direct output to a file instead of dumping it to the STDOUT.

curl http://whatever.com/whatever -o /home/ron's-site/whatever

Good luck.

--Jason

[Ron]

It took 8:51 using curl.... that's 8 minutes and 51 seconds.

Ran an average of 545K/second
(My download runs about 200-250K but my upload was ~25K)

You the man, Jason!
Saved me hours. Thanks!

[Spathiphyllum]

Originally posted by jason
A less interactive mehod is to use curl to save the file instead of lynx...

I haven't needed to use curl before but I'll have to store this tidbit of knowledge into my mental archive. Seems like one powerful, little utility.

[gerilya]

An easier alternative to curl would be wget
Although not as powerfull as curl, it can handle file download just fine and doesn't require any parameters beside URL.

[Spathiphyllum]

Originally posted by gerilya
An easier alternative to curl would be wget

Yes, I like wget too. But easier alternative? Who's looking for that? Remember, I use Perl.

[jason]

Originally posted by gerilya
An easier alternative to curl would be wget
Although not as powerfull as curl, it can handle file download just fine and doesn't require any parameters beside URL.

Yes, I forgot about wget.

For those of you interested in more about curl, check out http://curl.haxx.se/

--Jason

[gerilya]

Originally posted by Spathiphyllum
Yes, I like wget too. But easier alternative? Who's looking for that? Remember, I use Perl.

Here is something for you then:

use LWP::UserAgent;