Folders or files?

[jabbaonthedais]

I have a product list with hundreds of products, and thousands of images (around 2,500) and the list is growing. I had all the images in one folder, numbered 1.jpg to 2500.jpg. When I open the folder in any FTP client it only shows me 1998 files (I guess the max) so I have to delete files to see more. I imagine this is really slow. Would it be better to break them down into sub folders?

Like this:
/products/1/1.jpg - 1000.jpg
and
/products/2/1001.jpg - 2000.jpg

I'm looking for the fastest/best option for server load. I need to pull these images often on my site, but not all at the same time (obviously). Anyone know the best option?

[jason]

I don't think it will make much of a difference on server load if a folder has 10 files or 10,000, but a little bit of file organization will certainly make things easier for you to manage. If the benefit of easier file management outweights the cost of updating your site to handle the new paths then I'd probably update. However, if it was going to take me several hours to change my scripts or html then I probabaly wouldn't.

That's my 2 cents...

--Jason

[Ron]

Wasn't there a server setting someone mentioned in another thread about how many files are listed by default? Or was that an Apache directory listing? I forget.

Perhaps you should a ticket to support and ask them if there's a way to see more than the 2000 files.

[Vin DSL]

Source: http://download.pureftpd.org/pub/pure-ftpd/doc/README

-L <max files>:<max depth>: To avoid stupid denial-of-service attacks (or just CPU hogs), Pure-FTPd never displays more than 2000 files in response to an 'ls' command. Also, a recursive 'ls' (-R) never goes further than 5 subdirectories. You can increase/decrease those limits with the '-L' option.

[Vin DSL]

I don't think it will make much of a difference on server load if a folder has 10 files or 10,000...

Correct!

I just pruned a folder with 220,000+ files in it (average size 9k), down to 180,000+ files. Didn't make a bit of difference, except it went from 1.9GB -> 1.3GB.

[Ron]

There are other considerations one might think about.

Having single larger directories seems to be better than have many sub-directories when doing incremental backups with WinSCP; the most time consuming activity of my daily backup using it is waiting while it looks through each directory, one by one.

Having extremely large directories as has been said probably wont noticeably affect server load or performance anymore... computers do so much more than look at directory files nowadays. Back in the old days, having huge directories split into multiple pieces due to file size limitations was an issue, but I haven't heard much about that lately..

Good luck

[Gwaihir]

Wasn't it dos FAT that stopped at 256 files per folder? (FAT8?)

Unless you application itself has a habit of frequently throwing that big folder listing across the net or something (which I doubt), it looks like the only handicap you run into is that listing of folders via FTP. Support might be willing to up that 2000 limit for you (to 3000 or 4000 perhaps?) as it's somewhat arbitrary anyway. Once you're about to run out of that too, look for another way to manage that folder, like WinSCP.

BTW: 2000 = 1998 images + . + ..

[jabbaonthedais]

Thanks guys! So basically the only disadvantage of having thousands of files in one directory (vs several) would be viewing with an FTP client? When I pull an image on my site from that directory, it should just load that one image and not have to "view" the entire contents of the directory. I wasn't sure if thats how it was or not.

[Ron]

OPhh no, it has to read the directory, but it's all probably done with a single disk fetch, and the amount of time needed to look through the directory in memory (or on disk) is trivial nowadays.

Good luck!

[Vin DSL]

Once you're about to run out of that too, look for another way to manage that folder, like WinSCP...

Um...

WinSCP will only display 2000 files, too. Yes, WinSCP is built on top of PuTTY, but, the fact of the matter is, WinSCP is a Windows SFTP (Secure FTP) client with SCP (Secure Copy) capabilities.

Heh! WinSCP is soooo misunderstood! I guess that's why I relate to it...

I maintain these huge folders from CLI -- don't know of any other way!

[Gwaihir]

WinSCP is a Windows SFTP (Secure FTP) client

Um too..

That's a good link, to quote from it:

FTP over SSH is sometimes referred to as secure FTP; this should not be confused with other methods of securing FTP, such as with SSL/TLS (FTPS). Other methods of transferring files using SSH which are not related to FTP include SFTP or SCP; in both of these, the entire conversation (credentials and data) is always protected by the SSH protocol.

WinSCP doesn't use FTP over SSH, nor FTPS, but is built around those last two protocols mentioned. These, as the wikipedia says, are not related to FTP and thus it is in no way dependend on the Pure-FTPd on the server and its settings / limits. WinSCP also has nice caching features to remember the content of a folder. So all in all I'm a bit surprised to hear it too has a 2000 files per folder limit. I'm not sure where that would come from. I looked around for a bit but didn't find any setting relating to it. Or would that be server side?

[Vin DSL]

WinSCP doesn't use FTP over SSH...

Wrong!

So all in all I'm a bit surprised to hear it too has a 2000 files per folder limit. I'm not sure where that would come from. I looked around for a bit but didn't find any setting relating to it. Or would that be server side?

The 2000 file limit is a server side setting, when 'they' compiled PureFTP-D, as explained above!

Okay, let's try it this way...

WinSCP uses FTP over SSH...

Source: http://winscp.net/eng/index.php

What is WinSCP?

WinSCP is an open source SFTP client for Windows. Its main function is the secure file transfer between a local and a remote computer. Beyond this, WinSCP offers basic file manager functionality. It uses Secure Shell (SSH) and supports, in addition to Secure FTP, also legacy SCP protocol.

Who else uses Secure FTP?

Source: http://www.intranetjournal.com/artic...08_14_02b.html
Secure_FTP.png

What port does WinSCP use?
Secure_FTP2.png

What is port 22?

Source: http://en.wikipedia.org/wiki/List_of...P_port_numbers

22/TCP,UDP SSH (Secure Shell) - used for secure logins, file transfers (scp, sftp) and port forwarding

Make more sense now?

[Vin DSL]

Here's another one for you, if you still aren't convinced...

Source: http://www.stdnet.com/moveitdmz/pipeda.htm

Protecting Customer and Employee Privacy

The Canadian Personal Information Protection and Electronic Document Act (PIPEDA) requires companies to protect the privacy of their customers and employees when electronically exchanging personal information with third-parties, and MOVEit® DMZ by Standard Networks can help them do that.

MOVEit DMZ secure data transfer and storage servers are being used by the following Canadian organizations to safely exchange sensitive and confidential information over public networks, including the Internet.

Barrick Gold
Psion Teklogix
Great-West Life
Talisman Energy
Husky IMS
The Marketing Store
IQON Financial
Veteran's Affairs
Ontario Lottery
Workstream...

MOVEit DMZ helps them achieve and demonstrate PIPEDA compliance with the following capabilities.

* Authentication. Access to MOVEit requires a valid username and password or public key submitted over an encrypted link (IP address can be used as an optional third factor).
* Authorization. Access to stored files and messages requires prior user authorization on a per folder or group basis.
* Transfer Encryption. All file and message transfers are protected by Canadian Security Establishment (CSE) approved 128-bit SSL encryption.
* Storage Encryption. All files and messages are automatically stored using MOVEit's CSE approved 256-bit AES encryption.
* Audit Trail. A comprehensive record on each user, file and message is automatically created that can be used to prove who sent and received a file and that it was not altered in transit.
* Bilingual. Users can select MOVEit's English or French interface.

In addition to its PIPEDA related capabilities, MOVEit DMZ provides operational flexibility. All of the following can be used to securely exchange files with MOVEit DMZ on a manual or automated basis.

* Web Browsers. Including Firefox, Internet Explorer, Navigator, Netscape, Opera, and Safari (no applets, plugins, etc. required).
* FTP over SSL Clients. Including CuteFTP Pro, LexiCom, MOVEit Freely, TDAccess, WS_FTP Pro, and z/OS Secure Sockets FTP.
* FTP over SSH Clients. Including FileZilla, OpenSSH for sftp, PuTTY PSFTP, SSH Secure Shell FTP, Tectia, and WinSCP.
* MOVEit HTTPS Clients. Including MOVEit API Java, MOVEit API Windows, MOVEit Central, and MOVEit EZ.

[Gwaihir]

Eh.. Vin; there was a link in there directly to the relevant WinSCP docs which are pretty clear on the matter.

WinSCP doesn't use FTP over SSH, nor FTPS, but is built around those last two protocols mentioned.

I know it's darn confusing that the terms secure FTP and SFTP are used for each of three quite different methods (FTP over SSH, FTPS, and SSH File Transfer Protocol). I'm surprised though that you mix in SSH as well like it is all the same. You do know what that is, right?

FTP over SSH means that you use an SSH tunnel for regular FTP traffic. So of course you see the SSH port show up in the connection, but you're just looking at the outside of the tunnel. I regularly tunnel HTTP over SSH (to get to an intranet). Per your logic, is it then IE's built in FTP client that shows me my web pages?

[Vin DSL]

Heh! I know better than to argue with knot-headed krauts. Think what you want. It really doesn't make any difference.

I was just trying to explain why WinSCP has the same 2000 file limitation...