Re: Large Directory Problem

This is a problem that has been around since the early days of DOS. The solution is to limit the number of files that reside in any given directory.

Since you are storing the metadata in Oracle, the actual file name is irrelevant to the end user. If you create a hierarchy that limits the number of file or directory entries at any level to a new hundred entries then the performance should be very good.

I've implemented large scale document archival systems that maintained over 1Tb of documents and images that was dispersed between RAID and CD-ROM jukeboxes. The number of documents in the system exceeded 50 million, and file system performance was never an issue.

The original design when I was hired onto the original version of the project
had several hundred thousand documents maintained in the primary cache directory, which resulted in extremely poor performance of the file system when performing administrative tasks. By increasing the depth of the directory hierarchy and maintaining a limited number of entries across a large number of directories, the performance problem was eliminated.

As to storing BLOB data in a DBMS, that is for neophytes (or very small systems) who don't understand the issues of scalability. By storing the metadata in the database and the binary data in independent files, you have the flexibility to expand the file system storage to a less expensive optical storage solution should the need arise. For an example of a system that has been implemented in this fashion, check out Microsoft's TerraServer on the web. (Note: I would have used optical jukeboxes rather than build up a 1Tb drive array, but then MS has plenty of money and most real-world companies too cost conscious to take this approach.)

Michael D. Long
http://www.geocities.com/siliconvalley/heights/9478/

Randall <aslan-zzz_at_pcmagic.net> wrote in article <36150F83.F7088A06_at_pcmagic.net>...
> Hi!
>
> I'm collecting .jpg images in a single directory. I am going to be
>
> At this time I have well over a thousand images in the directory. When
> I open the directory in the NT Explorer or a window I get the flashlight
> waving and it takes several seconds to retrieve the directory.
>
> Is there anything I can do to speed NT's access to this directory? When
> I write my app is there anything I need to be aware of that will
> help/hurt performance in retrieving these images. Some people in my
> organization feel that saving the images in Oracle BLOBs (Binary Long
> OBjects) will server us better. Is that true?
Received on Sun Oct 04 1998 - 00:00:00 CDT