Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Apache::AutoIndex - Perl replacment for mod_autoindex

Reply
Thread Tools

Apache::AutoIndex - Perl replacment for mod_autoindex

 
 
Petyr David
Guest
Posts: n/a
 
      04-24-2008
anyone have an experience using this?

The real question:

does using this speed the creation of a directory index in Apache
significantly? We have directories with thousands of small files.

Thanks!
 
Reply With Quote
 
 
 
 
J. Gleixner
Guest
Posts: n/a
 
      04-24-2008
Petyr David wrote:
> anyone have an experience using this?
>
> The real question:
>
> does using this speed the creation of a directory index in Apache
> significantly? We have directories with thousands of small files.


It's more likely that it'll be slower because mod_autoindex is
written in C and compiled into the Apache daemon, you're not
going to get much faster than that.

Possibly you could list 500 at a time, or something, which
would be faster, however having thousands of files in a directory
isn't typically a good design.
 
Reply With Quote
 
 
 
 
Petyr David
Guest
Posts: n/a
 
      04-25-2008
On Apr 24, 5:15*pm, "J. Gleixner" <(E-Mail Removed)>
wrote:
> Petyr David wrote:
> > anyone have an experience using this?

>
> > The real question:

>
> > does using this speed the creation of a *directory index *in Apache
> > significantly? We have directories with thousands of small files.

>
> It's more likely that it'll be slower because mod_autoindex is
> written in C and compiled into the Apache daemon, you're not
> going to get much faster than that.
>
> Possibly you could list 500 at a time, or something, which
> would be faster, however having thousands of files in a directory
> isn't typically a good design.


agreed, but the nature of the data forces us to store the files in
this fashion so there's some sense of meaningfulness: every file is
named and then has a sequential number appended to it - there 64K
potential numbers. we might have to start something like breaking it
into 1000 files/directory.

either way - thanks for your opinion
 
Reply With Quote
 
smallpond
Guest
Posts: n/a
 
      04-25-2008
On Apr 24, 8:48 pm, Petyr David <(E-Mail Removed)> wrote:
> On Apr 24, 5:15 pm, "J. Gleixner" <(E-Mail Removed)>
> wrote:
>
> > Petyr David wrote:
> > > anyone have an experience using this?

>
> > > The real question:

>
> > > does using this speed the creation of a directory index in Apache
> > > significantly? We have directories with thousands of small files.

>
> > It's more likely that it'll be slower because mod_autoindex is
> > written in C and compiled into the Apache daemon, you're not
> > going to get much faster than that.

>
> > Possibly you could list 500 at a time, or something, which
> > would be faster, however having thousands of files in a directory
> > isn't typically a good design.

>
> agreed, but the nature of the data forces us to store the files in
> this fashion so there's some sense of meaningfulness: every file is
> named and then has a sequential number appended to it - there 64K
> potential numbers. we might have to start something like breaking it
> into 1000 files/directory.
>
> either way - thanks for your opinion



Why not use a real database instead of making one out of a
filesystem? Directory searches are slow, sequential string
compares. Database lookups use fast hash techniques.
 
Reply With Quote
 
J. Gleixner
Guest
Posts: n/a
 
      04-25-2008
Petyr David wrote:
> On Apr 24, 5:15 pm, "J. Gleixner" <(E-Mail Removed)>
> wrote:
>> Petyr David wrote:
>>> anyone have an experience using this?
>>> The real question:
>>> does using this speed the creation of a directory index in Apache
>>> significantly? We have directories with thousands of small files.

>> It's more likely that it'll be slower because mod_autoindex is
>> written in C and compiled into the Apache daemon, you're not
>> going to get much faster than that.
>>
>> Possibly you could list 500 at a time, or something, which
>> would be faster, however having thousands of files in a directory
>> isn't typically a good design.

>
> agreed, but the nature of the data forces us to store the files in
> this fashion so there's some sense of meaningfulness: every file is
> named and then has a sequential number appended to it - there 64K
> potential numbers. we might have to start something like breaking it
> into 1000 files/directory.
>
> either way - thanks for your opinion


You could also generate one or more static HTML files, once a day or
whenever new files are added, which would display the filename
and contain a link to the file. Heck, you could probably display
5-10 per line, too. That way you'd avoid having to stat
the thousands of files every time the directory is accessed.
 
Reply With Quote
 
xhoster@gmail.com
Guest
Posts: n/a
 
      04-25-2008
smallpond <(E-Mail Removed)> wrote:
> On Apr 24, 8:48 pm, Petyr David <(E-Mail Removed)> wrote:
> > On Apr 24, 5:15 pm, "J. Gleixner" <(E-Mail Removed)>
> > wrote:
> >
> > > Petyr David wrote:
> > > > anyone have an experience using this?

> >
> > > > The real question:

> >
> > > > does using this speed the creation of a directory index in Apache
> > > > significantly? We have directories with thousands of small files.

> >
> > > It's more likely that it'll be slower because mod_autoindex is
> > > written in C and compiled into the Apache daemon, you're not
> > > going to get much faster than that.

> >
> > > Possibly you could list 500 at a time, or something, which
> > > would be faster, however having thousands of files in a directory
> > > isn't typically a good design.

> >
> > agreed, but the nature of the data forces us to store the files in
> > this fashion so there's some sense of meaningfulness: every file is
> > named and then has a sequential number appended to it - there 64K
> > potential numbers. we might have to start something like breaking it
> > into 1000 files/directory.
> >
> > either way - thanks for your opinion

>
> Why not use a real database instead of making one out of a
> filesystem?


Why not use a file system rather than making one out of a database?
Admitted, appending a sequential number to the file name is rather strange,
but that still doesn't mean that he really wants a database rather than a
file system.

> Directory searches are slow, sequential string
> compares.


Not if you already know the exact name of the file. At least, not on any
reasonable file system. They use trees or hashes or something to find the
file quickly.

> Database lookups use fast hash techniques.


Same as reasonable file systems, given an exact name. Of course, if you
are doing globs, or just extracting all entries, then neither file systems
nor databases will use fast hash techniques.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Intel Pro Set software replacment? Ty Wireless Networking 3 01-18-2007 06:06 AM
drop-in replacment for function pointers Oliver Kowalke C++ 3 02-05-2005 08:28 PM
Re: HTML Textarea Replacment hyweljenkins@hotmail.com HTML 0 11-05-2004 03:38 PM
Re: replacment for OE Gordon Burgess-Parker Computer Support 0 08-25-2003 07:36 PM
Re: replacment for OE Conor Computer Support 0 08-25-2003 06:05 AM



Advertisments