Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > the fastest way to create a directory

Reply
Thread Tools

the fastest way to create a directory

 
 
George Mpouras
Guest
Posts: n/a
 
      07-15-2013

Create a directory with all upper directories if missing.
it uses the minimum possible disk access and checks.




Mkdir_recursive('/some/dir/d1/d2') or die;

sub Mkdir_recursive
{
return 1 if $_[0] eq '' || -d $_[0];
Mkdir_recursive( $_[0] =~/^(.*?)[\\\/][^\\\/]+$/ ) || return undef;
mkdir $_[0] || return undef
}

 
Reply With Quote
 
 
 
 
George Mpouras
Guest
Posts: n/a
 
      07-15-2013


Ο "Henry Law" *γραψε στο μήνυμα
news:(E-Mail Removed) ...

On 15/07/13 12:09, George Mpouras wrote:
> Create a directory with all upper directories if missing.


Use File:ath.


no it is slow

 
Reply With Quote
 
 
 
 
Charlton Wilbur
Guest
Posts: n/a
 
      07-15-2013
>>>>> "GM" == George Mpouras
>>>>> <(E-Mail Removed) m.com.nospam>
>>>>> writes:


GM> Ο "Henry Law" *γραψε στο μήνυμα
GM> news:(E-Mail Removed) ...

GM> On 15/07/13 12:09, George Mpouras wrote:
>> Create a directory with all upper directories if missing.


GM> Use File:ath.

GM> no it is slow

If you create so many directories that the execution time of File:ath
is greater than the debugging time of roll-your-own code, Perl is
probably the wrong langauge for the task.

Charlton


--
Charlton Wilbur
http://www.velocityreviews.com/forums/(E-Mail Removed)
 
Reply With Quote
 
Tim McDaniel
Guest
Posts: n/a
 
      07-15-2013
In article <ks0l7j$9j4$(E-Mail Removed)>,
George Mpouras <(E-Mail Removed) m> wrote:
>Create a directory with all upper directories if missing.


On Linux-like systems,
system("mkdir", "-p", $dir);
Unless you're creating millions of directories per day, doing it with
a known way is far faster and far more reliable than trying to code
your own.

--
Tim McDaniel, (E-Mail Removed)
 
Reply With Quote
 
George Mpouras
Guest
Posts: n/a
 
      07-15-2013
yes there are about thousands dirs. try to undestand the code. looks simple
but it is not



 
Reply With Quote
 
George Mpouras
Guest
Posts: n/a
 
      07-15-2013
I have found that usingn Perl with clever code you can be faster than the
usual c
of cource if you write the same code with C it will be faster but then, the
deadlines will pass !

 
Reply With Quote
 
Peter J. Holzer
Guest
Posts: n/a
 
      07-15-2013
On 2013-07-15 15:45, George Mpouras <(E-Mail Removed) m.com.nospam> wrote:
> Ο "Henry Law" *γραψε στο μήνυμα
> news:(E-Mail Removed) ...
>
> On 15/07/13 12:09, George Mpouras wrote:
>> Create a directory with all upper directories if missing.

>
> Use File:ath.
>
>
> no it is slow


On my system it is exactly as fast as your version, creating 2*70644
directories.

Here is the test program:


#!/usr/bin/perl
use warnings;
use strict;
use File:ath qw(make_path);
use Time::HiRes qw(time);

$| = 1;

my $n = $ARGV[0];
{
my $t0 = time;
for my $i (0 .. $n) {
for my $j (0 .. $n) {
for my $k (0 .. $n) {
make_path("t1/$i/$j/$k");
}
}
printf("%3d %9.6f\n", $i, time - $t0);
}
}
{
my $t0 = time;
for my $i (0 .. $n) {
for my $j (0 .. $n) {
for my $k (0 .. $n) {
make_path("t2/$k/$j/$i");
}
}
printf("%3d %9.6f\n", $i, time - $t0);
}
}
__END__

(second test program uses your Mkdir_recursive instead of make_path)

With $n = 40, The first loop takes about 20 seconds, the second about 24
seconds for both programs. As expected, they are completely disk-bound.
strace shows that they are also performing essentially the same system
calls.

hp

--
_ | Peter J. Holzer | Fluch der elektronischen Textverarbeitung:
|_|_) | Sysadmin WSR | Man feilt solange an seinen Text um, bis
| | | (E-Mail Removed) | die Satzbestandteile des Satzes nicht mehr
__/ | http://www.hjp.at/ | zusammenpaßt. -- Ralph Babel
 
Reply With Quote
 
Peter J. Holzer
Guest
Posts: n/a
 
      07-15-2013
On 2013-07-15 21:04, George Mpouras <(E-Mail Removed) m.com.nospam> wrote:
> yes there are about thousands dirs. try to undestand the code. looks
> simple but it is not


What? It's trivial. It also has a bug: It doesn't perform silently under
use warnings. Ok, so maybe it isn't trivial after all ...

hp

--
_ | Peter J. Holzer | Fluch der elektronischen Textverarbeitung:
|_|_) | Sysadmin WSR | Man feilt solange an seinen Text um, bis
| | | (E-Mail Removed) | die Satzbestandteile des Satzes nicht mehr
__/ | http://www.hjp.at/ | zusammenpat. -- Ralph Babel
 
Reply With Quote
 
Tim McDaniel
Guest
Posts: n/a
 
      07-15-2013
In article <ks1o4c$5jj$(E-Mail Removed)>,
George Mpouras <(E-Mail Removed) m.com.nospam> wrote:
>yes there are about thousands dirs. try to undestand the code. looks simple
>but it is not


It *IS* simple. Microsecond efficiency is highly unlikely to be a
problem for you, so you should, in almost all cases, so with the
reliable solution.

--
Tim McDaniel, (E-Mail Removed)
 
Reply With Quote
 
Peter J. Holzer
Guest
Posts: n/a
 
      07-16-2013
On 2013-07-15 22:40, Ben Morrow <(E-Mail Removed)> wrote:
> Quoth George Mpouras <(E-Mail Removed)> :
>> Create a directory with all upper directories if missing.
>> it uses the minimum possible disk access and checks.
>>
>> Mkdir_recursive('/some/dir/d1/d2') or die;
>>
>> sub Mkdir_recursive
>> {
>> return 1 if $_[0] eq '' || -d $_[0];
>> Mkdir_recursive( $_[0] =~/^(.*?)[\\\/][^\\\/]+$/ ) || return undef;
>> mkdir $_[0] || return undef
>> }

>
> You'd be better off calling mkdir blind and keying off $! if it fails.
> That way you save a stat in the case where the creation succeeds.


That shouldn't make a noticeable difference. If the stat does cause any
disk accesses, those would also have been caused by the mkdir, and if it
doesn't (i.e. everything is already in the cache) the time for the stat
calls is completely swamped by the mkdir's.

To my surprise the second loop of my test program seems actually to be
a bit faster with a blind mkdir, but the difference is less than the
variability, so I'd need more runs to see if the difference is
significant.

hp


--
_ | Peter J. Holzer | Fluch der elektronischen Textverarbeitung:
|_|_) | Sysadmin WSR | Man feilt solange an seinen Text um, bis
| | | (E-Mail Removed) | die Satzbestandteile des Satzes nicht mehr
__/ | http://www.hjp.at/ | zusammenpat. -- Ralph Babel
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
The node.js Community is Quietly Changing the Face of Open Source Rodrick Brown Python 2 04-17-2013 04:47 PM
Re: The node.js Community is Quietly Changing the Face of Open Source Sven Python 0 04-16-2013 04:41 PM
Re: The node.js Community is Quietly Changing the Face of Open Source Ned Batchelder Python 0 04-16-2013 04:25 PM
Is there a difference between the use of the word montage vscollage Danny D. Digital Photography 8 04-15-2013 02:24 PM
Windows 8 - so bad it's hastening the death of the PC? ~misfit~ NZ Computing 18 04-15-2013 04:15 AM



Advertisments