Velocity Reviews > Perl > Problem in nested sorts

# Problem in nested sorts

Anno Siegel
Guest
Posts: n/a

 03-25-2006
Peter J. Holzer <(E-Mail Removed)> wrote in comp.lang.perl.misc:
> Anno Siegel wrote:
>
> > MSG <(E-Mail Removed)> wrote in comp.lang.perl.misc:
> >> David McNerney wrote:

> >
> > [...]
> >
> >> > Below is a test case, verified to demonstrate the problem on Perl
> >> > 5.8.5 (Linux) and 5.8.6 (Mac OS X):
> >> >
> >> >
> >> > #!/usr/bin/perl -w
> >> > use strict;
> >> >
> >> > package another;
> >> > #package my_package;
> >> > sub testFunction
> >> > {
> >> > my @otherList = sort
> >> > {
> >> > print "[testFunction, sort 1] a=\$a b=\$b\n";
> >> > \$a <=> \$b
> >> > } (5,4,3,2,1);
> >> > print "\@otherList, sorted: ".join(", ", @otherList)."\n";
> >> >
> >> > my @otherList2 = sort
> >> > {
> >> > print "[testFunction, sort 2] a=\$a b=\$b\n";
> >> > \$a <=> \$b
> >> > } (5,4,3,2,1);
> >> > print "\@otherList2, sorted: ".join(", ", @otherList2)."\n";
> >> > }
> >> >
> >> > package my_package;
> >> > my @list = (13,12,11);
> >> > @list = sort
> >> > {
> >> > &another::testFunction;
> >> > #&testFunction;
> >> > return \$a <=> \$b;
> >> > } @list;
> >> > print "\@list, sorted: ".join(", ", @list)."\n";
> >> >
> >> > #&another::testFunction;

> >
> > [...]
> >
> >> ... So if you go
> >> back to your code and comment out those two print statements
> >> inside the sort sub block, your code works perfectly!!

> >
> > I don't see the effect you claim to see.

>
> I see it.
>
> > Have you run your modified test with warnings switched on?

>
> -w is in the shebang line, isn't it?

Ah, right -w. How unfashionable I had to change the shebang line
and -w got lost in the process. That's perhaps another reason to
avoid flags in the shebang line if possible. It makes a routine
replacement easier.

> > Do so. If you still believe the code works perfectly, please post
> > exactly what you are running.

>
> Exactly the code posted by David, except that the two print statements
> in another::testFunction are commented out. Works with perl 5.8.0, 5.8.4
> and 5.8.7 on Linux. I also modified the second half of testFunction to:
>
> my @values2 = sort
> {
> # print "[testFunction, sort function 2] a=\$a b=\$b\n";
> \$a <=> \$b
> } (1,3,4,2,5);
> die unless (@values2 == 5 && \$values2[0] == 1 && \$values2[2] == 3);
>
> to verify that the sort works correctly. It does.
>
> I only get warnings if there is a print statement in both sorts.

So commenting out only one of the print statements is enough?

I get warnings with and without the print statements. Perl 5.8.7,
on Darwin 8.5.0 and Linux 2.0.36.

[discussion snipped]

Anno
--
If you want to post a followup via groups.google.com, don't use
the broken "Reply" link at the bottom of the article. Click on
"show options" at the top of the article, then click on the

Peter J. Holzer
Guest
Posts: n/a

 03-25-2006
Anno Siegel wrote:
> Peter J. Holzer <(E-Mail Removed)> wrote in comp.lang.perl.misc:
>> Anno Siegel wrote:
>> > MSG <(E-Mail Removed)> wrote in comp.lang.perl.misc:
>> >> David McNerney wrote:
>> >> > Below is a test case, verified to demonstrate the problem on
>> >> > Perl 5.8.5 (Linux) and 5.8.6 (Mac OS X):

[...]
>> >> ... So if you go back to your code and comment out those two print
>> >> statements inside the sort sub block, your code works perfectly!!
>> >
>> > I don't see the effect you claim to see.

>>
>> I see it.
>>
>> > Have you run your modified test with warnings switched on?

>>
>> -w is in the shebang line, isn't it?

>
> Ah, right -w. How unfashionable I had to change the shebang line
> and -w got lost in the process. That's perhaps another reason to
> avoid flags in the shebang line if possible. It makes a routine
> replacement easier.

Hmm. I don't think I ever accidentally removed it when I replaced the
interpreter in the file. But I agree that it is easy to forget if you
invoke the perl-interpreter explicitely on the command line.

Your comment made me think that maybe there is a difference between -w
and "use warnings;" in this case, but there isn't.

>> I only get warnings if there is a print statement in both sorts.

>
> So commenting out only one of the print statements is enough?

Yes.

> I get warnings with and without the print statements. Perl 5.8.7,
> on Darwin 8.5.0 and Linux 2.0.36.

Very strange. Perl 5.8.7 on Linux was also one of the versions I tested:

Summary of my perl5 (revision 5 version 8 subversion 7) configuration:
Platform:
osname=linux, osvers=2.4.21-32.elsmp, archname=i686-linux
uname='linux habanero.wsr.ac.at 2.4.21-32.elsmp #1 smp fri apr 15
21:17:59 edt 2005 i686 i686 i386 gnulinux '
config_args=''
hint=recommended, useposix=true, d_sigaction=define
usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-fno-strict-aliasing -pipe -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2',
cppflags='-fno-strict-aliasing -pipe -I/usr/local/include'
ccversion='', gccversion='3.2.3 20030502 (Red Hat Linux 3.2.3-53)',
gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
ld='cc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -ldl -lm -lcrypt -lutil -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
libc=/lib/libc-2.3.2.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.3.2'
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl):
Compile-time options: USE_LARGE_FILES
Built under linux
Compiled at Jan 19 2006 14:53:11
@INC:
/usr/local/lib/perl5/5.8.7/i686-linux
/usr/local/lib/perl5/5.8.7
/usr/local/lib/perl5/site_perl/5.8.7/i686-linux
/usr/local/lib/perl5/site_perl/5.8.7
/usr/local/lib/perl5/site_perl

MSG
Guest
Posts: n/a

 03-25-2006
Anno Siegel wrote:

>
> I get warnings with and without the print statements. Perl 5.8.7,
> on Darwin 8.5.0 and Linux 2.0.36.
>

What kind of warnings? I don''t get any thing except the correct
result when taking out those two pritnt statements.
Here is the exact code I ran. It is David's code but with print
commented out. Also I switched -w to 'use warnings' to avoide
any possible confusion.

##############
#!/usr/bin/perl
use strict;
use warnings;

package another;
#package my_package;
sub testFunction
{
my @otherList = sort
{
# print "[testFunction, sort 1] a=\$a b=\$b\n";
\$a <=> \$b
} (5,4,3,2,1);
print "\@otherList, sorted: ".join(", ", @otherList)."\n";

my @otherList2 = sort
{
# print "[testFunction, sort 2] a=\$a b=\$b\n";
\$a <=> \$b
} (5,4,3,2,1);
print "\@otherList2, sorted: ".join(", ", @otherList2)."\n";

}

package my_package;
my @list = (13,12,11);
@list = sort
{
&another::testFunction;
#&testFunction;
return \$a <=> \$b;
} @list;

print "\@list, sorted: ".join(", ", @list)."\n";

###############
Here is the result:
@otherList, sorted: 1, 2, 3, 4, 5
@otherList2, sorted: 1, 2, 3, 4, 5
@otherList, sorted: 1, 2, 3, 4, 5
@otherList2, sorted: 1, 2, 3, 4, 5
@list, sorted: 11, 12, 13

It gives clean and correct result as long as you keep the sort sub
clean.

Anno Siegel
Guest
Posts: n/a

 03-26-2006
MSG <(E-Mail Removed)> wrote in comp.lang.perl.misc:
> Anno Siegel wrote:
>
> >
> > I get warnings with and without the print statements. Perl 5.8.7,
> > on Darwin 8.5.0 and Linux 2.0.36.
> >

>
> What kind of warnings? I don''t get any thing except the correct
> result when taking out those two pritnt statements.
> Here is the exact code I ran. It is David's code but with print
> commented out. Also I switched -w to 'use warnings' to avoide
> any possible confusion.

The behavior is erratic. As reported earlier, I have seen the warnings
(and invalid sort results) without any of the two critical print statements.
Currently I have two scripts, one derived from the OP, the other from
the code you posted (now snipped). Both don't show the warnings when
all prints are commented out, but only with one of them is it sufficient
to comment out one of the warnings. Go figure. I don't even bother
to identify which script behaves in which way. It makes no sense on
Perl level.

Anno
--
If you want to post a followup via groups.google.com, don't use
the broken "Reply" link at the bottom of the article. Click on
"show options" at the top of the article, then click on the

MSG
Guest
Posts: n/a

 03-26-2006
Anno Siegel wrote:
>
> The behavior is erratic. As reported earlier, I have seen the warnings
> (and invalid sort results) without any of the two critical print statements.

I am afraid that you were not testing it right. For the code I posted
above, how many times did it give you a warning or a wrong result? Can
you re-produce it?

> Currently I have two scripts, one derived from the OP, the other from
> the code you posted (now snipped). Both don't show the warnings when
> all prints are commented out, but only with one of them is it sufficient
> to comment out one of the warnings.

I don't understand what you are saying here. Can you elaborate what
you are trying to test?

David McNerney
Guest
Posts: n/a

 03-26-2006
I duplicate your finding ... if those two print statements are
commented out, both the nested sorts work correctly. So, it certainly
does seem that the sort operator cannot always be relied upon when used
with anonymous functions that contain anything but simple comparison.
That's really fine, as far as I'm concerned, now that I know ... I can
always declare a prototyped sort function in the future if I'm writing
code that I expect to get called from within a sort. I don't, however,
find any warnings about such code in the "sort" function documentation
in the Camel book, my primary reference.

There are any number of reasons why one might want to have what you
refer to as "extra" code in a sort. This comes up when sorting objects.
We have a large class library at our site, and one of the chief reasons
that we use Perl, and not some other language like Java, C, or Visual
Basic, is because it allows us to program at a high conceptual level,
throwing around lists of objects with sort, grep and map. If you are
sorting a list of objects, you might have code like this:

my @sortedList = sort { \$a->method() <=> \$b->method() } @list;

Now, the "method" method might be a simple attribute accessor, or it
might have more complex code inside. It could even call other methods,
that in turn call others, one of which happens to do a couple of brief
sorts in its implementation. Not an efficiency issue, if the list of
objects is not so long, and the code that does all this is not to be
executed many times. Again, one of the main reasons we use Perl at our
site is to leverage >programmer< efficiency.

Thanks again to everyone who has taken an interest in this little
quirk. I posted it to get some perspective on what was going on; in
around 8 years of heavy Perl use, this is only the second or third time
that we've been bitten by a bug in the language. That's a great record,
and this little issue can easily be worked around now that I know about
it.

MSG wrote:
> David McNerney wrote:
> > Thanks very much to everyone who replied here. Sorry I was a little
> > terse Brian! I just sent the following to http://www.velocityreviews.com/forums/(E-Mail Removed):
> >
> > When an anonymous sort function invokes a method or function defined in
> > another package, and that
> > method or function does multiple sorts using its own anonymous sort
> > functions, the second and later
> > of these latter sorts fail; \$a and \$b are not defined. This does not
> > occur if the function is moved
> > to the same package as the code performing the first sort, nor does it
> > occur if the latter "nested"
> > sorts use a seperately defined sort function with prototypes (in which
> > case, of course, the \$a and \$b
> > variables are not used). And, as implied above, the first "nested" sort
> > executes correctly.
> >
> > One person on usenet suggested that nobody ought to be doing such a
> > thing as described above. Actually,
> > there are many times when working with a large object-oriented Perl
> > library when one might need
> > to sort a (often short) list of objects, and compare values returned by
> > object methods in the comparison function.
> > A programmer writing such code should not have to think about whether
> > the object methods they choose
> > to invoke happen to use multiple non-prototyped sorts in their
> > implementation.
> >
> > Below is a test case, verified to demonstrate the problem on Perl 5.8.5
> > (Linux) and 5.8.6 (Mac OS X):
> >
> >
> > #!/usr/bin/perl -w
> > use strict;
> >
> > package another;
> > #package my_package;
> > sub testFunction
> > {
> > my @otherList = sort
> > {
> > print "[testFunction, sort 1] a=\$a b=\$b\n";
> > \$a <=> \$b
> > } (5,4,3,2,1);
> > print "\@otherList, sorted: ".join(", ", @otherList)."\n";
> >
> > my @otherList2 = sort
> > {
> > print "[testFunction, sort 2] a=\$a b=\$b\n";
> > \$a <=> \$b
> > } (5,4,3,2,1);
> > print "\@otherList2, sorted: ".join(", ", @otherList2)."\n";
> > }
> >
> > package my_package;
> > my @list = (13,12,11);
> > @list = sort
> > {
> > &another::testFunction;
> > #&testFunction;
> > return \$a <=> \$b;
> > } @list;
> > print "\@list, sorted: ".join(", ", @list)."\n";
> >
> > #&another::testFunction;

>
> Although I agree that you have an rather interesting discovery,
> (which may take a real Perl expert to explain), there is one even
> more "interesting" that you haven't discovered yet, and this one is
> going to reveal the fault on your part:
>
> Every Perl textbook states that sort() is "special", its sub is
> "special" and its \$a and \$b are "special". Some even warns "weird
> and unexpected result" about messing with those \$a and \$b. Your
> code demenstrated exactly that -- The subs for sort should contain
> only comparision tests. Nothing more, nothing less! So if you go
> back to your code and comment out those two print statements
> inside the sort sub block, your code works perfectly!!
>
> In retrospect, I don't remember ever seeing anyone putting "extra"
> statements inside a sort sub.

Brian McCauley
Guest
Posts: n/a

 03-26-2006

Brian McCauley wrote:
> I've reproduced your problem and it does look like a particularly nasty
> bug in perl.

Well I've not tracked it down but here's something interesting.

If I evalute \\$a,\\$a,*a{SCALAR} (where \$a is not declared lexically)
then I'd expect to get the same value three times. And in the absense
of sort thats what I do see. But inside sort something odd happens.
And it looks like under some circumstances this something odd fails to
happen properly. And now I think about it this may explain problems
I've seen in my programs in the past.

I haven't yet fully understood the behaviour of the following test but
here it is anyhow.

use strict;
no warnings;

package P;

{
my (%xref,@keep_alive);
my \$i=1;
sub p {
printf "%-7s", shift;
for (@_) {
# Give each a new number the first time we see it
printf "%10s ", \$xref{\$_} ||= do {
# Avoid same address ever being alocated to another SV
push @keep_alive => \$_;
\$i++;
};
}
print "\n";
}
}

my \$A='a'; # A symref

print " \\\\$a \\\\$a *a{SCALAR}\n";

sub testFunction {
no strict 'refs';
p 'before',\\$a,\\$a,*a{SCALAR};
my @values = sort
{
p 'sort1',\\$a,\\$a,*a{SCALAR};
\$a <=> \$b;
} (1,2);
p 'between',\\$a,\\$a,*a{SCALAR};
my @values2 = sort
{
p 'sort2',\\$a,\\$a,*a{SCALAR};
\$a <=> \$b;
} (1,2);
p 'after',\\$a,\\$a,*a{SCALAR};
}

print "------------------------------------------------------ in P\n";
testFunction;
print "------------------------------------------------------ in sort
in P\n";
my @seqnums1 = sort
{
testFunction;
return \$a <=> \$b;
} 48561006,488561006;
print "------------------------------------------------------ in Q\n";
package Q;
P::testFunction;
print "------------------------------------------------------ in sort
in Q\n";
my @seqnums2 = sort
{
&P::testFunction;
return \$a <=> \$b;
} 486098561006,486098561006;
__END__
\\$a \\$a *a{SCALAR}
------------------------------------------------------ in P
before 1 1 1
sort1 2 3 4
between 1 1 1
sort2 5 6 7
after 1 1 1
------------------------------------------------------ in sort in P
before 8 9 10
sort1 11 12 4
between 13 14 10
sort2 15 16 7
after 17 18 10
------------------------------------------------------ in Q
before 1 1 1
sort1 19 20 4
between 1 1 1
sort2 21 22 7
after 1 1 1
------------------------------------------------------ in sort in Q
before 1 1 1
sort1 23 24 4
between 1 1 1
sort2 1 1 1
after 1 1 1

Anno Siegel
Guest
Posts: n/a

 03-26-2006
MSG <(E-Mail Removed)> wrote in comp.lang.perl.misc:
> Anno Siegel wrote:
> >
> > The behavior is erratic. As reported earlier, I have seen the warnings
> > (and invalid sort results) without any of the two critical print statements.

> I am afraid that you were not testing it right. For the code I posted
> above, how many times did it give you a warning or a wrong result? Can
> you re-produce it?

Hm? What has the number of warnings to do with anything? It is determined
by how many comparisons the sort algorithm does for the given lists.
I get 16 warnings, reproducibly.

> > Currently I have two scripts, one derived from the OP, the other from
> > the code you posted (now snipped). Both don't show the warnings when
> > all prints are commented out, but only with one of them is it sufficient
> > to comment out one of the warnings.

> I don't understand what you are saying here. Can you elaborate what
> you are trying to test?

Sigh. This is pretty useless because the behavior seems to be governed
by factors we don't control. Even if you run my code and see different
results we'd be none the wiser. If you really want to follow this up:

This code gives me warnings:

#!/usr/local/bin/perl
use strict; use warnings; \$| = 1;

package another;
sub testFunction
{
my @values = sort
{
print "[testFunction, sort 1] a=\$a b=\$b\n";
\$a <=> \$b
} (5,4,3,2,1);

my @values2 = sort
{
# print "[testFunction, sort 2] a=\$a b=\$b\n";
\$a <=> \$b
} (5,4,3,2,1);

}

package my_package;
my @list = (13,12,11);
@list = sort
{
&another::testFunction;
return \$a <=> \$b;
} @list;

print "\@list, sorted: ".join(", ", @list)."\n";
__END__

This code runs without warnings:

#!/usr/local/bin/perl
use strict; use warnings; \$| = 1;

package another;
sub testFunction
{
my @values = sort
{
print "[testFunction, sort function 1] a=\$a b=\$b\n";
\$a <=> \$b
} (1,2,3,4,5);

my @values2 = sort
{
# print "[testFunction, sort function 2] a=\$a b=\$b\n";
\$a <=> \$b
} (1,2,3,4,5);
}

package my_package;
my @seqnums = (486098561006,486098561006);
@seqnums = sort
{
&another::testFunction;
return \$a <=> \$b;
} @seqnums;

print "\@seqnums, sorted: ".join(", ", @seqnums)."\n";
__END__

Anno
--
If you want to post a followup via groups.google.com, don't use
the broken "Reply" link at the bottom of the article. Click on
"show options" at the top of the article, then click on the

robic0
Guest
Posts: n/a

 03-27-2006
On Sat, 25 Mar 2006 19:29:12 +0100, "Peter J. Holzer" <(E-Mail Removed)> wrote:

>Anno Siegel wrote:
>> Peter J. Holzer <(E-Mail Removed)> wrote in comp.lang.perl.misc:
>>> Anno Siegel wrote:
>>> > MSG <(E-Mail Removed)> wrote in comp.lang.perl.misc:
>>> >> David McNerney wrote:
>>> >> > Below is a test case, verified to demonstrate the problem on
>>> >> > Perl 5.8.5 (Linux) and 5.8.6 (Mac OS X):

>[...]
>>> >> ... So if you go back to your code and comment out those two print
>>> >> statements inside the sort sub block, your code works perfectly!!
>>> >
>>> > I don't see the effect you claim to see.
>>>
>>> I see it.
>>>
>>> > Have you run your modified test with warnings switched on?
>>>
>>> -w is in the shebang line, isn't it?

>>
>> Ah, right -w. How unfashionable I had to change the shebang line
>> and -w got lost in the process. That's perhaps another reason to
>> avoid flags in the shebang line if possible. It makes a routine
>> replacement easier.

>
>Hmm. I don't think I ever accidentally removed it when I replaced the
>interpreter in the file. But I agree that it is easy to forget if you
>invoke the perl-interpreter explicitely on the command line.
>
>Your comment made me think that maybe there is a difference between -w
>and "use warnings;" in this case, but there isn't.
>
>

[snipped]
Just a comment on -w and use warnings under windows.
Using the "shebanged the football team" line with -w
warnings showed up in the command console the first time it was
run. Subsequent "uparrow return"'s didn't show the results
depending on wheather the notepad code file was re-saved between
runs. It was intermittent.

Probably just my Perl installation but have seen this console behaviour
before. Just got rid of it and used the use warnings.
I can probably tell you that Perl's phase 1 (?) parser has problems.
Who knows how far back thats existed. Its pretty niaeve to think of
Perl as perfect.

robic0

robic0
Guest
Posts: n/a

 03-27-2006
On 25 Mar 2006 18:58:53 -0800, "MSG" <(E-Mail Removed)> wrote:

>Anno Siegel wrote:
>>
>> The behavior is erratic. As reported earlier, I have seen the warnings
>> (and invalid sort results) without any of the two critical print statements.

>I am afraid that you were not testing it right. For the code I posted
>above, how many times did it give you a warning or a wrong result? Can
>you re-produce it?
>
>> Currently I have two scripts, one derived from the OP, the other from
>> the code you posted (now snipped). Both don't show the warnings when
>> all prints are commented out, but only with one of them is it sufficient
>> to comment out one of the warnings.

>I don't understand what you are saying here. Can you elaborate what
>you are trying to test?

I think its pretty fair to say to solve the argument, the results of the
the "other package" sort, are printed outside of the "other package" sort,
as a final step from the main thread execution chain.
Just make it a package global.

ONE other thing im not comprehending from this post, given the docs on sort.
sort block list
sort subname list

Am I missing something here?

sort {...} (...);
^^^^^

Is this NOT a block? Seems to be implied in this thread its
an "anonymous subroutine".

If it is contextually an anonymous sub, how is a block represented?

Im my eyes, the problem is as simple as I percieved in a different post in this

I can understand not "feeding the troll", but don't take my idea's and make it your

It is as stands an anomoly. The OP did the right thing by submitting this!

robic0