«

Arvin Portlock <> wrote in
news:dqehgs$auv$:

> The line in question in ExpatXS is this:
>
> $result = $args->{ParseFunc}->($args->{Parser},
> $args->{ParseFuncParam});
>
> In some context that's:
>
> sub _parse_systemid {
> my $self = shift;
> my $fh = IO::File->new(shift);

It appears that the author of the module has committed a classic error of
the sort that posters here are admonished for: opening a file without
checking whether it actually opened. If it didn't, then $fh will be undef
and attempting to call any methods on it will give an error.

> $self->{ParseOptions}->{ParseFunc} = \&ParseStream;
> $self->{ParseOptions}->{ParseFuncParam} = $fh;
> $self->_parse;
> }

wrote:

> Arvin Portlock wrote:
>
> ># $Id: ExpatXS.pm,v 1.39 2005/11/10 09:38:31 cvspetr Exp $
>
> I've now tried your program with this exact version of ExpatXS.pm, and
> it also works for at least 10_000 files.

And I just tried it on Windows and it works fine, though not
quite the same version of perl and an old version of ExpatXS.
I'm not a system administrator so I had to compile and install
my own local copy of perl and miscellaneous modules, something
I do infrequently. Perhaps I did something wrong there. At
least it isn't a bug in my program, and it seems not to be a
bug in ExpatXS. It seems doubtful it would be a solaris-specific
bug of any kind.

So I could try creating a single-file version of the program
and call it through a loop in a shell script. It's a bit less
convenient to maintain state. I wonder what kind of hit I'll
take doing that? Or I could use pure perl. Except the reason I'm
doing this is the Java DOM solution in place is way too slow
so I was hoping for a more impressive boost in speed. I might
try and bite the bullet and struggle through installing Xerces
which is pretty fast, and, come to think of it, would be useful
in other applications. Maybe I'll try and hack a destructor for
the parse object.

Anyway, I'm just thinking out loud now. Thank you very much
for all of your help. Though the problem remains, it still
helped me out very much.

Arvin

>
>
>
> >This is perl, v5.8.7 built for sun4-solaris-thread-multi
>
>
> This is perl, v5.8.7 built for x86_64-linux
> This is perl, v5.8.3 built for x86_64-linux-thread-multi
> This is perl, v5.8.0 built for i386-linux-thread-multi
>
> (I would have replied to myself, but my first post hasn't showed up for
> me yet. I hope it does eventually.)
>
> Xho
>

"Arvin Portlock" <> wrote in message
news:dqehgs$auv$...
> Matt Garrish wrote:
>
>> wrote in message
>> news:20060115160930.889$...
>>
>> >"Matt Garrish" wrote:
>> >
>> >>"Arvin Portlock" wrote in message
>> >>news:dq9gb7$219p$...
>> >>
>> >>>In using ExpatXS to parse large batches of XML files
>> >>>something curious happens. After parsing something close
>> >>>to 500 files the program crashes with the error:
>> >>>
>> >>>Can't call method "read" on an undefined value at
>> >>>/xxx/perl5.8/lib/site_perl/5.8.7/sun4-solaris-thread-multi/
>> >>>XML/SAX/ExpatXS.pm line 155.
>> >>>
>> >>
>> >>You aren't checking whether you succesfully get an object. I would
>> >>doubt
>> >>it's a memory leak or your system would have ground to a halt.
>> >
>> >But it could easily be a file-handle leak. Around 500 would be a common
>> >per-process limit on open file-handles.
>> >
>>
>>
>> That seems more likely. I just wonder what the read method is that is
>> being
>> called on what undefined object. It's not a particularly helpful error
>> message. It could be coming right from Expat from the sound of it.
>>
>> Matt
>
> The line in question in ExpatXS is this:
>
> $result = $args->{ParseFunc}->($args->{Parser},
> $args->{ParseFuncParam});
>

Sorry, for some reason I thought your script was reporting the error at line
175 (the .pm never seemed to register until just now). That's why I was
wondering what line 175 was. The evals were for nothing, I guess... : )

Matt

>>>In using ExpatXS to parse large batches of XML files
>>>something curious happens. After parsing something close
>>>to 500 files the program crashes with the error:
>>>
>>>Can't call method "read" on an undefined value at
>>>/xxx/perl5.8/lib/site_perl/5.8.7/sun4-solaris-thread-multi/
>>>XML/SAX/ExpatXS.pm line 155.

> wrote:
>I've now tried your program with this exact version of ExpatXS.pm, and
>it also works for at least 10_000 files.

So I made one change to ExpatXS.pm which worked. I explicitly
closed $fh here:

sub _parse_systemid {
my $self = shift;
my $fh = IO::File->new(shift);
$self->{ParseOptions}->{ParseFunc} = \&ParseStream;
$self->{ParseOptions}->{ParseFuncParam} = $fh;
$self->_parse;
$fh->close;
}

I'm a little nervous this solution may be simplistic since
individual XML documents can span multiple files, though
none in this particular batch do. I should test that on
some other files I have. A Google search for:

IO::File perl bug file close solaris

Brings up some interesting things. Problems closing files
are noticeable on Solaris because its descriptor limit
is typically quite low. Quoting:

You can reproduce the problem on other systems, too.
Just set the file descriptor limit to a low value (e.g.
by using limits -n). Other systems like Linux and
FreeBSD have much higher default limits than Solaris.

But I don't know, 10,000 files seems like a pretty large
limit for any system.

Best regards,

Arvin

Arvin Portlock <> wrote:
> >>>In using ExpatXS to parse large batches of XML files
> >>>something curious happens. After parsing something close
> >>>to 500 files the program crashes with the error:
> >>>
> >>>Can't call method "read" on an undefined value at
> >>>/xxx/perl5.8/lib/site_perl/5.8.7/sun4-solaris-thread-multi/
> >>>XML/SAX/ExpatXS.pm line 155.
>
> > wrote:
> >I've now tried your program with this exact version of ExpatXS.pm, and
> >it also works for at least 10_000 files.
>
> So I made one change to ExpatXS.pm which worked. I explicitly
> closed $fh here:
>
> sub _parse_systemid {
> my $self = shift;
> my $fh = IO::File->new(shift);
> $self->{ParseOptions}->{ParseFunc} = \&ParseStream;
> $self->{ParseOptions}->{ParseFuncParam} = $fh;
> $self->_parse;
> $fh->close;
> }

I think you would be better off changing the 3rd line to:

my $fh = IO::File->new(shift) or die ("file open failed with !=$! and
\@=$@");

And seeing exactly what the problem is. That might lead you to a better
way to fix it. (or it might not...) (maybe croak rather than die)


>
> I'm a little nervous this solution may be simplistic since
> individual XML documents can span multiple files, though
> none in this particular batch do. I should test that on
> some other files I have. A Google search for:
>
> IO::File perl bug file close solaris
>
> Brings up some interesting things. Problems closing files
> are noticeable on Solaris because its descriptor limit
> is typically quite low. Quoting:
>
> You can reproduce the problem on other systems, too.
> Just set the file descriptor limit to a low value (e.g.
> by using limits -n). Other systems like Linux and
> FreeBSD have much higher default limits than Solaris.
>
> But I don't know, 10,000 files seems like a pretty large
> limit for any system.

Yes, and I've even let it go much higher than 10,000 (although at that
point it was processing the same 10,000 files over and over, but I don't
think that should matter, each independent opening of the file gets a new
descriptor).

Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB

Arvin Portlock <> wrote:
> wrote:
>
> > Arvin Portlock wrote:
> >
> > ># $Id: ExpatXS.pm,v 1.39 2005/11/10 09:38:31 cvspetr Exp $
> >
> > I've now tried your program with this exact version of ExpatXS.pm, and
> > it also works for at least 10_000 files.
>
> And I just tried it on Windows and it works fine, though not
> quite the same version of perl and an old version of ExpatXS.
> I'm not a system administrator so I had to compile and install
> my own local copy of perl and miscellaneous modules, something
> I do infrequently. Perhaps I did something wrong there. At
> least it isn't a bug in my program, and it seems not to be a
> bug in ExpatXS. It seems doubtful it would be a solaris-specific
> bug of any kind.

Actually I wouldn't be at all surprised if it is a bug in ExpatXS (or a
module it is derived from) which only presents itself on Solaris, or a bug
in the Solaris port of Perl which presents itself when used with ExpatXS.
Try to hack your modules so that you can get the actual error message.


> So I could try creating a single-file version of the program
> and call it through a loop in a shell script. It's a bit less
> convenient to maintain state. I wonder what kind of hit I'll
> take doing that? Or I could use pure perl. Except the reason I'm
> doing this is the Java DOM solution in place is way too slow
> so I was hoping for a more impressive boost in speed.

Since it so easy to do, I would at least try pure perl and prove/quantify
its slowness before trying the single-file hack.



> Maybe I'll try and hack a destructor for
> the parse object.

If you have any luck on that, please let us know. I'm not sure how a lack
of a needed destructor would show up as OS-specific bug, though.

Thanks,

Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB

<> wrote in comp.lang.perl.misc:
> Arvin Portlock <> wrote:
> > wrote:
> > > Arvin Portlock wrote:

[...]

> > Maybe I'll try and hack a destructor for
> > the parse object.
>
> If you have any luck on that, please let us know. I'm not sure how a lack
> of a needed destructor would show up as OS-specific bug, though.

Accumulation of open file handles would be tolerated to varying degrees
by different OSes. A destructor just might avoid the accumulation. Then
again, it may not be called at all, for the same reason the accumulation
happens.

Anno
--
If you want to post a followup via groups.google.com, don't use
the broken "Reply" link at the bottom of the article. Click on
"show options" at the top of the article, then click on the
"Reply" at the bottom of the article headers.

wrote:

> Arvin Portlock wrote:
>
> > >>>In using ExpatXS to parse large batches of XML files
> > >>>something curious happens. After parsing something close
> > >>>to 500 files the program crashes with the error:
> > >>>
> > >>>Can't call method "read" on an undefined value at
> > >>>/xxx/perl5.8/lib/site_perl/5.8.7/sun4-solaris-thread-multi/
> > >>>XML/SAX/ExpatXS.pm line 155.
> >
> > > wrote:
> > >I've now tried your program with this exact version of ExpatXS.pm, and
> > >it also works for at least 10_000 files.
> >
> >So I made one change to ExpatXS.pm which worked. I explicitly
> >closed $fh here:
> >
> >sub _parse_systemid {
> > my $self = shift;
> > my $fh = IO::File->new(shift);
> > $self->{ParseOptions}->{ParseFunc} = \&ParseStream;
> > $self->{ParseOptions}->{ParseFuncParam} = $fh;
> > $self->_parse;
> > $fh->close;
> >}
>
>
> I think you would be better off changing the 3rd line to:
>
> my $fh = IO::File->new(shift) or die ("file open failed with !=$! and
> \@=$@");
>
> And seeing exactly what the problem is. That might lead you to a better
> way to fix it. (or it might not...) (maybe croak rather than die)

The error is as expected, that no more file handles are available.
For now I'll leave the $fh->close, but I want to test it when I have
some time with some XML documents that include external entities.
Not sure how Expat deals with those internally. Perhaps it leaves
unfinished files open while it hunts down entity files, maybe it
closes them first then opens them again where it left off. The former
seems a bit faster but not as safe, e.g., lots of external entity
files = run out of file handles at some point. But in any case I
want to make sure I'm not forcing a file to be closed before Expat
is good and ready. That's why a $parser destructor seems safest
in light of not knowing enough about what's going on. I suppose
I could store the files handles in a static array variable in
_parse_systemid then loop through and close them all upon destruction.

Arvin

[A complimentary Cc of this posting was sent to
Arvin Portlock
<>], who wrote in article <dqgvvg$10ek$>:
> sub _parse_systemid {
> my $self = shift;
> my $fh = IO::File->new(shift);
> $self->{ParseOptions}->{ParseFunc} = \&ParseStream;
> $self->{ParseOptions}->{ParseFuncParam} = $fh;
> $self->_parse;
> $fh->close;
> }

What happens if you apply `local' to assignments to ParseOptions?
There is a leak, and it makes sense to plug the leak as high in the
chain as possible.

What is the lifetime of $self, BTW?

Puzzled,
Ilya