Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Excessive Memory Usage with libxml

Reply
Thread Tools

Excessive Memory Usage with libxml

 
 
Daniel N
Guest
Posts: n/a
 
      05-22-2007
Hi,

I hope this is the right place for this.

I'm writing a markup mangler that I'm using libxml for. Primarily for
speed reasons.

I've got some basic functionality going, it's not finished, and when
I'm doing a benchmark I get some good figures.

I'm using a recursive function to process my data. The general gist
of it is as follows

http://pastie.caboo.se/63584


But...

When I run the benchmark I get weird things happening.

All benchmarks exhibit steadily increasing memory usage until the end
of the benchmark
10000 iterations were fine
very much above this and I started to get

malloc errors
or
segmentation fault


I changed
doc.root.to_s.gsub( /\<\/?#{artificial_root_tag}\>/, "" )

to
doc.root.to_a.join.gsub( /\<\/?#{artificial_root_tag}\>/, "" )

taking a hit on performance, but it allowed me to get through 100,000
iterations.
The memory usage though was stupid at 235Mb Real and 233Mb Virtual

The HTML I am parsing is a fairly small snippet

I'm running on Mac OsX.

Any ideas what and why?
thankyou

Daniel

 
Reply With Quote
 
 
 
 
Daniel N
Guest
Posts: n/a
 
      05-22-2007
On 5/23/07, Daniel N <(E-Mail Removed)> wrote:
> Hi,
>
> I hope this is the right place for this.
>
> I'm writing a markup mangler that I'm using libxml for. Primarily for
> speed reasons.
>
> I've got some basic functionality going, it's not finished, and when
> I'm doing a benchmark I get some good figures.
>
> I'm using a recursive function to process my data. The general gist
> of it is as follows
>
> http://pastie.caboo.se/63584
>
>
> But...
>
> When I run the benchmark I get weird things happening.
>
> All benchmarks exhibit steadily increasing memory usage until the end
> of the benchmark
> 10000 iterations were fine
> very much above this and I started to get
>
> malloc errors
> or
> segmentation fault
>
>
> I changed
> doc.root.to_s.gsub( /\<\/?#{artificial_root_tag}\>/, "" )
>
> to
> doc.root.to_a.join.gsub( /\<\/?#{artificial_root_tag}\>/, "" )
>
> taking a hit on performance, but it allowed me to get through 100,000
> iterations.
> The memory usage though was stupid at 235Mb Real and 233Mb Virtual
>
> The HTML I am parsing is a fairly small snippet
>
> I'm running on Mac OsX.
>
> Any ideas what and why?
> thankyou
>
> Daniel
>
>


I've managed to reproduce the error by changing to Benchmark.bmbm. It is

ruby(3775) malloc: *** Deallocation of a pointer not malloced:
0x3903a60; This could be a double free(), or free() called with the
middle of an allocated block; Try setting environment variable
MallocHelp to see tools to help debug
ruby(3775) malloc: *** Deallocation of a pointer not malloced:
0x75b0750; This could be a double free(), or free() called with the
middle of an allocated block; Try setting environment variable
MallocHelp to see tools to help debug
ruby(3775) malloc: *** Deallocation of a pointer not malloced:
0x75c4950; This could be a double free(), or free() called with the
middle of an allocated block; Try setting environment variable
MallocHelp to see tools to help debug
/markup_mangler_benchmark.rb:79: [BUG] Segmentation fault

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
example of excessive memory use (shootout) fft1976 Java 9 07-01-2009 02:59 AM
best practice to avoiding excessive memory usage?? Chris Perl Misc 9 11-20-2006 11:05 AM
C++ libraries: Xerces, libxml/libxml++ or perhaps Arabica? Olav XML 3 01-20-2005 02:51 PM
Pentax Optio S40 Excessive Battery Usage Tom Digital Photography 5 12-25-2004 04:48 PM
Problems with libxml, XML::LibXML and Perl Ian Gregory XML 1 07-25-2003 04:20 PM



Advertisments