Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Re: Load huge xml file

Reply
Thread Tools

Re: Load huge xml file

 
 
Roedy Green
Guest
Posts: n/a
 
      04-14-2009
On Tue, 14 Apr 2009 20:48:47 +0200, Zibi <(E-Mail Removed)> wrote,
quoted or indirectly quoted someone who said :

><root>
> <element1>...</element1>
> <element1>...</element1>
> <element2>...</element2>
> <element2>...</element2>
></root>


That was a file that never should have been XML in the first place
since there is no nesting.

I suggest writing a custom little program that reads it line by line
and writes out the equivalent CSV file using CSV Writer,
http://mindprod.com/products1.html#CSV (now supports comments) of the
form:

element1, data
element1, data
element2, data
element2, data

Then you can process it easily with CSVReader. You can also look and
edit it with a text editor.

You might use regex split or simple indexOf to find the pieces.

XML was intended for small complex files. They are analysed as a tree
totally ram resident. 100 MB is still only 0.1 gig. Use JET which auto
adjusts the heap size depending on available RAM.
http://mindprod.com/jgloss/jet.html

With the Sun JVM, there are command line options to increase the
virtual address space. http://mindprod.com/jgloss/javaexe.html

Another approach is to split your file in pieces and process the
pieces separately, then join the data in a binary DataOutputSTream
file.

See http://mindprod.com/products1.html#SPLICE (cut a file up based on
offsets)

http://mindprod.com/products1.html#SPLITTER (break a file up based on
embedded markers)

--
Roedy Green Canadian Mind Products
http://mindprod.com

For most people, a global average temperature rise of 3C (5F) does not sound
alarming. They dont understand that the warming is uneven, mostly over the
continents which will rise 6C (11F). The maximum annual temperature will
increase considerably more. Paris will have the temperature of Algiers,
Moscow the temperature of Paris, San Francisco the temperature of Tijuana and
New York the temperature of Charleston, and Houston the temperature of some
obscure dusty town in southern Mexico.
 
Reply With Quote
 
 
 
 
Arne Vajhj
Guest
Posts: n/a
 
      04-20-2009
bugbear wrote:
> Roedy Green wrote:
>> On Tue, 14 Apr 2009 20:48:47 +0200, Zibi <(E-Mail Removed)> wrote,
>> quoted or indirectly quoted someone who said :
>>
>>> <root>
>>> <element1>...</element1>
>>> <element1>...</element1>
>>> <element2>...</element2>
>>> <element2>...</element2>
>>> </root>

>>
>> That was a file that never should have been XML in the first place
>> since there is no nesting.

>
> He *did* say:
>
> "Elements are quite complicated I have XSD for entire XML."


Not everyone bother to read what they are replying to.

Arne
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Load huge xml file lord.zoltar@gmail.com Java 1 04-15-2009 01:12 AM
Memory error due to the huge/huge input file size tejsupra@gmail.com Python 3 11-20-2008 07:21 PM
Java and huge XML file to be parsed Katrin Tomanek Java 47 04-15-2006 04:13 AM
Different results parsing a XML file with XML::Simple (XML::Sax vs. XML::Parser) Erik Wasser Perl Misc 5 03-05-2006 10:09 PM
how to write some xml into huge xml file into speceific location??? Xenia XML 4 11-25-2003 07:14 PM



Advertisments