Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > XML > How to process 1000 files xml to 1 file?

Reply
Thread Tools

How to process 1000 files xml to 1 file?

 
 
AMDx64BT
Guest
Posts: n/a
 
      03-07-2011
-- DESCRIPTION -----------------------

I would like to write a script with awk or vim to process Lab Blood Tests in xml format to

import with SPSS.

Each blood test is an xml file.

If I have (for example):
1000 Lab Blood Tests (1000 xml files)
250 patients
4 blood tests/patient

Each Blood Test file has the name format: rapport_33405954.xml

Each blood test has a variable number of components but I am interested to analyze only 3

elements: K, Na and Ca. (These elements are not included in all the Blood Tests.)


-- PATIENT -----------------------

-- SOURCE:

<Patient>
<lbpa_Npa>1234</lbpa_Npa>
<lbpa_Nai>02-Oct-1923 00:00:00</lbpa_Nai>
<Entree>15-Oct-1582 01:00:00</Entree>
<Pid>0</Pid>
<Ncas>0</Ncas>
<pre10 />
<lbpa_Pre>Peter</lbpa_Pre>
<lbpa_Num_Npat>1234567</lbpa_Num_Npat>
<lbrq_Nom1 />
<lbpa_Adr2>Paris</lbpa_Adr2>
<lbrq_Nom2 />
<lbpa_Sexe>M</lbpa_Sexe>
<nom10 />
<lbrq_Rid>0</lbrq_Rid>
<Actif />
<lbpa_Adr />
<lbpa_Nom>Smith</lbpa_Nom>
<Adm />
</Patient>

-- RESULT:

(first_name second_name, date_born)
lbpa_Nom lbpa_Pre, lbpa_Nai
Smith Peter, 1923.10.02


-- DATE TAKEN BLOOD -----------------------

--SOURCE:

<Demande>
<Entree>15-Oct-1582 01:00:00</Entree>
<lbde_Rid>12345</lbde_Rid>
<lbde_Nlab>12345</lbde_Nlab>
<Sortie>15-Oct-1582 01:00:00</Sortie>
<NarunaFile />
<Ncas>0</Ncas>
<Etabl />
<lbde_Num_Npat>12345</lbde_Num_Npat>
<Naruna />
<Date_Mod>01-Jan-1900 00:00:00</Date_Mod>
<Taille>0</Taille>
<lbde_pid>12345/111</lbde_pid>
<TCollection>0</TCollection>
<Semgr>0</Semgr>
<lbrq_nom1 />
<lbde_Dtprv>02-Mar-2011 06:00:00</lbde_Dtprv>
<Pathologique>FALSE</Pathologique>
<lbrq_nom2 />
<Bacterio>FALSE</Bacterio>
<Volume>0</Volume>
<Type_www />
<Poids>0</Poids>
<lbde_Dtdem>02-Mar-2011 07:18:32</lbde_Dtdem>
<PasVue>FALSE</PasVue>
<par />
<Domaine />
</Demande>

-- RESULT:

(Date_taken_blood)
lbde_Dtprv
2011.03.02


-- ELEMENT -----------------------

-- SOURCE:

<Analyse>
<OrdreImpression>12345</OrdreImpression>
<CodeMateriel />
<TypeLigne>0</TypeLigne>
<Formulaire>21</Formulaire>
<Norme>136 - 145 mmol/l</Norme>
<Code>2039</Code>
<Commentaire />
<Anterieur />
<TypeResultat>0</TypeResultat>
<Resultat>136</Resultat>
<Unite>mmol/l</Unite>
<Remarque />
<Clos>O</Clos>
<Libelle>Sodium</Libelle>
</Analyse>

-- RESULT:

(Element number)
Libelle Resultat
Sodium 136


-- SORT ELEMENT BY DATE -----------------------

Sodium 05.01.2011 --> Na1
Sodium 08.01.2011 --> Na3
Sodium 06.01.2011 --> Na2


-- FINAL RESULT -----------------------

From 1000 files I want to obtain a file with this format. To be able to import it with

SPSS:

Na1 Na2 K1 K2
Smith Peter 19231002 136 133 4 3.5
Gates Edward 19801204 145 166 3.1 3.4

(In this case the date of Na1 of Smith and Gates, could be different, but the variable Na1

is the same)


Any advice is appreciated
 
Reply With Quote
 
 
 
 
Joe Kesselman
Guest
Posts: n/a
 
      03-08-2011
Well, that is the kind of problem which can be solved by the canonical
Desperate Perl Hacker (DPH) by writing a specialized parser. I don't
know enough about either awk or vim to know whether they would be able
to do this for you or not.

Personally, I'd do it in Java using an off-the-shelf XML parser. The DPH
code might actually run faster, but would be less robust and less
maintainable.
 
Reply With Quote
 
 
 
 
Peter Flynn
Guest
Posts: n/a
 
      03-08-2011
On 07/03/11 19:06, AMDx64BT wrote:
> -- DESCRIPTION -----------------------
>
> I would like to write a script with awk or vim to process Lab Blood Tests in xml format to
>
> import with SPSS.
>
> Each blood test is an xml file.
>
> If I have (for example):
> 1000 Lab Blood Tests (1000 xml files)
> 250 patients
> 4 blood tests/patient
>
> Each Blood Test file has the name format: rapport_33405954.xml
>
> Each blood test has a variable number of components but I am interested to analyze only 3
>
> elements: K, Na and Ca. (These elements are not included in all the Blood Tests.)
>
>
> -- PATIENT -----------------------
>
> -- SOURCE:
>
> <Patient>
> <lbpa_Npa>1234</lbpa_Npa>
> <lbpa_Nai>02-Oct-1923 00:00:00</lbpa_Nai>
> <Entree>15-Oct-1582 01:00:00</Entree>
> <Pid>0</Pid>
> <Ncas>0</Ncas>
> <pre10 />
> <lbpa_Pre>Peter</lbpa_Pre>
> <lbpa_Num_Npat>1234567</lbpa_Num_Npat>
> <lbrq_Nom1 />
> <lbpa_Adr2>Paris</lbpa_Adr2>
> <lbrq_Nom2 />
> <lbpa_Sexe>M</lbpa_Sexe>
> <nom10 />
> <lbrq_Rid>0</lbrq_Rid>
> <Actif />
> <lbpa_Adr />
> <lbpa_Nom>Smith</lbpa_Nom>
> <Adm />
> </Patient>
>
> -- RESULT:
>
> (first_name second_name, date_born)
> lbpa_Nom lbpa_Pre, lbpa_Nai
> Smith Peter, 1923.10.02
>
>
> -- DATE TAKEN BLOOD -----------------------
>
> --SOURCE:
>
> <Demande>
> <Entree>15-Oct-1582 01:00:00</Entree>
> <lbde_Rid>12345</lbde_Rid>
> <lbde_Nlab>12345</lbde_Nlab>
> <Sortie>15-Oct-1582 01:00:00</Sortie>
> <NarunaFile />
> <Ncas>0</Ncas>
> <Etabl />
> <lbde_Num_Npat>12345</lbde_Num_Npat>
> <Naruna />
> <Date_Mod>01-Jan-1900 00:00:00</Date_Mod>
> <Taille>0</Taille>
> <lbde_pid>12345/111</lbde_pid>
> <TCollection>0</TCollection>
> <Semgr>0</Semgr>
> <lbrq_nom1 />
> <lbde_Dtprv>02-Mar-2011 06:00:00</lbde_Dtprv>
> <Pathologique>FALSE</Pathologique>
> <lbrq_nom2 />
> <Bacterio>FALSE</Bacterio>
> <Volume>0</Volume>
> <Type_www />
> <Poids>0</Poids>
> <lbde_Dtdem>02-Mar-2011 07:18:32</lbde_Dtdem>
> <PasVue>FALSE</PasVue>
> <par />
> <Domaine />
> </Demande>
>
> -- RESULT:
>
> (Date_taken_blood)
> lbde_Dtprv
> 2011.03.02
>
>
> -- ELEMENT -----------------------
>
> -- SOURCE:
>
> <Analyse>
> <OrdreImpression>12345</OrdreImpression>
> <CodeMateriel />
> <TypeLigne>0</TypeLigne>
> <Formulaire>21</Formulaire>
> <Norme>136 - 145 mmol/l</Norme>
> <Code>2039</Code>
> <Commentaire />
> <Anterieur />
> <TypeResultat>0</TypeResultat>
> <Resultat>136</Resultat>
> <Unite>mmol/l</Unite>
> <Remarque />
> <Clos>O</Clos>
> <Libelle>Sodium</Libelle>
> </Analyse>
>
> -- RESULT:
>
> (Element number)
> Libelle Resultat
> Sodium 136
>
>
> -- SORT ELEMENT BY DATE -----------------------
>
> Sodium 05.01.2011 --> Na1
> Sodium 08.01.2011 --> Na3
> Sodium 06.01.2011 --> Na2
>
>
> -- FINAL RESULT -----------------------
>
> From 1000 files I want to obtain a file with this format. To be able to import it with
>
> SPSS:
>
> Na1 Na2 K1 K2
> Smith Peter 19231002 136 133 4 3.5
> Gates Edward 19801204 145 166 3.1 3.4
>
> (In this case the date of Na1 of Smith and Gates, could be different, but the variable Na1
>
> is the same)
>
>
> Any advice is appreciated


1000 files, but how many Patient, how many Demande, how many Analyse?

Concatenate them into a single patient file, a single demande file and a
single analyse file, then write XSLT2 to process them with lookups using
document(). 1000 is not very many.

Or use lxprintf to extract the data to three files, and then use
standard tools like join(1) to create a single master file.

///Peter
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Re: How to process 1000 files xml to 1 file? AMDx64BT XML 0 03-08-2011 07:25 AM
Optimizing the expression (x * 1000 / 1000) pozz C Programming 27 03-04-2011 04:46 AM
How to process 1000 request using Thread niraj Java 3 03-22-2007 10:07 PM
Sony Mavica CD-1000 deleting files from a disk Jo Digital Photography 1 11-08-2003 09:48 PM



Advertisments
 



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57