Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > semi-Newbie question

Reply
Thread Tools

semi-Newbie question

 
 
len
Guest
Posts: n/a
 
      08-10-2006
Hi all

I have a file that I receive from another party which is basicly a csv
file containing the following type of information;

Tagname Scope Value
"first_name","POL01","John"
"last_name","POL01","Doe"
"birthday","POL01","04/03/61"
etc

I need to convert this file info into my file format used in my
application.

I have been given a file from the other company that gives me all of
the tagname that could be used and their scope. I want to build a
table which would have all of the various tagnames, scope, and put a
fieldname equivalent in the fieldname in my file structure in my
application. Then read through the vendors csv file index into my
table file and get the field name of where to move the data into my
data structure.

Here is my question? basicly I need to get the data referenced by
fieldname variable in my table and then use that data as a variable
name in a python assignment statement. Thus changing the actual python
code at each iteration through the lines in the csv file.

for i in tagfile
find tagname in tagtable

*** the following line of code would change through each iteration ***
myfirst = value

*** next iteration
mylast = value

*** next iteration
mybirth = value

etc

I hope this make sense. I remember seeing something like this
somewhere.

Any help appreciated.

Len Sumnler
Unique Insurance

 
Reply With Quote
 
 
 
 
davideugenewarren@gmail.com
Guest
Posts: n/a
 
      08-10-2006
Do you absolutely need to use variables? A dictionary would serve if
each case has a unique identifier.

client_info_dict = {}

for i in tagfile:
tagname,scope,value = i.replace('"','').split(',') # split fields,
strip redundant characters
client_info_dict.setdefault(scope,{}) # set up an empty nested
dict for the unique ID
client_info_dict[scope][tagname] = value # set the tagname's value

Then client info can be retrieved from the dictionary using unique IDs
and stereotyped tags (with .get() if some tags are not always present).
I won't make any claims about the efficiency of this approach, but it
works for me.

len wrote:
> Hi all
>
> I have a file that I receive from another party which is basicly a csv
> file containing the following type of information;
>
> Tagname Scope Value
> "first_name","POL01","John"
> "last_name","POL01","Doe"
> "birthday","POL01","04/03/61"
> etc
>
> I need to convert this file info into my file format used in my
> application.
>
> I have been given a file from the other company that gives me all of
> the tagname that could be used and their scope. I want to build a
> table which would have all of the various tagnames, scope, and put a
> fieldname equivalent in the fieldname in my file structure in my
> application. Then read through the vendors csv file index into my
> table file and get the field name of where to move the data into my
> data structure.
>
> Here is my question? basicly I need to get the data referenced by
> fieldname variable in my table and then use that data as a variable
> name in a python assignment statement. Thus changing the actual python
> code at each iteration through the lines in the csv file.
>
> for i in tagfile
> find tagname in tagtable
>
> *** the following line of code would change through each iteration ***
> myfirst = value
>
> *** next iteration
> mylast = value
>
> *** next iteration
> mybirth = value
>
> etc
>
> I hope this make sense. I remember seeing something like this
> somewhere.
>
> Any help appreciated.
>
> Len Sumnler
> Unique Insurance


 
Reply With Quote
 
 
 
 
danielx
Guest
Posts: n/a
 
      08-10-2006
len wrote:
> Hi all
>
> I have a file that I receive from another party which is basicly a csv
> file containing the following type of information;


Python has a csv module. I'm not sure if you're already using that or
if it would be useful to you:

http://docs.python.org/lib/module-csv.html

>
> Tagname Scope Value
> "first_name","POL01","John"
> "last_name","POL01","Doe"
> "birthday","POL01","04/03/61"
> etc
>
> I need to convert this file info into my file format used in my
> application.
>
> I have been given a file from the other company that gives me all of
> the tagname that could be used and their scope. I want to build a


So, a tag name is like a field name, right? What's a scope?

> table which would have all of the various tagnames, scope, and put a
> fieldname equivalent in the fieldname in my file structure in my
> application. Then read through the vendors csv file index into my
> table file and get the field name of where to move the data into my
> data structure.
>
> Here is my question? basicly I need to get the data referenced by
> fieldname variable in my table and then use that data as a variable
> name in a python assignment statement. Thus changing the actual python
> code at each iteration through the lines in the csv file.


I'm not sure that you need to have a different statement execute during
each loop. Why not have something like this:

for line in tagfile:
foreignTag, scope, value = parse(line)
currentRow[ myEqvTag(foreignTag) ] = value
## do you want to do something with the scope information?

If you really wanted to, you could use exec/eval and construct a string
you want to execute/evaluate, but people tend to look down on that. At
least one of the reasons is that it would be slower. Just for
completeness, maybe this is what you want:

for line in tagfile:
ft, sc, v = parse(line)
exec "%s = %s" % ( myEqvTag( ft ), v )

Is this something like what you wanted? I'm not quite sure what you're
asking... More code might help.

Good Luck!

>
> for i in tagfile
> find tagname in tagtable
>
> *** the following line of code would change through each iteration ***
> myfirst = value
>
> *** next iteration
> mylast = value
>
> *** next iteration
> mybirth = value
>
> etc
>
> I hope this make sense. I remember seeing something like this
> somewhere.
>
> Any help appreciated.
>
> Len Sumnler
> Unique Insurance


 
Reply With Quote
 
len
Guest
Posts: n/a
 
      08-10-2006
I appoligize I don't think I have done a very good job of explaining my
problem.

I work in the an insurance company that sells non standard auto. We
have an in house policy management system that uses MySQL for the data
storage.

An individual policy consists of a policy header and 1 to n drivers, 1
to n vehicles and 1 to n coverage records per vehicle.

A third party company will sell a policy and place it with our company.
They collect all of the information on their system and then send this
information to us as a CSV file (one file per policy). **This CSV file
is like an XML file and is formated as followes;
1st field is a tagname
2nd field is a scope
3rd field is a value

see example

"totalpolicypremium","pol0","1584"
"quotenumber","pol0","5"
"address1","pol0","123 Testing Street"
"address2","pol0",""
"apartmentnumber","pol0",""
"cellphone","pol0","( ) -"
....
"annualmiles","car1","0"
"antilock","car1","A"
"antitheft","car1","1"
"bodytype","car1","4D"
"buybackpip","car1","N"
"carphone","car1","N"
"city","car1","ALEXANDRIA"
"coaccdeathlimit","car1","0"
....
"annualmiles","car2","0"
"antilock","car2","N"
"antitheft","car2","1"
"bodytype","car2","4D"
"buybackpip","car2","N"
"carphone","car2","N"
"city","car2","ALEXANDRIA"
....
"address1","drv1","123 Testing Street"
"address2","drv1",""
"agerated","drv1","0"
"banklienjudgstat","drv1","N"
"cellphone","drv1","( ) -"
"city","drv1","Testing City"
"cluestatus","drv1","N"

etc

The third party company sent me a file that contained all of their
valid tagnames and scope which I then took and create a crossreference
file with three fields:

xreffile
tagname (key)
scopy
SQL_fieldname

The program I am writing is nothing more than a conversion program to
take the value out of the CSV file and map it into the appropriate
field in my SQL files. Rather than creating some huge if than else
(there are over 1000 tagnames) I created the xreffile.

Now when I read a record from the tagfile I use the data in the tagname
field to lookup the tagname in my xreffile. The data in the
SQL_fieldname is the fieldname in my SQL files I want to place the data
from the tagfile in the tagfile.value field into this field in my SQL
files;

data referenced by(xreffile.SQL_fieldname) = tagfile.value

what I see as the problem is I want to use what is the data reference
by xreffile.SQL.fieldname and now make it part of the python code as a
reference variable in an assignement code statement.

I hope this helps

Len Sumnler
Unique Insurance

http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> Do you absolutely need to use variables? A dictionary would serve if
> each case has a unique identifier.
>
> client_info_dict = {}
>
> for i in tagfile:
> tagname,scope,value = i.replace('"','').split(',') # split fields,
> strip redundant characters
> client_info_dict.setdefault(scope,{}) # set up an empty nested
> dict for the unique ID
> client_info_dict[scope][tagname] = value # set the tagname's value
>
> Then client info can be retrieved from the dictionary using unique IDs
> and stereotyped tags (with .get() if some tags are not always present).
> I won't make any claims about the efficiency of this approach, but it
> works for me.
>
> len wrote:
> > Hi all
> >
> > I have a file that I receive from another party which is basicly a csv
> > file containing the following type of information;
> >
> > Tagname Scope Value
> > "first_name","POL01","John"
> > "last_name","POL01","Doe"
> > "birthday","POL01","04/03/61"
> > etc
> >
> > I need to convert this file info into my file format used in my
> > application.
> >
> > I have been given a file from the other company that gives me all of
> > the tagname that could be used and their scope. I want to build a
> > table which would have all of the various tagnames, scope, and put a
> > fieldname equivalent in the fieldname in my file structure in my
> > application. Then read through the vendors csv file index into my
> > table file and get the field name of where to move the data into my
> > data structure.
> >
> > Here is my question? basicly I need to get the data referenced by
> > fieldname variable in my table and then use that data as a variable
> > name in a python assignment statement. Thus changing the actual python
> > code at each iteration through the lines in the csv file.
> >
> > for i in tagfile
> > find tagname in tagtable
> >
> > *** the following line of code would change through each iteration ***
> > myfirst = value
> >
> > *** next iteration
> > mylast = value
> >
> > *** next iteration
> > mybirth = value
> >
> > etc
> >
> > I hope this make sense. I remember seeing something like this
> > somewhere.
> >
> > Any help appreciated.
> >
> > Len Sumnler
> > Unique Insurance


 
Reply With Quote
 
John Machin
Guest
Posts: n/a
 
      08-10-2006

len wrote:
> Hi all
>
> I have a file that I receive from another party which is basicly a csv
> file containing the following type of information;
>
> Tagname Scope Value
> "first_name","POL01","John"
> "last_name","POL01","Doe"
> "birthday","POL01","04/03/61"
> etc
>
> I need to convert this file info into my file format used in my
> application.
>
> I have been given a file from the other company that gives me all of
> the tagname that could be used and their scope. I want to build a
> table which would have all of the various tagnames, scope, and put a
> fieldname equivalent in the fieldname in my file structure in my
> application. Then read through the vendors csv file index into my
> table file and get the field name of where to move the data into my
> data structure.
>
> Here is my question? basicly I need to get the data referenced by
> fieldname variable in my table and then use that data as a variable
> name in a python assignment statement. Thus changing the actual python
> code at each iteration through the lines in the csv file.
>
> for i in tagfile
> find tagname in tagtable
>
> *** the following line of code would change through each iteration ***
> myfirst = value
>
> *** next iteration
> mylast = value
>
> *** next iteration
> mybirth = value
>
> etc
>
> I hope this make sense. I remember seeing something like this
> somewhere.
>


You need to define the problem much better than you have in your two
postings so far. Give a small realistic sample of input file and the
relevant parts of the xref file plus (the main missing component so
far) the actual OUTPUT that you desire from that input.

Then we can talk about implementatiion. However while we're waiting,
wtite this on your whiteboard:

(1) *DO* use the csv module; DIY approaches blow up spectacularly when
presented with data which has embedded commas and quotes, and don't say
it can't happen because it does.

(2) *DON'T* pay any attention to suggestions that you should use exec
or eval. The likelihood that your problem *needs* that "the following
line of code would change through each iteration" is very small. The
likelihood that the solution can be written in a straight-forward
manner is high.

What are your "SQL files"? Do you mean insert scripts that you will use
to inject the transformed incoming into a database, or something else?
Again, a short *example* of what you are referring to would help.

HTH,
John

 
Reply With Quote
 
Cameron Laird
Guest
Posts: n/a
 
      08-10-2006
In article <(E-Mail Removed) om>,
len <(E-Mail Removed)> wrote:
>I appoligize I don't think I have done a very good job of explaining my
>problem.

.
.
.
>The program I am writing is nothing more than a conversion program to
>take the value out of the CSV file and map it into the appropriate
>field in my SQL files. Rather than creating some huge if than else
>(there are over 1000 tagnames) I created the xreffile.
>
>Now when I read a record from the tagfile I use the data in the tagname
>field to lookup the tagname in my xreffile. The data in the
>SQL_fieldname is the fieldname in my SQL files I want to place the data
>from the tagfile in the tagfile.value field into this field in my SQL
>files;
>
>data referenced by(xreffile.SQL_fieldname) = tagfile.value
>
>what I see as the problem is I want to use what is the data reference
>by xreffile.SQL.fieldname and now make it part of the python code as a
>reference variable in an assignement code statement.

.
.
.
1. Take Daniel Wong's advice, elsewhere in this thread,
and use the Python CSV module.
2. "what I see as the problem is I want ...": what you
want *is* rather a problem, because it's a troublesome
way to achieve what I understand to be your larger
aims. It was certainly good that you didn't create
"some huge if than else".

Once you have an SQL_fieldname, and a tagfile.value,
what do you want to do? Are you stuffing data into
an SQL table? Continuing on with Python computations?
In almost any case, it sounds as though you will profit
greatly from study of Python's dictionaries <URL:
http://www.developer.com/lang/other/article.php/630721 >
<URL: http://www.diveintopython.org/gettin...tionaries.html >.

 
Reply With Quote
 
len
Guest
Posts: n/a
 
      08-11-2006
Thank all for your reply.

I will try again to state the problem.

I have three files.

1. Tagfile - This file contains data in a CSV format each record in
the file contains three fields 'Tagname', 'Scope', and 'Value' and
exampe of this data file follows;
"totalpolicypremium","pol0","1584"
"quotenumber","pol0","5"
"address1","pol0","123 Testing Street"
"address2","pol0",""
"apartmentnumber","pol0",""
"cellphone","pol0","( ) -"
....
"annualmiles","car1","0"
"antilock","car1","A"
"antitheft","car1","1"
"bodytype","car1","4D"
"buybackpip","car1","N"
"carphone","car1","N"
"city","car1","ALEXANDRIA"
"coaccdeathlimit","car1","0"
....
"annualmiles","car2","0"
"antilock","car2","N"
"antitheft","car2","1"
"bodytype","car2","4D"
"buybackpip","car2","N"
"carphone","car2","N"
"city","car2","ALEXANDRIA"
....
"address1","drv1","123 Testing Street"
"address2","drv1",""
"agerated","drv1","0"
"banklienjudgstat","drv1","N"
"cellphone","drv1","( ) -"
"city","drv1","Testing City"
"cluestatus","drv1","N"

etc

I have already written the code that can pass through this file using
the CSV module and extract the data as I need it from this file, no
problem here.

2. TagToSQL - This is a file which I created which also contains 3
fields as follows
'theirTagname', 'theirScope', and 'mySQLfieldname' and acts as a
crossreference file between the Tagfile and my SQL file example of
TagToSQL this file has a primary index on theirTagname;

"address1","POL1","bnd.addr1"
"address2","POL1","bnd.addr2"
"appartmentnumber","POL1","bnd.apptno"

etc
3. Binder - This is the primary Policy header file and is in MySQL
this file contains information such as;
bnd.policyno
bnd.last
bnd.first
bnd.addr1
bnd.addr2
bnd.city
bnd.state

etc

Now most of my coding experience is in compiled languages such as
cobol, c, assembler etc

I have all of the file access code completed as far as reading through
the CSV file and indexing into my TagToSQL file and writing to my SQL
files the only problem I have is how to create what I believe to be the
couple of lines of code which would allow me to move the date in
'Value' from the Tagfile into my SQL file using the data in the
TagToSQL field mySQLfieldname.

I have done some more reading and I think the code I need is as
follows;

mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
exec mycode

This is very new to me because I don't believe this can be done in a
compiled language or at least not as easily as in an interpeted
language like Python.

I hope this clarifies the problem

Len Sumnler
Unique Insurance

Cameron Laird wrote:
> In article <(E-Mail Removed) om>,
> len <(E-Mail Removed)> wrote:
> >I appoligize I don't think I have done a very good job of explaining my
> >problem.

> .
> .
> .
> >The program I am writing is nothing more than a conversion program to
> >take the value out of the CSV file and map it into the appropriate
> >field in my SQL files. Rather than creating some huge if than else
> >(there are over 1000 tagnames) I created the xreffile.
> >
> >Now when I read a record from the tagfile I use the data in the tagname
> >field to lookup the tagname in my xreffile. The data in the
> >SQL_fieldname is the fieldname in my SQL files I want to place the data
> >from the tagfile in the tagfile.value field into this field in my SQL
> >files;
> >
> >data referenced by(xreffile.SQL_fieldname) = tagfile.value
> >
> >what I see as the problem is I want to use what is the data reference
> >by xreffile.SQL.fieldname and now make it part of the python code as a
> >reference variable in an assignement code statement.

> .
> .
> .
> 1. Take Daniel Wong's advice, elsewhere in this thread,
> and use the Python CSV module.
> 2. "what I see as the problem is I want ...": what you
> want *is* rather a problem, because it's a troublesome
> way to achieve what I understand to be your larger
> aims. It was certainly good that you didn't create
> "some huge if than else".
>
> Once you have an SQL_fieldname, and a tagfile.value,
> what do you want to do? Are you stuffing data into
> an SQL table? Continuing on with Python computations?
> In almost any case, it sounds as though you will profit
> greatly from study of Python's dictionaries <URL:
> http://www.developer.com/lang/other/article.php/630721 >
> <URL: http://www.diveintopython.org/gettin...tionaries.html >.


 
Reply With Quote
 
Marc 'BlackJack' Rintsch
Guest
Posts: n/a
 
      08-11-2006
In <(E-Mail Removed) .com>, len wrote:

> mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
> exec mycode


Why do you use ``exec`` here? Why not simply::

TagToSQL['mySQLfieldname'] = Tagfile['Value']

Ciao,
Marc 'BlackJack' Rintsch
 
Reply With Quote
 
Cameron Laird
Guest
Posts: n/a
 
      08-12-2006
In article <(E-Mail Removed) .com>,
len <(E-Mail Removed)> wrote:
.
.
.
>I have done some more reading and I think the code I need is as
>follows;
>
>mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
>exec mycode
>
>This is very new to me because I don't believe this can be done in a
>compiled language or at least not as easily as in an interpeted
>language like Python.
>
>I hope this clarifies the problem

.
.
.
I don't understand how

TagToSQL[mySQLfieldname] = Tagfile[Value]

fails to meet requirements that

mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
exec mycode

satisfies.

This thread confuses me. Maybe you already have all the answers
you seek. If not, I recommend that you simplify--perhaps work
through a single example datum in detail. In the meantime, I
applaud your judgment that you can achieve what you're after with
table lookups and such rather than the thousand-way if-else at
which you hinted at least once.
 
Reply With Quote
 
len
Guest
Posts: n/a
 
      08-13-2006
Sample and test code shows you are correct.

tpsFile - is really the SQL file I will be inserting new policy records
into

tagFile - is a CVS file containing all of the information for a new
policy in an XMLish fashion (one record per filed of the policy) I will
receive from a third party

tagIdxFile - is just a file that where the data from the tagFile should
be mapped into the tpsFile

CODE

tpsFile = {'tpsFirstName' : 'Kate', 'tpsLastName' : 'Sumner',
'tpsPhone': '532-1234'}
tagFile = {'tagname' : 'tagFirst', 'tagScope' : 'POL0', 'tagValue' :
'Rose'}
tagIdxFile = {'idxtagname' : 'tagFirst', 'idxtpsname' : 'tpsFirstName'}
print tpsFile['tpsFirstName']
tpsFile[tagIdxFile['idxtpsname']] = tagFile['tagValue']
print tpsFile['tpsFirstName']

RESULTS

>>>

Kate
Rose
>>>


Just a small note: As trivial as this may seem this task was not
possible in the compiled language I work in due to the fact that there
was no way for me to get the data referenced by
tagIdxFile['idxtpsname'] and then use it as a field label on the left
side of the assignment statement because in the compiled language the
left side must be a label and NOT and expression.

Just strengthens my commitment to learn Python.


I would like to thank every one for their help, advice and patients.

Len Sumnler
Cameron Laird wrote:
> In article <(E-Mail Removed) .com>,
> len <(E-Mail Removed)> wrote:
> .
> .
> .
> >I have done some more reading and I think the code I need is as
> >follows;
> >
> >mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
> >exec mycode
> >
> >This is very new to me because I don't believe this can be done in a
> >compiled language or at least not as easily as in an interpeted
> >language like Python.
> >
> >I hope this clarifies the problem

> .
> .
> .
> I don't understand how
>
> TagToSQL[mySQLfieldname] = Tagfile[Value]
>
> fails to meet requirements that
>
> mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
> exec mycode
>
> satisfies.
>
> This thread confuses me. Maybe you already have all the answers
> you seek. If not, I recommend that you simplify--perhaps work
> through a single example datum in detail. In the meantime, I
> applaud your judgment that you can achieve what you're after with
> table lookups and such rather than the thousand-way if-else at
> which you hinted at least once.


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
question row filter (more of sql query question) =?Utf-8?B?YW5kcmV3MDA3?= ASP .Net 2 10-06-2005 01:07 PM
Quick Question - Newby Question =?Utf-8?B?UnlhbiBTbWl0aA==?= ASP .Net 4 02-16-2005 11:59 AM
Question on Transcender Question :-) eddiec MCSE 6 05-20-2004 06:59 AM
Question re: features of the 831 router (also a 924 question) Wayne Cisco 0 03-02-2004 07:57 PM
Syntax Question - Novice Question sean ASP .Net 1 10-20-2003 12:18 PM



Advertisments