Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Is it possible to pass CSV Reader Object As Argument to anotherPython File ???

Reply
Thread Tools

Is it possible to pass CSV Reader Object As Argument to anotherPython File ???

 
 
bansi
Guest
Posts: n/a
 
      01-26-2011
I have following two python scripts
-namelookupWrapper.py
-namelookup.py


The namelookupWrapper.py takes input of "memberId", "memberName" from
CLI and has following code snippet

idf = sys.argv[1]
namef = sys.argv[2]
real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py "
r = csv.reader(sys.stdin)
os.execv(python_executable, [ python_executable, real_script ] +
sys.argv[1:] )



Wondering how would i pass csv reader object "r" as an argument using
os.execv() to another python script i.e. namelookup.py


 
Reply With Quote
 
 
 
 
Emile van Sebille
Guest
Posts: n/a
 
      01-26-2011
On 1/26/2011 7:51 AM bansi said...
> I have following two python scripts
> -namelookupWrapper.py
> -namelookup.py
>
>
> The namelookupWrapper.py takes input of "memberId", "memberName" from
> CLI and has following code snippet
>
> idf = sys.argv[1]
> namef = sys.argv[2]
> real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py "
> r = csv.reader(sys.stdin)
> os.execv(python_executable, [ python_executable, real_script ] +
> sys.argv[1:] )
>
>
>
> Wondering how would i pass csv reader object "r" as an argument using
> os.execv() to another python script i.e. namelookup.py
>



I suspect you're on the wrong path. You probably want to import
namelookup within namelooupWrapper to use the functions it defines.

Consider:

[root@fcfw2 src]# cat > test1.py

def say(what): print what

[root@fcfw2 src]# cat > test2.py

#!/usr/local/bin/python
import sys
from test1 import say
say(sys.argv[1])

[root@fcfw2 src]# chmod a+x test2.py

[root@fcfw2 src]# ./test2.py hello
hello


HTH,

Emile



 
Reply With Quote
 
 
 
 
bansi
Guest
Posts: n/a
 
      01-26-2011
On Jan 26, 11:30*am, Emile van Sebille <(E-Mail Removed)> wrote:
> On 1/26/2011 7:51 AM bansi said...
>
>
>
>
>
> > I have following two python scripts
> > -namelookupWrapper.py
> > -namelookup.py

>
> > The namelookupWrapper.py takes input of "memberId", "memberName" from
> > CLI and has following code snippet

>
> > idf = sys.argv[1]
> > namef = sys.argv[2]
> > real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py "
> > r = csv.reader(sys.stdin)
> > os.execv(python_executable, [ python_executable, real_script ] +
> > sys.argv[1:] )

>
> > Wondering how would i pass csv reader object "r" as an argument using
> > os.execv() to another python script i.e. namelookup.py

>
> I suspect you're on the wrong path. *You probably want to import
> namelookup within namelooupWrapper to use the functions it defines.
>
> Consider:
>
> [root@fcfw2 src]# cat > test1.py
>
> def say(what): print what
>
> [root@fcfw2 src]# cat > test2.py
>
> #!/usr/local/bin/python
> import sys
> from test1 import say
> say(sys.argv[1])
>
> [root@fcfw2 src]# chmod a+x test2.py
>
> [root@fcfw2 src]# ./test2.py hello
> hello
>
> HTH,
>
> Emile- Hide quoted text -
>
> - Show quoted text -


Emile,
Thanks for quick response. I am not sure if "import
namelookup within namelooupWrapper" helps because they are two
independent scripts which has to be executed in sequence.
First namelookupWrapper.py running under Python 2.6 accept arguments
from stdin and uses csv reader object to read it i.e.
r=csv.reader(sys.stdin)

And then it has to pass csv reader object to another python script
namelookup.py running under Python 2.7 because it uses pyodbc to
connect to database and iterates thru reader object

Any better ideas/suggestions will be greatly appreciated
 
Reply With Quote
 
Chris Rebert
Guest
Posts: n/a
 
      01-26-2011
On Wed, Jan 26, 2011 at 7:51 AM, bansi <(E-Mail Removed)> wrote:
> I have following two python scripts
> -namelookupWrapper.py
> -namelookup.py
>
>
> The namelookupWrapper.py takes input of "memberId", "memberName" from
> CLI and has following code snippet
>
> idf = sys.argv[1]
> namef = sys.argv[2]
> real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py "
> r = csv.reader(sys.stdin)
> os.execv(python_executable, [ python_executable, real_script ] +
> sys.argv[1:] )
>
> Wondering how would i pass csv reader object "r" as an argument using
> os.execv() to another python script i.e. namelookup.py


It's not possible to pass Python objects between processes in such a
manner. Given that "independent" scripts can't directly take objects
as input anyway, I doubt the two scripts are truly independent from
each other. I would therefore concur with van Sebille that you should
just rewrite them so that one script imports from the other rather
than spawning the other. It should not be too hard to port the Python
2.6 script to Python 2.7 (or vice-versa if necessary).

Cheers,
Chris
--
http://blog.rebertia.com
 
Reply With Quote
 
bansi
Guest
Posts: n/a
 
      01-26-2011
On Jan 26, 1:31*pm, Chris Rebert <(E-Mail Removed)> wrote:
> On Wed, Jan 26, 2011 at 7:51 AM, bansi <(E-Mail Removed)> wrote:
> > I have following two python scripts
> > -namelookupWrapper.py
> > -namelookup.py

>
> > The namelookupWrapper.py takes input of "memberId", "memberName" from
> > CLI and has following code snippet

>
> > idf = sys.argv[1]
> > namef = sys.argv[2]
> > real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py "
> > r = csv.reader(sys.stdin)
> > os.execv(python_executable, [ python_executable, real_script ] +
> > sys.argv[1:] )

>
> > Wondering how would i pass csv reader object "r" as an argument using
> > os.execv() to another python script i.e. namelookup.py

>
> It's not possible to pass Python objects between processes in such a
> manner. Given that "independent" scripts can't directly take objects
> as input anyway, I doubt the two scripts are truly independent from
> each other. I would therefore concur with van Sebille that you should
> just rewrite them so that one script imports from the other rather
> than spawning the other. It should not be too hard to port the Python
> 2.6 script to Python 2.7 (or vice-versa if necessary).
>
> Cheers,
> Chris
> --http://blog.rebertia.com- Hide quoted text -
>
> - Show quoted text -


Thanks Chris. Sorry for mis-communicating, the two python scripts are
dependant in a way that
namelookupWrapper.py needs to pass csv record object to another
python script

If thats not possible then please let me know how to do the workaround
i didnt understood the import thing and not sure if it helps in my
case

Here are the details
namelookupwrapper.py - takes input from stdin. Using csv reader object
i iterate thru the input which looks like as shown below

[MemberId, MemberName]
[123, ]
[456, ]
[989, ]


Now i have another script i.e. namelookup.py running under Python 2.7
using pyodbc to retrieve Member Names from database for a given Member
Id in namelooupWrapper.py

So please let me know how to accomplish this
 
Reply With Quote
 
bansi
Guest
Posts: n/a
 
      01-26-2011
On Jan 26, 4:36*pm, Ben Finney <(E-Mail Removed)> wrote:
> bansi <(E-Mail Removed)> writes:
> > Thanks Chris. Sorry for mis-communicating, the two python scripts are
> > dependant in a way that namelookupWrapper.py needs to pass csv record
> > object to another python script

>
> Why have you structured them that way, though? What constraint is
> keeping you from doing the work in a single process, where the CSV
> reader object can be shared?
>
> > If thats not possible then please let me know how to do the workaround
> > i didnt understood the import thing and not sure if it helps in my
> > case

>
> The problem as you've described it so far is best solved by having a
> single process accessing the CSV reader object in memory. If that
> doesn't suit your use case, you'll need to explain why not.
>
> --
> *\ * * * “To have the choice between proprietary software packages, is |
> * `\ * * *being able to choose your master. Freedom means not having a |
> _o__) * * * * * * * * * * * *master.” —Richard M. Stallman, 2007-05-16 |
> Ben Finney


Thanks Ben for quick response. The constraint is in using third party
tool called Splunk which has in-built Python 2.6 which doesnt support
pyodbc on Windows 64 bit OS. Hence i have to install Python 2.7 for
pyodbc.
That means namelookupwrapper.py acts as a wrapper running under
Splunk environment taking input from Splunk via stdin and storing it
in csv reader object and then needs to call actual script
namelookup.py under Python 2.7 for making connection to database

Hope i have clarified a bit
 
Reply With Quote
 
Ethan Furman
Guest
Posts: n/a
 
      01-26-2011
bansi wrote:
> First namelookupWrapper.py running under Python 2.6 accept arguments
> from stdin and uses csv reader object to read it i.e.
> r=csv.reader(sys.stdin)
>
> And then it has to pass csv reader object to another python script
> namelookup.py running under Python 2.7 because it uses pyodbc to
> connect to database and iterates thru reader object


Ben Finney wrote:
> bansi <(E-Mail Removed)> writes:
>
>> Thanks Chris. Sorry for mis-communicating, the two python scripts are
>> dependant in a way that namelookupWrapper.py needs to pass csv record
>> object to another python script

>
> Why have you structured them that way, though? What constraint is
> keeping you from doing the work in a single process, where the CSV
> reader object can be shared?
>
>> If thats not possible then please let me know how to do the workaround
>> i didnt understood the import thing and not sure if it helps in my
>> case

>
> The problem as you've described it so far is best solved by having a
> single process accessing the CSV reader object in memory. If that
> doesn't suit your use case, you'll need to explain why not.
>



In other words, why can't you use Python 2.7 to accept input and
generate a csv.reader?

~Ethan~
 
Reply With Quote
 
bansi
Guest
Posts: n/a
 
      01-27-2011
On Jan 26, 6:25*pm, Ethan Furman <(E-Mail Removed)> wrote:
> bansi wrote:
>
> *> First namelookupWrapper.py running under Python 2.6 accept arguments
> *> from stdin and uses csv reader object to read it i.e.
> *> r=csv.reader(sys.stdin)
> *>
> *> And then it has to pass csv reader object to another python script
> *> namelookup.py running under Python 2.7 because it uses pyodbc to
> *> connect to database and iterates thru reader object
>
>
>
>
>
> Ben Finney wrote:
> > bansi <(E-Mail Removed)> writes:

>
> >> Thanks Chris. Sorry for mis-communicating, the two python scripts are
> >> dependant in a way that namelookupWrapper.py needs to pass csv record
> >> object to another python script

>
> > Why have you structured them that way, though? What constraint is
> > keeping you from doing the work in a single process, where the CSV
> > reader object can be shared?

>
> >> If thats not possible then please let me know how to do the workaround
> >> i didnt understood the import thing and not sure if it helps in my
> >> case

>
> > The problem as you've described it so far is best solved by having a
> > single process accessing the CSV reader object in memory. If that
> > doesn't suit your use case, you'll need to explain why not.

>
> In other words, why can't you use Python 2.7 to accept input and
> generate a csv.reader?
>
> ~Ethan~- Hide quoted text -
>
> - Show quoted text -


Ethan,
The python script takes the input from Splunk (http://www.splunk.com/
base/Documentation/) which supports only Python 2.6
So the real constraint is Splunk supports only Python 2.6 .

As you know Python 2.6 doesnt support or doesnt have pyodbc install
for Windows 64 bit OS
So i installed Python 2.7 and thereafter pyodbc install for Windows 64
bit OS for Python 2.7
 
Reply With Quote
 
MRAB
Guest
Posts: n/a
 
      01-27-2011
On 27/01/2011 00:57, bansi wrote:
> On Jan 26, 6:25 pm, Ethan Furman<(E-Mail Removed)> wrote:
>> bansi wrote:
>>
>> > First namelookupWrapper.py running under Python 2.6 accept arguments
>> > from stdin and uses csv reader object to read it i.e.
>> > r=csv.reader(sys.stdin)
>> >
>> > And then it has to pass csv reader object to another python script
>> > namelookup.py running under Python 2.7 because it uses pyodbc to
>> > connect to database and iterates thru reader object

>>
>>
>>
>>
>>
>> Ben Finney wrote:
>>> bansi<(E-Mail Removed)> writes:

>>
>>>> Thanks Chris. Sorry for mis-communicating, the two python scripts are
>>>> dependant in a way that namelookupWrapper.py needs to pass csv record
>>>> object to another python script

>>
>>> Why have you structured them that way, though? What constraint is
>>> keeping you from doing the work in a single process, where the CSV
>>> reader object can be shared?

>>
>>>> If thats not possible then please let me know how to do the workaround
>>>> i didnt understood the import thing and not sure if it helps in my
>>>> case

>>
>>> The problem as you've described it so far is best solved by having a
>>> single process accessing the CSV reader object in memory. If that
>>> doesn't suit your use case, you'll need to explain why not.

>>
>> In other words, why can't you use Python 2.7 to accept input and
>> generate a csv.reader?
>>
>> ~Ethan~- Hide quoted text -
>>
>> - Show quoted text -

>
> Ethan,
> The python script takes the input from Splunk (http://www.splunk.com/
> base/Documentation/) which supports only Python 2.6
> So the real constraint is Splunk supports only Python 2.6 .
>
> As you know Python 2.6 doesnt support or doesnt have pyodbc install
> for Windows 64 bit OS
> So i installed Python 2.7 and thereafter pyodbc install for Windows 64
> bit OS for Python 2.7


Have you actually tried Splunk with Python 2.7? It might not work with
versions which are earlier than Python 2.6, but that doesn't
necessarily mean that it won't work with versions of Python 2 which are
later than Python 2.6 (unless the documentation says that it must be
Python 2.6).
 
Reply With Quote
 
bansi
Guest
Posts: n/a
 
      01-28-2011
On Jan 26, 8:31*pm, MRAB <(E-Mail Removed)> wrote:
> On 27/01/2011 00:57, bansi wrote:
>
>
>
>
>
> > On Jan 26, 6:25 pm, Ethan Furman<(E-Mail Removed)> *wrote:
> >> bansi wrote:

>
> >> * > *First namelookupWrapper.py running under Python 2.6 accept arguments
> >> * > *from stdin and uses csv reader object to read it i.e.
> >> * > *r=csv.reader(sys.stdin)

>
> >> * > *And then it has to pass csv reader object to another python script
> >> * > *namelookup.py running under Python 2.7 because it uses pyodbc to
> >> * > *connect to database and iterates thru reader object

>
> >> Ben Finney wrote:
> >>> bansi<(E-Mail Removed)> *writes:

>
> >>>> Thanks Chris. Sorry for mis-communicating, the two python scripts are
> >>>> dependant in a way that namelookupWrapper.py needs to pass csv record
> >>>> object to another python script

>
> >>> Why have you structured them that way, though? What constraint is
> >>> keeping you from doing the work in a single process, where the CSV
> >>> reader object can be shared?

>
> >>>> If thats not possible then please let me know how to do the workaround
> >>>> i didnt understood the import thing and not sure if it helps in my
> >>>> case

>
> >>> The problem as you've described it so far is best solved by having a
> >>> single process accessing the CSV reader object in memory. If that
> >>> doesn't suit your use case, you'll need to explain why not.

>
> >> In other words, why can't you use Python 2.7 to accept input and
> >> generate a csv.reader?

>
> >> ~Ethan~- Hide quoted text -

>
> >> - Show quoted text -

>
> > Ethan,
> > The python script takes the input from Splunk (http://www.splunk.com/
> > base/Documentation/) which supports only Python 2.6
> > So the real constraint is Splunk supports only Python 2.6 .

>
> > As you know Python 2.6 doesnt support or doesnt have pyodbc install
> > for Windows *64 bit OS
> > So i installed Python 2.7 and thereafter pyodbc install for Windows 64
> > bit OS for Python 2.7

>
> Have you actually tried Splunk with Python 2.7? It might not work with
> versions which are earlier than Python 2.6, but that doesn't
> necessarily mean that it won't work with versions of Python 2 which are
> later than Python 2.6 (unless the documentation says that it must be
> Python 2.6).- Hide quoted text -
>
> - Show quoted text -


Splunk's latest version 4.1.6 doesn't support Python 2.7
I tried the import trick but it didnt work because the real script
which runs under Python 2.7 has import pyodbc so it results in
following error

c:\Splunk\etc\apps\search\bin>splunk cmd python namelookupWrapper.py
memberId memberName < memberInput.csv
Traceback (most recent call last):
File "namelookupWrapper.py", line 3, in <module>
import namelookup
File "c:\Splunk\etc\apps\search\bin\namelookup.py", line 7, in
<module>
import pyodbc
ImportError: DLL load failed: The specified module could not be found.

Please let me know if i am missing something on import. If so please
provide me with an example
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
throwing exceptions from csv.DictReader or even csv.reader Tim Python 1 07-05-2010 05:32 PM
read and write csv file using csv module jliu66 Python 0 10-19-2007 03:12 PM
How to move data from a CSV file to a JTable, and from a JTable to a CSV file ? Tintin92 Java 1 02-14-2007 06:51 PM
looking for way to include many times some .py code from anotherpython code =?ISO-8859-2?Q?Martin_MOKREJ=A9?= Python 22 03-09-2005 08:59 AM
How to pass variable argument list to another function w/ variable argument list? Ben Kial C Programming 1 11-15-2004 01:51 AM



Advertisments