«

Is it possible to write a file open, then read program in C and then
call the C program from a Python script like this:

for root, files, dirs in os.walk(path)
for f in files:
try:
EXECUTE_C_PROGRAM

If possible, how much faster would this be over a pure Python solution?

Thank you,

Brad

On 2004-12-09, Brad Tilley <> wrote:

> Is it possible to write a file open, then read program in C and then
> call the C program from a Python script like this:

Huh? What do you mean "write a file open"? You want to read a
C source file and execute the C source? If you have access to
a C interpreter, I guess you could invoke the interpreter from
python using popen, and feed the C source to it. Alternatively
you could invoke a compiler and linker from C to generate an
executable and then execute the resulting file.

> for root, files, dirs in os.walk(path)
> for f in files:
> try:
> EXECUTE_C_PROGRAM

You're going to have to explain clearly what you mean by
"EXECUTE_C_PROGRAM". If you want to, you can certainly run a
binary executable that was generated from C source, (e.g. an
ELF file under Linux or whatever a .exe file is under Windows).

> If possible, how much faster would this be over a pure Python
> solution?

Solution to what?

--
Grant Edwards grante Yow! And furthermore,
at my bowling average is
visi.com unimpeachable!!!

Brad Tilley wrote:

> If possible, how much faster would this be over a pure Python solution?

It is like the difference between Batman and Ever.

batman is faster than ever

Grant Edwards wrote:
> Huh? What do you mean "write a file open"? You want to read a
> C source file and execute the C source? If you have access to
> a C interpreter, I guess you could invoke the interpreter from
> python using popen, and feed the C source to it. Alternatively
> you could invoke a compiler and linker from C to generate an
> executable and then execute the resulting file.
>
>
>>for root, files, dirs in os.walk(path)
>> for f in files:
>> try:
>> EXECUTE_C_PROGRAM
>
>
> You're going to have to explain clearly what you mean by
> "EXECUTE_C_PROGRAM". If you want to, you can certainly run a
> binary executable that was generated from C source, (e.g. an
> ELF file under Linux or whatever a .exe file is under Windows).

Appears I was finger-tied. I meant "a C program that opens and reads
files" while Python does everything else. How does one integrate C into
a Python script like that?

So, instead of this:

for root, files, dirs in os.walk(path)
for f in files:
try:
x = file(f, 'rb')
data = x.read()
x.close()
this:

for root, files, dirs in os.walk(path)
for f in files:
try:
EXECUTE_C_PROGRAM

From the Simpsons:
Frink: "Here we have an ordinary square."
Wiggum: "Whoa! Slow down egghead!"

I would expect C to run circles around the same operation under Python. As
a general rule of thumb, you should use C for time cirtical operations
(computer time, that is), and use Python for human time critical situations
(you can get a program developed much faster).

I just discovered a magical package call SWIG (http://www.swig.org) that
makes writing C wrappers for Python always a child's play. It's incredible!
Where were these guys years ago when I had to pay somebody moocho money to
develop a script language wrapper for my application!!!

--
It's me


"Brad Tilley" <> wrote in message
news:cpa5l5$aua$...
> Grant Edwards wrote:
> > Huh? What do you mean "write a file open"? You want to read a
> > C source file and execute the C source? If you have access to
> > a C interpreter, I guess you could invoke the interpreter from
> > python using popen, and feed the C source to it. Alternatively
> > you could invoke a compiler and linker from C to generate an
> > executable and then execute the resulting file.
> >
> >
> >>for root, files, dirs in os.walk(path)
> >> for f in files:
> >> try:
> >> EXECUTE_C_PROGRAM
> >
> >
> > You're going to have to explain clearly what you mean by
> > "EXECUTE_C_PROGRAM". If you want to, you can certainly run a
> > binary executable that was generated from C source, (e.g. an
> > ELF file under Linux or whatever a .exe file is under Windows).
>
> Appears I was finger-tied. I meant "a C program that opens and reads
> files" while Python does everything else. How does one integrate C into
> a Python script like that?
>
> So, instead of this:
>
> for root, files, dirs in os.walk(path)
> for f in files:
> try:
> x = file(f, 'rb')
> data = x.read()
> x.close()
> this:
>
> for root, files, dirs in os.walk(path)
> for f in files:
> try:
> EXECUTE_C_PROGRAM
>
> From the Simpsons:
> Frink: "Here we have an ordinary square."
> Wiggum: "Whoa! Slow down egghead!"

Brad Tilley wrote:

> for root, files, dirs in os.walk(path)
> for f in files:
> try:
> EXECUTE_C_PROGRAM

http://docs.python.org/lib/module-subprocess.html

this module in new in 2.4; for older version, os.system() or the os.popen()
family might be what you're looking for. see the "Replacing Older Functions
with the subprocess Module" section for more info.

</F>

It's me wrote:
> I would expect C to run circles around the same operation under Python.

You should probably only expect C to run circles around the same
operations when those operations implemented entirely in Python. In the
specific (trivial) example given, I wouldn't expect Python to be much
slower:

>>for root, files, dirs in os.walk(path)
>> for f in files:
>> try:
>> x = file(f, 'rb')
>> data = x.read()
>> x.close()

Remember that CPython is implemented in C, and so all the builtin types
(including file) basically execute C code directly. My experience with
Python file objects is that they are quite fast when you're doing simple
things like the example above. (In fact, I usually find that Python is
faster than Java for things like this.)

Of course, the example above is almost certainly omitting some code that
really gets executed, and without knowing what that code does, it would
be difficult to predict exactly what performance gain you would get from
reimplementing it in C. Profile the app first, find out where the tight
spots are, and then reimplement in C if necessary (often, it isn't).

STeve

Steven Bethard wrote:
>>> for root, files, dirs in os.walk(path)
>>> for f in files:
>>> try:
>>> x = file(f, 'rb')
>>> data = x.read()
>>> x.close()
>
>
> Remember that CPython is implemented in C, and so all the builtin types
> (including file) basically execute C code directly. My experience with
> Python file objects is that they are quite fast when you're doing simple
> things like the example above.

I'm dealing with a terabyte of files. Perhaps I should have mentioned that.

On 2004-12-09, Brad Tilley <> wrote:

>> You're going to have to explain clearly what you mean by
>> "EXECUTE_C_PROGRAM". If you want to, you can certainly run a
>> binary executable that was generated from C source, (e.g. an
>> ELF file under Linux or whatever a .exe file is under
>> Windows).
>
> Appears I was finger-tied. I meant "a C program that opens and
> reads files"

That's still too vague to be meaningful. Just reading a file
seems pointless:

cat foo >/dev/null

Here, "cat" is a C program that "opens and reads" the file
named foo.

> while Python does everything else. How does one
> integrate C into a Python script like that?
>
> So, instead of this:
>
> for root, files, dirs in os.walk(path)
> for f in files:
> try:
> x = file(f, 'rb')
> data = x.read()
> x.close()
> this:
>
> for root, files, dirs in os.walk(path)
> for f in files:
> try:
> EXECUTE_C_PROGRAM

So you want the data returned to your Python program? If so,
you can't just execute a C program. If you want to use the
data in the file, you have to read the data from _somewhere_.

You can read it directly from the file, or you can read it from
a pipe, where it was put by the program that read it from the
file. The former is going to be far, far faster.

--
Grant Edwards grante Yow! Am I SHOPLIFTING?
at
visi.com

On 2004-12-09, Brad Tilley <> wrote:
> Steven Bethard wrote:
>>>> for root, files, dirs in os.walk(path)
>>>> for f in files:
>>>> try:
>>>> x = file(f, 'rb')
>>>> data = x.read()
>>>> x.close()
>>
>>
>> Remember that CPython is implemented in C, and so all the builtin types
>> (including file) basically execute C code directly. My experience with
>> Python file objects is that they are quite fast when you're doing simple
>> things like the example above.
>
> I'm dealing with a terabyte of files. Perhaps I should have mentioned that.

And you think you're going to read the entire file consisting
of terabytes of data into memory using either C or Python?
[That's the example you gave.]

Sounds like maybe you need to mmap() the files?

Or at least tell us what you're trying to do so we can make
more intelligent suggestions.

--
Grant Edwards grante Yow! I invented skydiving
at in 1989!
visi.com