Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   regd efficient methods to manipulate *large* files (http://www.velocityreviews.com/forums/t357184-regd-efficient-methods-to-manipulate-large-files.html)

Madhusudhanan Chandrasekaran 05-01-2006 02:28 PM

regd efficient methods to manipulate *large* files
 
Hi:

This question is not directed "entirely" at python only. But since
I want to know how to do it in python, I am posting here.


I am constructing a huge matrix (m x n), whose columns n are stored in
smaller files. Once I read m such files, my matrix is complete. I want to
pass this matrix as an input to another script of mine (I just have the
binary.) Currently, the script reads a file (which is nothing but the
matrix) and processes it. Is there any way of doing this in memory,
without writing the matrix onto the disk?

Since I have to repeat my experimentation for multiple iterations, it
becomes expensive to write the matrix onto the disk.

Thanks in advance. Help appreciated.

-Madhu

Dave Hughes 05-01-2006 08:05 PM

Re: regd efficient methods to manipulate *large* files
 
Madhusudhanan Chandrasekaran wrote:

> Hi:
>
> This question is not directed "entirely" at python only. But since
> I want to know how to do it in python, I am posting here.
>
>
> I am constructing a huge matrix (m x n), whose columns n are stored in
> smaller files. Once I read m such files, my matrix is complete. I
> want to pass this matrix as an input to another script of mine (I
> just have the binary.) Currently, the script reads a file (which is
> nothing but the matrix) and processes it. Is there any way of doing
> this in memory, without writing the matrix onto the disk?
>
> Since I have to repeat my experimentation for multiple iterations, it
> becomes expensive to write the matrix onto the disk.
>
> Thanks in advance. Help appreciated.
>
> -Madhu


Basically, you're asking about Inter Process Communication (IPC), for
which Python provides several interfaces to mechanisms provided by the
operating system (whatever that may be). Here's a couple of commonly
used methods:

Redirected I/O

Have a look at the popen functions in the os module, or better still
the subprocess module (which is a higher level interface to the same
functionality). Specifically, the "Replacing the shell pipe line"
example in the subprocess module's documentation should be interesting:

output=`dmesg | grep hda`
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

Here, the stdout of the "dmesg" process has been redirected to the
stdin of the "grep" process. You could do something similar with your
two scripts: e.g., the first script simply writes the content of the
matrix in some format to stdout (e.g. print, sys.stdout.write), while
the second script reads the content of the matrix from stdin (e.g.
raw_input, sys.stdin.read). Here's some brutally simplistic scripts
that demonstrate the method:

in.py
=====
#!/bin/env python
#
# I read integers from stdin until I encounter 0

import sys

while True:
i = int(sys.stdin.readline())
print "Read %d from stdin" % i
if i == 0:
break


out.py
======
#!/bin/env python
#
# I write some numbers to stdout

for i in [1, 2, 3, 4, 5, 0]:
print i


run.py
======
#!/bin/env python
#
# I run out.py and in.py with a pipe between them, capture the
# output of in.py and print it

from subprocess import Popen, PIPE

process1 = Popen(["./out.py"], stdout=PIPE)
process2 = Popen(["./in.py"], stdin=process1.stdout, stdout=PIPE)
output = process2.communicate()[0]

print output


Sockets

Another form of IPC uses sockets to communicate between two processes
(see the socket module or one of the higher level modules like
SocketServer). Hence, the second process would listen on a port
(presumably on the localhost interface, although there's no reason it
couldn't listen on a LAN interface for example), and the first process
connects to that port and sends the matrix data across it to the second
process.


Summary

Given that your second script currently reads a file containing the
complete matrix (if I understand your post correctly), it's probably
easiest for you to use the Redirected I/O method (as it's very similar
to reading a file, although there are some differences, and sometimes
one must be careful about closing pipe ends to avoid deadlocks).
However, the sockets method has the advantage that you can easily move
one of the processes onto a different machine.

There are other methods of IPC (for example, shared memory: see the
mmap module) however the two mentioned above are available on most
platforms whereas others may be specific to a given platform, or have
platform specific subtleties (for example, mmap is only available on
Windows and UNIX, and has a slightly different constructor on each).


HTH,

Dave.

--


Paddy 05-01-2006 09:19 PM

Re: regd efficient methods to manipulate *large* files
 
I take it that you have a binary file that takes a file name and
proceses the file contents.
Sometimes Unix binaries are written so that a file name of '-', (just a
dash), causes it to take input from stdin so that the piping mentioned
in a previous reply could work.
On some of our unix systems /tmp is set up as a 'virtual disk' It
quacks like a normal disk filesystem but is actually implimented in
RAM/virtual memory, and is faster than normal disk access.
(Unfortunately we are not allowed to save multi-gigabyte files there as
it affects other aspects of the OS).

Maybe you can mount a similar filesystem if you have the RAM.

-- Pad.



All times are GMT. The time now is 01:05 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.