Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > creating tar file and streaming it over HTTP?

Reply
Thread Tools

creating tar file and streaming it over HTTP?

 
 
pbienst
Guest
Posts: n/a
 
      01-06-2010
I would like to bundle up a number of files in a tar file and send it
over a HTTP connection, but I would like to do this without creating
the tar file on disk first.

I know I can get tarfile to output to a stream by doing something like

tar_pipe = tarfile.open(mode="w|", fileobj=my_file_obj)

However, I can't figure out which file object to use to send this over
a HTTP connection.

I tried

conn = httplib.HTTPConnection(hostname, port)
conn.putrequest("PUT", "/client/files")
conn.endheaders()
tar_pipe = tarfile.open(mode="w|", fileobj=conn.sock.makefile())
for filename in filenames:
tar_pipe.add(filename)
tar_pipe.close()
conn.getresponse()

but that does not send any data...

 
Reply With Quote
 
 
 
 
Steve Holden
Guest
Posts: n/a
 
      01-06-2010
pbienst wrote:
> I would like to bundle up a number of files in a tar file and send it
> over a HTTP connection, but I would like to do this without creating
> the tar file on disk first.
>
> I know I can get tarfile to output to a stream by doing something like
>
> tar_pipe = tarfile.open(mode="w|", fileobj=my_file_obj)
>
> However, I can't figure out which file object to use to send this over
> a HTTP connection.
>
> I tried
>
> conn = httplib.HTTPConnection(hostname, port)
> conn.putrequest("PUT", "/client/files")
> conn.endheaders()
> tar_pipe = tarfile.open(mode="w|", fileobj=conn.sock.makefile())
> for filename in filenames:
> tar_pipe.add(filename)
> tar_pipe.close()
> conn.getresponse()
>
> but that does not send any data...
>

I haven't used socket.makefile() in a blue age so this is untested, but
I'm guessing you need a *write* file (remember, sockets have two
directions). So try

.... fileobj=conn.sock.makefile("w")...

and let me know if that helps.

regards
Steve

--
Steve Holden +1 571 484 6266 +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS: http://holdenweb.eventbrite.com/

 
Reply With Quote
 
 
 
 
pbienst
Guest
Posts: n/a
 
      01-06-2010
Thanks for the tip! It doesn't change anything, though, so I've
debugged this a little bit further. The problem seems to be that the
receiving end (wsgi server) does not see the end of the data:

socket = environ["wsgi.input"]
while True:
sys.stderr.write("before")
chunk = socket.read(4096)
sys.stderr.write("after")
if not chunk:
sys.stderr.write("done")
break
sys.stderr.write(chunk)

There is data from the tar file being printed, but in the end it hangs
in the 'before' statement, and never gets to 'done'. I tried flushing
the fileobj created by makefile(), but that doesn't seem to help.
There is also no environ["CONTENT_LENGTH"] that I can use to detect
the end of the stream.

I'm probably missing something basic here...
 
Reply With Quote
 
r0g
Guest
Posts: n/a
 
      01-06-2010
pbienst wrote:
> I would like to bundle up a number of files in a tar file and send it
> over a HTTP connection, but I would like to do this without creating
> the tar file on disk first.
>


Stringio lets you treat a strings as a files...

http://docs.python.org/library/stringio.html

Roger.
 
Reply With Quote
 
Steve Holden
Guest
Posts: n/a
 
      01-06-2010
r0g wrote:
> pbienst wrote:
>> I would like to bundle up a number of files in a tar file and send it
>> over a HTTP connection, but I would like to do this without creating
>> the tar file on disk first.
>>

>
> Stringio lets you treat a strings as a files...
>
> http://docs.python.org/library/stringio.html
>
> Roger.


.... though that does mean that the whole tar file has to be created
before it can be sent, I suspect. If it will comfortably fit into memory
that shouldn't matter, of course.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS: http://holdenweb.eventbrite.com/
 
Reply With Quote
 
Steve Holden
Guest
Posts: n/a
 
      01-06-2010
r0g wrote:
> pbienst wrote:
>> I would like to bundle up a number of files in a tar file and send it
>> over a HTTP connection, but I would like to do this without creating
>> the tar file on disk first.
>>

>
> Stringio lets you treat a strings as a files...
>
> http://docs.python.org/library/stringio.html
>
> Roger.


.... though that does mean that the whole tar file has to be created
before it can be sent, I suspect. If it will comfortably fit into memory
that shouldn't matter, of course.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS: http://holdenweb.eventbrite.com/

 
Reply With Quote
 
Gabriel Genellina
Guest
Posts: n/a
 
      01-07-2010
En Wed, 06 Jan 2010 13:39:08 -0300, pbienst <(E-Mail Removed)>
escribió:

> The problem seems to be that the
> receiving end (wsgi server) does not see the end of the data:
>
> socket = environ["wsgi.input"]
> while True:
> sys.stderr.write("before")
> chunk = socket.read(4096)
> sys.stderr.write("after")
> if not chunk:
> sys.stderr.write("done")
> break
> sys.stderr.write(chunk)
>
> There is data from the tar file being printed, but in the end it hangs
> in the 'before' statement, and never gets to 'done'. I tried flushing
> the fileobj created by makefile(), but that doesn't seem to help.
> There is also no environ["CONTENT_LENGTH"] that I can use to detect
> the end of the stream.
>
> I'm probably missing something basic here...


Either send a "Connection: Close" header (and close the connection at the
end), or a "Transfer-Encoding: chunked" header (see
http://en.wikipedia.org/wiki/Chunked_transfer_encoding )

--
Gabriel Genellina

 
Reply With Quote
 
pbienst
Guest
Posts: n/a
 
      01-09-2010
OK, thanks to the feedback from everyone I got the PUT from a client
to the WSGI server working.

I'm now trying to go the other way around: use a tar stream in one of
the functions in the WSGI server in order to send files to the client.
Problem is that the WSGI specs expects an iterator as return value for
streaming, whereas TarFile needs to write to a file obj.

Is there any way I can get these two to work together?

Peter
 
Reply With Quote
 
Gabriel Genellina
Guest
Posts: n/a
 
      01-13-2010
En Sat, 09 Jan 2010 10:19:00 -0300, pbienst <(E-Mail Removed)>
escribió:

> OK, thanks to the feedback from everyone I got the PUT from a client
> to the WSGI server working.
>
> I'm now trying to go the other way around: use a tar stream in one of
> the functions in the WSGI server in order to send files to the client.
> Problem is that the WSGI specs expects an iterator as return value for
> streaming, whereas TarFile needs to write to a file obj.
>
> Is there any way I can get these two to work together?


I haven't actually tried it, but aren't you looking for a pipe? (os.pipe()
+ os.fdopen())

--
Gabriel Genellina

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Pipelining tar create and tar extract the "Python" way... Ray Van Dolson Python 0 09-25-2009 03:48 PM
Pipelining tar create and tar extract the "Python" way... Ray Van Dolson Python 0 09-23-2009 10:52 PM
os.system('tar -c * | tar -C dst') ##Any other suggestions... list.repository@gmail.com Python 2 04-24-2007 10:29 PM
Version of TAR in tarfile module? TAR 1.14 or 1.15 port to Windows? Claudio Grondi Python 4 08-20-2005 08:01 PM



Advertisments