Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   creating tar file and streaming it over HTTP? (http://www.velocityreviews.com/forums/t710862-creating-tar-file-and-streaming-it-over-http.html)

pbienst 01-06-2010 12:26 PM

creating tar file and streaming it over HTTP?
 
I would like to bundle up a number of files in a tar file and send it
over a HTTP connection, but I would like to do this without creating
the tar file on disk first.

I know I can get tarfile to output to a stream by doing something like

tar_pipe = tarfile.open(mode="w|", fileobj=my_file_obj)

However, I can't figure out which file object to use to send this over
a HTTP connection.

I tried

conn = httplib.HTTPConnection(hostname, port)
conn.putrequest("PUT", "/client/files")
conn.endheaders()
tar_pipe = tarfile.open(mode="w|", fileobj=conn.sock.makefile())
for filename in filenames:
tar_pipe.add(filename)
tar_pipe.close()
conn.getresponse()

but that does not send any data...


Steve Holden 01-06-2010 01:27 PM

Re: creating tar file and streaming it over HTTP?
 
pbienst wrote:
> I would like to bundle up a number of files in a tar file and send it
> over a HTTP connection, but I would like to do this without creating
> the tar file on disk first.
>
> I know I can get tarfile to output to a stream by doing something like
>
> tar_pipe = tarfile.open(mode="w|", fileobj=my_file_obj)
>
> However, I can't figure out which file object to use to send this over
> a HTTP connection.
>
> I tried
>
> conn = httplib.HTTPConnection(hostname, port)
> conn.putrequest("PUT", "/client/files")
> conn.endheaders()
> tar_pipe = tarfile.open(mode="w|", fileobj=conn.sock.makefile())
> for filename in filenames:
> tar_pipe.add(filename)
> tar_pipe.close()
> conn.getresponse()
>
> but that does not send any data...
>

I haven't used socket.makefile() in a blue age so this is untested, but
I'm guessing you need a *write* file (remember, sockets have two
directions). So try

.... fileobj=conn.sock.makefile("w")...

and let me know if that helps.

regards
Steve

--
Steve Holden +1 571 484 6266 +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS: http://holdenweb.eventbrite.com/


pbienst 01-06-2010 04:39 PM

Re: creating tar file and streaming it over HTTP?
 
Thanks for the tip! It doesn't change anything, though, so I've
debugged this a little bit further. The problem seems to be that the
receiving end (wsgi server) does not see the end of the data:

socket = environ["wsgi.input"]
while True:
sys.stderr.write("before")
chunk = socket.read(4096)
sys.stderr.write("after")
if not chunk:
sys.stderr.write("done")
break
sys.stderr.write(chunk)

There is data from the tar file being printed, but in the end it hangs
in the 'before' statement, and never gets to 'done'. I tried flushing
the fileobj created by makefile(), but that doesn't seem to help.
There is also no environ["CONTENT_LENGTH"] that I can use to detect
the end of the stream.

I'm probably missing something basic here...

r0g 01-06-2010 09:52 PM

Re: creating tar file and streaming it over HTTP?
 
pbienst wrote:
> I would like to bundle up a number of files in a tar file and send it
> over a HTTP connection, but I would like to do this without creating
> the tar file on disk first.
>


Stringio lets you treat a strings as a files...

http://docs.python.org/library/stringio.html

Roger.

Steve Holden 01-06-2010 10:03 PM

Re: creating tar file and streaming it over HTTP?
 
r0g wrote:
> pbienst wrote:
>> I would like to bundle up a number of files in a tar file and send it
>> over a HTTP connection, but I would like to do this without creating
>> the tar file on disk first.
>>

>
> Stringio lets you treat a strings as a files...
>
> http://docs.python.org/library/stringio.html
>
> Roger.


.... though that does mean that the whole tar file has to be created
before it can be sent, I suspect. If it will comfortably fit into memory
that shouldn't matter, of course.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS: http://holdenweb.eventbrite.com/

Steve Holden 01-06-2010 10:03 PM

Re: creating tar file and streaming it over HTTP?
 
r0g wrote:
> pbienst wrote:
>> I would like to bundle up a number of files in a tar file and send it
>> over a HTTP connection, but I would like to do this without creating
>> the tar file on disk first.
>>

>
> Stringio lets you treat a strings as a files...
>
> http://docs.python.org/library/stringio.html
>
> Roger.


.... though that does mean that the whole tar file has to be created
before it can be sent, I suspect. If it will comfortably fit into memory
that shouldn't matter, of course.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
PyCon is coming! Atlanta, Feb 2010 http://us.pycon.org/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS: http://holdenweb.eventbrite.com/


Gabriel Genellina 01-07-2010 11:09 AM

Re: creating tar file and streaming it over HTTP?
 
En Wed, 06 Jan 2010 13:39:08 -0300, pbienst <peter.bienstman@gmail.com>
escribió:

> The problem seems to be that the
> receiving end (wsgi server) does not see the end of the data:
>
> socket = environ["wsgi.input"]
> while True:
> sys.stderr.write("before")
> chunk = socket.read(4096)
> sys.stderr.write("after")
> if not chunk:
> sys.stderr.write("done")
> break
> sys.stderr.write(chunk)
>
> There is data from the tar file being printed, but in the end it hangs
> in the 'before' statement, and never gets to 'done'. I tried flushing
> the fileobj created by makefile(), but that doesn't seem to help.
> There is also no environ["CONTENT_LENGTH"] that I can use to detect
> the end of the stream.
>
> I'm probably missing something basic here...


Either send a "Connection: Close" header (and close the connection at the
end), or a "Transfer-Encoding: chunked" header (see
http://en.wikipedia.org/wiki/Chunked_transfer_encoding )

--
Gabriel Genellina


pbienst 01-09-2010 01:19 PM

Re: creating tar file and streaming it over HTTP?
 
OK, thanks to the feedback from everyone I got the PUT from a client
to the WSGI server working.

I'm now trying to go the other way around: use a tar stream in one of
the functions in the WSGI server in order to send files to the client.
Problem is that the WSGI specs expects an iterator as return value for
streaming, whereas TarFile needs to write to a file obj.

Is there any way I can get these two to work together?

Peter

Gabriel Genellina 01-13-2010 05:32 AM

Re: creating tar file and streaming it over HTTP?
 
En Sat, 09 Jan 2010 10:19:00 -0300, pbienst <peter.bienstman@gmail.com>
escribió:

> OK, thanks to the feedback from everyone I got the PUT from a client
> to the WSGI server working.
>
> I'm now trying to go the other way around: use a tar stream in one of
> the functions in the WSGI server in order to send files to the client.
> Problem is that the WSGI specs expects an iterator as return value for
> streaming, whereas TarFile needs to write to a file obj.
>
> Is there any way I can get these two to work together?


I haven't actually tried it, but aren't you looking for a pipe? (os.pipe()
+ os.fdopen())

--
Gabriel Genellina



All times are GMT. The time now is 07:54 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.