Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Receive data from socket stream

Reply
Thread Tools

Receive data from socket stream

 
 
s0suk3@gmail.com
Guest
Posts: n/a
 
      04-25-2008
I wanted to ask for standard ways to receive data from a socket stream
(with socket.socket.recv()). It's simple when you know the amount of
data that you're going to receive, or when you'll receive data until
the remote peer closes the connection. But I'm not sure which is the
best way to receive a message with undetermined length from a stream
in a connection that you expect to remain open. Until now, I've been
doing this little trick:

data = client.recv(256)
new = data
while len(new) == 256:
new = client.recv(256)
data += new

That works well in most cases. But it's obviously error-prone. What if
the client sent *exactly* two hundred and fifty six bytes? It would
keep waiting for data inside the loop. Is there really a better and
standard way, or is this as best as it gets?

Sorry if this is a little off-topic and more related to networking,
but I'm using Python anyway.

Thanks,
Sebastian
 
Reply With Quote
 
 
 
 
s0suk3@gmail.com
Guest
Posts: n/a
 
      04-25-2008
On Apr 25, 5:52*pm, Erik Max Francis <(E-Mail Removed)> wrote:
> (E-Mail Removed) wrote:
> > I wanted to ask for standard ways to receive data from a socket stream
> > (with socket.socket.recv()). It's simple when you know the amount of
> > data that you're going to receive, or when you'll receive data until
> > the remote peer closes the connection. But I'm not sure which is the
> > best way to receive a message with undetermined length from a stream
> > in a connection that you expect to remain open. Until now, I've been
> > doing this little trick:

>
> > data = client.recv(256)
> > new = data
> > while len(new) == 256:
> > * * new = client.recv(256)
> > * * data += new

>
> > That works well in most cases. But it's obviously error-prone. What if
> > the client sent *exactly* two hundred and fifty six bytes? It would
> > keep waiting for data inside the loop. Is there really a better and
> > standard way, or is this as best as it gets?

>
> > Sorry if this is a little off-topic and more related to networking,
> > but I'm using Python anyway.

>
> You solve this by having a protocol that the client and server both
> agree on, so that the client knows how much to read from the server.
> There are any number of ways of doing this, all of which depend on the
> kind of data you want to transfer and for what purpose.
>
> --
> Erik Max Francis && (E-Mail Removed) &&http://www.alcyone.com/max/
> * San Jose, CA, USA && 37 18 N 121 57 W && AIM, Y!M erikmaxfrancis
> * *In the final choice a solider's pack is not so heavy a burden as a
> * * prisoner's chains. -- Dwight D. Eisenhower, 1890-1969


So, in an HTTP client/server, I'd had to look in a Content-Length
header?
 
Reply With Quote
 
 
 
 
hdante
Guest
Posts: n/a
 
      04-26-2008
On Apr 25, 7:39*pm, (E-Mail Removed) wrote:
> I wanted to ask for standard ways to receive data from a socket stream
> (with socket.socket.recv()). It's simple when you know the amount of
> data that you're going to receive, or when you'll receive data until
> the remote peer closes the connection. But I'm not sure which is the
> best way to receive a message with undetermined length from a stream
> in a connection that you expect to remain open. Until now, I've been
> doing this little trick:
>
> data = client.recv(256)
> new = data
> while len(new) == 256:
> * * new = client.recv(256)
> * * data += new
>
> That works well in most cases. But it's obviously error-prone. What if
> the client sent *exactly* two hundred and fifty six bytes? It would
> keep waiting for data inside the loop. Is there really a better and
> standard way, or is this as best as it gets?
>
> Sorry if this is a little off-topic and more related to networking,
> but I'm using Python anyway.
>
> Thanks,
> Sebastian



done = False
remaining = ''
while done == False:
data = client.recv(256)
done, remaining = process(remaining + data)

PS: are you sure you shouldn't be using RPC or SOAP ?
 
Reply With Quote
 
Irmen de Jong
Guest
Posts: n/a
 
      04-26-2008
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> Until now, I've been
> doing this little trick:
>
> data = client.recv(256)
> new = data
> while len(new) == 256:
> new = client.recv(256)
> data += new


Are you aware that recv() will not always return the amount of bytes asked for?
(send() is similar; it doesn't guarantee that the full buffer you pass to it will be
sent at once)

I suggest reading this: http://www.amk.ca/python/howto/sockets/sockets.html


--irmen
 
Reply With Quote
 
s0suk3@gmail.com
Guest
Posts: n/a
 
      04-27-2008
On Apr 26, 7:25 am, Irmen de Jong <(E-Mail Removed)> wrote:
> (E-Mail Removed) wrote:
> > Until now, I've been
> > doing this little trick:

>
> > data = client.recv(256)
> > new = data
> > while len(new) == 256:
> > new = client.recv(256)
> > data += new

>
> Are you aware that recv() will not always return the amount of bytes asked for?
> (send() is similar; it doesn't guarantee that the full buffer you pass to it will be
> sent at once)
>
> I suggest reading this:http://www.amk.ca/python/howto/sockets/sockets.html
>
> --irmen


So every time I use I want to send some thing, I must use

totalsent = 0
while sent < len(data):
sent = sock.send(data[totalsent:])
totalsent += sent

instead of a simple sock.send(data)? That's kind of nasty. Also, is it
better then to use sockets as file objects? Maybe unbuffered?
 
Reply With Quote
 
Hrvoje Niksic
Guest
Posts: n/a
 
      04-28-2008
Nick Craig-Wood <(E-Mail Removed)> writes:

> What you are missing is that if the recv ever returns no bytes at all
> then the other end has closed the connection. So something like this
> is the correct thing to write :-
>
> data = ""
> while True:
> new = client.recv(256)
> if not new:
> break
> data += new


This is a good case for the iter() function:

buf = cStringIO.StringIO()
for new in iter(partial(client.recv, 256), ''):
buf.write(new)
data = buf.getvalue()

Note that appending to a string is almost never a good idea, since it
can result in quadratic allocation.
 
Reply With Quote
 
s0suk3@gmail.com
Guest
Posts: n/a
 
      04-28-2008
On Apr 28, 4:30 am, Nick Craig-Wood <(E-Mail Removed)> wrote:
> (E-Mail Removed) <(E-Mail Removed)> wrote:
> > I wanted to ask for standard ways to receive data from a socket stream
> > (with socket.socket.recv()). It's simple when you know the amount of
> > data that you're going to receive, or when you'll receive data until
> > the remote peer closes the connection. But I'm not sure which is the
> > best way to receive a message with undetermined length from a stream
> > in a connection that you expect to remain open. Until now, I've been
> > doing this little trick:

>
> > data = client.recv(256)
> > new = data
> > while len(new) == 256:
> > new = client.recv(256)
> > data += new

>
> > That works well in most cases. But it's obviously error-prone. What if
> > the client sent *exactly* two hundred and fifty six bytes? It would
> > keep waiting for data inside the loop. Is there really a better and
> > standard way, or is this as best as it gets?

>
> What you are missing is that if the recv ever returns no bytes at all
> then the other end has closed the connection. So something like this
> is the correct thing to write :-
>
> data = ""
> while True:
> new = client.recv(256)
> if not new:
> break
> data += new
>
> From the man page for recv
>
> RETURN VALUE
>
> These calls return the number of bytes received, or -1 if an
> error occurred. The return value will be 0 when the peer has
> performed an orderly shutdown.
>
> In the -1 case python will raise a socket.error.
>
> --
> Nick Craig-Wood <(E-Mail Removed)> --http://www.craig-wood.com/nick


But as I said in my first post, it's simple when you know the amount
of data that you're going to receive, or when you'll receive data
until the remote peer closes the connection. But what about receiving
a message with undetermined length in a connection that you don't want
to close? I already figured it out: in the case of an HTTP server/
client (which is what I'm doing), you have to look for an empty line
in the message, which signals the end of the message headers. As for
the body, you have to look at the Content-Length header, or, if the
message body contains the "chunked" transfer-coding, you have to
dynamically decode the encoding. There are other cases but those are
the most influent.

BTW, has anybody used sockets as file-like objects
(client.makefile())? Is it more secure? More efficient?

Sebastian
 
Reply With Quote
 
Hrvoje Niksic
Guest
Posts: n/a
 
      04-28-2008
Nick Craig-Wood <(E-Mail Removed)> writes:

>> Note that appending to a string is almost never a good idea, since it
>> can result in quadratic allocation.

>
> My aim was clear exposition rather than the ultimate performance!


That would normally be fine. My post wasn't supposed to pick
performance nits, but to point out potentially quadratic behavior.

> Anyway str += was optimised in python 2.4 or 2.5 (forget which) wasn't
> it?


That optimization works only in certain cases, when working with
uninterned strings with a reference count of 1, and then only when the
strings are in stored local variables, rather than in global vars or
in slots. And then, it only works in CPython, not in other
implementations. The optimization works by "cheating" -- breaking the
immutable string abstraction in the specific cases in which it is
provably safe to do so.
http://utcc.utoronto.ca/~cks/space/b...tringConcatOpt
examines it in some detail.

Guido was reluctant to accept the patch that implements the
optimization because he thought it would "change the way people write
code", a sentiment expressed in
http://mail.python.org/pipermail/pyt...st/046702.html
This discussion shows that he was quite right in retrospect. (I'm not
saying that the optimization is a bad thing, just that it is changing
the "recommended" way of writing Python in a way that other
implementations cannot follow.)
 
Reply With Quote
 
s0suk3@gmail.com
Guest
Posts: n/a
 
      04-28-2008
On Apr 28, 4:42 am, Hrvoje Niksic <(E-Mail Removed)> wrote:
> Nick Craig-Wood <(E-Mail Removed)> writes:
> > What you are missing is that if the recv ever returns no bytes at all
> > then the other end has closed the connection. So something like this
> > is the correct thing to write :-

>
> > data = ""
> > while True:
> > new = client.recv(256)
> > if not new:
> > break
> > data += new

>
> This is a good case for the iter() function:
>
> buf = cStringIO.StringIO()
> for new in iter(partial(client.recv, 256), ''):
> buf.write(new)
> data = buf.getvalue()
>
> Note that appending to a string is almost never a good idea, since it
> can result in quadratic allocation.


A question regarding cStringIO.StringIO(): is there a way to do get
getvalue() to return all the bytes after the current file position
(not before)? For example

buf = cStringIO.StringIO()
buf.write("foo bar")
buf.seek(3)
buf.getvalue(True) # the True argument means
# to return the bytes up
# to the current file position

That returns 'foo'. Is there a way to get it to return ' bar'?
 
Reply With Quote
 
Gabriel Genellina
Guest
Posts: n/a
 
      04-29-2008
En Mon, 28 Apr 2008 19:29:33 -0300, <(E-Mail Removed)> escribió:

> A question regarding cStringIO.StringIO(): is there a way to do get
> getvalue() to return all the bytes after the current file position
> (not before)? For example
>
> buf = cStringIO.StringIO()
> buf.write("foo bar")
> buf.seek(3)
> buf.getvalue(True) # the True argument means
> # to return the bytes up
> # to the current file position
>
> That returns 'foo'. Is there a way to get it to return ' bar'?


buf.read() - the obvious answer, once you know it

--
Gabriel Genellina

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to receive a data file of unknown length using a python socket? twgray Python 15 07-19-2009 04:09 PM
socket.unbind or socket.unlisten? - socket.error: (48, 'Addressalready in use') Laszlo Nagy Python 1 01-27-2009 05:05 PM
How to receive data using socket programming Kamaljeet Saini Ruby 10 01-26-2009 08:51 AM
How to receive data from a socket Suk Perl Misc 1 03-30-2006 08:20 AM
using pthread to receive data on a socket and simultaneously play it using /dev/dsp kanchan C Programming 1 11-13-2004 10:33 PM



Advertisments