Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Scatter/gather on sockets?

Reply
Thread Tools

Scatter/gather on sockets?

 
 
Roy Smith
Guest
Posts: n/a
 
      04-01-2006
I've got a bunch of strings in a list:

vector = []
vector.append ("foo")
vector.append ("bar")
vector.append ("baz")

I want to send all of them out a socket in a single send() call, so
they end up in a single packet (assuming the MTU is large enough). I
can do:

mySocket.send ("".join (vector))

but that involves creating an intermediate string. Is there a more
efficient way, that doesn't involve that extra data copy?
 
Reply With Quote
 
 
 
 
Peter Hansen
Guest
Posts: n/a
 
      04-01-2006
Roy Smith wrote:
> I've got a bunch of strings in a list:
>
> vector = []
> vector.append ("foo")
> vector.append ("bar")
> vector.append ("baz")
>
> I want to send all of them out a socket in a single send() call, so
> they end up in a single packet (assuming the MTU is large enough). I
> can do:
>
> mySocket.send ("".join (vector))
>
> but that involves creating an intermediate string. Is there a more
> efficient way, that doesn't involve that extra data copy?


Two possible answers that I can see:

A. No, the send call is implemented in C and requires a single buffer
with the entire piece of data that will be sent. Ultimately this gets
passed down to the NIC hardware in some fashion, so there's certainly no
hope of using something like a generator to send it in pieces.

B. Don't bother trying, because even if the MTU is large enough there is
absolutely no guarantee that the packet will stay intact all the way
through the network anyway (even if you use sendall() instead of send()).

So fixing your design not to require this appears to be the only viable
solution.

-Peter

 
Reply With Quote
 
 
 
 
Anthony Greene
Guest
Posts: n/a
 
      04-01-2006
On Sat, 01 Apr 2006 14:56:02 -0500, Roy Smith wrote:

> I've got a bunch of strings in a list:
>
> vector = []
> vector.append ("foo")
> vector.append ("bar")
> vector.append ("baz")
>
> I want to send all of them out a socket in a single send() call, so
> they end up in a single packet (assuming the MTU is large enough). I
> can do:
>
> mySocket.send ("".join (vector))
>
> but that involves creating an intermediate string. Is there a more
> efficient way, that doesn't involve that extra data copy?


Is sendall() what you're looking for?

--
A wise man knows he knows nothing.

 
Reply With Quote
 
Roy Smith
Guest
Posts: n/a
 
      04-01-2006
Peter Hansen <(E-Mail Removed)> wrote:
> B. Don't bother trying, because even if the MTU is large enough there is
> absolutely no guarantee that the packet will stay intact all the way
> through the network anyway (even if you use sendall() instead of send()).


This is true, but I'm generating the message being sent in very small
chunks (often as small as 4 bytes at a time), and typically need to flush a
packet out onto the network after a few dozen bytes. Maybe at most a few
hundred. I don't know of any networks with MTU's smaller than that.
Measurements show a 10-fold improvement in protocol throughput with large
packets vs. small ones. The only question is what's the most efficient way
in Python to generate the large packets.

> So fixing your design not to require this appears to be the only viable
> solution.


My design is not broken. I'm writing code to drive a pre-existing binary
communications protocol. It is what it is. The functionality I seek
exists at the Unix system call level (writev, sendmsg), but doesn't appear
to be exposed in the Python socket API.
 
Reply With Quote
 
Roy Smith
Guest
Posts: n/a
 
      04-01-2006
In article <(E-Mail Removed)>,
Anthony Greene <(E-Mail Removed)> wrote:

> On Sat, 01 Apr 2006 14:56:02 -0500, Roy Smith wrote:
>
> > I've got a bunch of strings in a list:
> >
> > vector = []
> > vector.append ("foo")
> > vector.append ("bar")
> > vector.append ("baz")
> >
> > I want to send all of them out a socket in a single send() call, so
> > they end up in a single packet (assuming the MTU is large enough). I
> > can do:
> >
> > mySocket.send ("".join (vector))
> >
> > but that involves creating an intermediate string. Is there a more
> > efficient way, that doesn't involve that extra data copy?

>
> Is sendall() what you're looking for?


No. Sendall() is actually what I'm using now. It handles the other side
of the issue; issuing repeated send() calls if the system fragments your
buffer. I'm trying to aggregate lots of small buffers into one large one.
 
Reply With Quote
 
Paul Rubin
Guest
Posts: n/a
 
      04-01-2006
Roy Smith <(E-Mail Removed)> writes:
> This is true, but I'm generating the message being sent in very small
> chunks (often as small as 4 bytes at a time), and typically need to flush a
> packet out onto the network after a few dozen bytes. Maybe at most a few
> hundred. I don't know of any networks with MTU's smaller than that.
> Measurements show a 10-fold improvement in protocol throughput with large
> packets vs. small ones. The only question is what's the most efficient way
> in Python to generate the large packets.


Probably: build up the packet with cStringIO or with the array module
instead of as a list of small strings. But if you time both versions
I don't think it'll matter much. Python (at least CPython) simply
will not be very fast no matter what you do. The overhead of building
a large string (with ''.join, say) from a bunch of small ones isn't
that big a deal compared with what you already lose in interpreter
overhead running the application.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off




Advertisments