Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > strange sockets

Reply
Thread Tools

strange sockets

 
 
Skink
Guest
Posts: n/a
 
      11-04-2005
Hi,

I'm preparing a python server that sends java classes and resources to
custom java class loader. In order to make it faster I don't want to use
URLClassLoader that uses HTTP protocol 1.0 and for each class/resource
creates own connection.
Instead I'd like to use raw sockets with simple protocol:

- class loader sends a line terminated with \n with resource to get
- python server reads that line, gets the file and sends back an
integer with file length and then the file itself
- class loader reads a lenght integer and then reads the remainig data


The problem is when I try to read several files the first one is read
quite fast, but the rest is read 40 x slower. For example (time is in
seconds):

% python client.py client.py client.py client.py server.py server.py
init 0.00066089630127
client.py 0.000954866409302
client.py 0.0408389568329
client.py 0.0409188270569
server.py 0.0409059524536
server.py 0.0409259796143

what's wrong here?

thanks,
skink

client.py
------------------------------------------------------------------------------------
import socket, sys, struct, time

HOST = 'localhost'
PORT = 8080
t1 = time.time()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
t2 = time.time()
print "init", t2-t1
for arg in sys.argv[1:]:
t1 = time.time()
s.send(arg + "\n")
len, = struct.unpack("!i", s.recv(4))
data = s.recv(len)
t2 = time.time()
print arg, t2-t1
s.close()
------------------------------------------------------------------------------------

server.py
------------------------------------------------------------------------------------
import socket, struct, binascii

HOST = ''
PORT = 8080
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
while 1:
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
f = conn.makefile()
while 1:
resource = f.readline().rstrip()
print "[%s]" % resource
if not resource:
break
data = open(resource, "rb").read()
conn.sendall(struct.pack("!i", len(data)))
conn.sendall(data)
conn.close()
------------------------------------------------------------------------------------
 
Reply With Quote
 
 
 
 
Sion Arrowsmith
Guest
Posts: n/a
 
      11-04-2005
In article <dkfqhb$l1l$(E-Mail Removed)>, Skink <(E-Mail Removed)> wrote:
>% python client.py client.py client.py client.py server.py server.py
>init 0.00066089630127
>client.py 0.000954866409302
>client.py 0.0408389568329
>client.py 0.0409188270569
>server.py 0.0409059524536
>server.py 0.0409259796143
>
>what's wrong here?


That smells of a Nagle/delayed ACK problem to me (see, for instance,
http://www.port80software.com/200ok/...1/31/317.aspx). 40ms
is the default delayed ACK timeout on Linux, IIRC (pretty much
everything else uses 200ms). I *think* what's happening from the
server's point of view is:

receive request 1
send length (first undersized packet is sent immediately by Nagle)
(client delays ack #1)
send data (larger than 1 packet, send immediately)
(client delays acks #2--#n)
receive request 2 with ack #1
buffer sending length (undersized packet, not received last ack)
-> 40ms passes <-
(client timesout delayed acks and sends)
send length
send data (as before)

although why the undersized packet at the end of the first chunk of
data isn't buffered, I don't know.

Solutions: either change

> conn.sendall(struct.pack("!i", len(data)))
> conn.sendall(data)


to

conn.sendall(struct.pack("!i", len(data)) + data)

or after creating conn

conn.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)

to disable Nagle.

--
\S -- http://www.velocityreviews.com/forums/(E-Mail Removed) -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu becomež se bera eadward ofdun hlęddre heafdes bęce bump bump bump
 
Reply With Quote
 
 
 
 
Jim Segrave
Guest
Posts: n/a
 
      11-04-2005
In article <dkfqhb$l1l$(E-Mail Removed)>, Skink <(E-Mail Removed)> wrote:
>Hi,
>
>I'm preparing a python server that sends java classes and resources to
>custom java class loader. In order to make it faster I don't want to use
>URLClassLoader that uses HTTP protocol 1.0 and for each class/resource
>creates own connection.
>Instead I'd like to use raw sockets with simple protocol:
>
> - class loader sends a line terminated with \n with resource to get
> - python server reads that line, gets the file and sends back an
>integer with file length and then the file itself
> - class loader reads a lenght integer and then reads the remainig data
>
>
>The problem is when I try to read several files the first one is read
>quite fast, but the rest is read 40 x slower. For example (time is in
>seconds):
>
>% python client.py client.py client.py client.py server.py server.py
>init 0.00066089630127
>client.py 0.000954866409302
>client.py 0.0408389568329
>client.py 0.0409188270569
>server.py 0.0409059524536
>server.py 0.0409259796143
>
>what's wrong here?


At a guess, what you've measured is how long it takes to transfer the
data to the underlying OS socket buffers. The first transfer fills the
buffers, subsequent ones have to wait until the data has been put on
the wire and acknowledged before there's space for the writes.

--
Jim Segrave ((E-Mail Removed))

 
Reply With Quote
 
Bryan Olson
Guest
Posts: n/a
 
      11-04-2005
Skink wrote:
[...]

> what's wrong here?


Sion Arrowsmith is right about what causes the delay.
Just in case your real code looks like this, I'll note:

> len, = struct.unpack("!i", s.recv(4))
> data = s.recv(len)


First, you almost certainly don't want to use the name 'len'.
Ought not to be allowed. Second, recv can return fewer bytes
than requested, even when the connection is still open for
reading. You might replace the lines above with (untested):

length = struct.unpack("!i", s.recv(4))
data = []
while length:
data.append(s.recv(length))
length -= len(data[-1])
data = ''.join(data)


There's still a robustness problem, but in the absense of errors
and malice, that should work. I think.


--
--Bryan
 
Reply With Quote
 
Skink
Guest
Posts: n/a
 
      11-07-2005
Sion,

> Solutions: either change
>
>
>> conn.sendall(struct.pack("!i", len(data)))
>> conn.sendall(data)

>
>
> to
>
> conn.sendall(struct.pack("!i", len(data)) + data)
>
> or after creating conn
>
> conn.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
>
> to disable Nagle.


thank yuo so much, both solutions work perfect!

%python client.py client.py client.py client.py
init 0.00101184844971
client.py 0.000586986541748
client.py 0.000448942184448
client.py 0.000470161437988

I think that I'll use the second one


skink
 
Reply With Quote
 
Skink
Guest
Posts: n/a
 
      11-07-2005
Bryan,
>
> Sion Arrowsmith is right about what causes the delay.
> Just in case your real code looks like this, I'll note:
>
>> len, = struct.unpack("!i", s.recv(4))
>> data = s.recv(len)

yes, my mistake

>
>
> First, you almost certainly don't want to use the name 'len'.
> Ought not to be allowed. Second, recv can return fewer bytes
> than requested, even when the connection is still open for
> reading. You might replace the lines above with (untested):
>
> length = struct.unpack("!i", s.recv(4))
> data = []
> while length:
> data.append(s.recv(length))
> length -= len(data[-1])
> data = ''.join(data)
>
>

i know, i know, i sent fake python client: the real will be done in java.

> There's still a robustness problem, but in the absense of errors
> and malice, that should work. I think.
>
>

 
Reply With Quote
 
Skink
Guest
Posts: n/a
 
      11-07-2005
Sion Arrowsmith wrote:
>
> conn.sendall(struct.pack("!i", len(data)) + data)
>
> or after creating conn
>
> conn.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
>
> to disable Nagle.
>


Sion,

thank you for your help,

it works but...
it works when client & server is in python
i tried both solutions and they work when client is client.py
they both don't work when client is java client
when i tried to connect python's server by java client i have the same:

% java Loader server.py server.py server.py
init 29
server.py reading 631 1
server.py reading 631 40
server.py reading 631 41

why?

thanks,
skink.
 
Reply With Quote
 
Steve Holden
Guest
Posts: n/a
 
      11-07-2005
Skink wrote:
> Sion Arrowsmith wrote:
>
>>conn.sendall(struct.pack("!i", len(data)) + data)
>>
>>or after creating conn
>>
>>conn.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
>>
>>to disable Nagle.
>>

>
>
> Sion,
>
> thank you for your help,
>
> it works but...
> it works when client & server is in python
> i tried both solutions and they work when client is client.py
> they both don't work when client is java client
> when i tried to connect python's server by java client i have the same:
>
> % java Loader server.py server.py server.py
> init 29
> server.py reading 631 1
> server.py reading 631 40
> server.py reading 631 41
>
> why?
>

Seems to me that should probably be a question for comp.lang.java.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

 
Reply With Quote
 
Skink
Guest
Posts: n/a
 
      11-07-2005
Steve Holden wrote:
> Skink wrote:
>
>> Sion Arrowsmith wrote:
>>
>>> conn.sendall(struct.pack("!i", len(data)) + data)
>>>
>>> or after creating conn
>>>
>>> conn.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
>>>
>>> to disable Nagle.
>>>

>>
>>
>> Sion,
>>
>> thank you for your help,
>>
>> it works but...
>> it works when client & server is in python
>> i tried both solutions and they work when client is client.py
>> they both don't work when client is java client
>> when i tried to connect python's server by java client i have the same:
>>
>> % java Loader server.py server.py server.py
>> init 29
>> server.py reading 631 1
>> server.py reading 631 40
>> server.py reading 631 41
>>
>> why?
>>

> Seems to me that should probably be a question for comp.lang.java.


ok, my falt. again...
i forgot to use Buffered[Input|Output]Stream

>
> regards
> Steve

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
AVG Email Scanner activating at strange times with strange IP addresses dennispublic@hotmail.com Computer Support 1 08-26-2006 04:27 AM
A strange error with Sockets pieterblomme Java 4 12-27-2005 06:38 PM
strange problem with nonblocking sockets anki C Programming 3 03-01-2005 05:26 AM
Strange problem with System.Net.Sockets.TcpClient() Torsten Brasch ASP .Net 5 01-07-2004 04:51 AM
Question About Strange 'C' Code Syntax ( Well strange to me anyway ) Harvey Twyman C Programming 8 10-25-2003 05:54 AM



Advertisments