Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   Client-side TCP socket receiving "Address already in use" upon connect (http://www.velocityreviews.com/forums/t368277-client-side-tcp-socket-receiving-address-already-in-use-upon-connect.html)

Tor Erik 09-02-2006 09:05 PM

Client-side TCP socket receiving "Address already in use" upon connect
 
Hi,

The reason is that my application does about 16 connects and data
transfers per second, to the same 16 remote hosts. After approx 200 secs
there are 4000 sockets waiting to be garbage collected by the OS. At
this point is seems that connect loops and starts using the same local
addresses it used 4000 connections ago, resulting in an "Address already
in use" exception.

A possible solution to this would be to keep the connection to each
remote host open, but that would require parsing of the received data
into the data items (which are files by the way) sent by the application.

My question is if there are any other methods of solving this? Maybe a
socket option of some sort...

regards

Sybren Stuvel 09-02-2006 10:42 PM

Re: Client-side TCP socket receiving "Address already in use" upon connect
 
Tor Erik enlightened us with:
> The reason is that my application does about 16 connects and data
> transfers per second, to the same 16 remote hosts. After approx 200
> secs there are 4000 sockets waiting to be garbage collected by the
> OS.


Which OS are we talking about?

> At this point is seems that connect loops and starts using the same
> local addresses it used 4000 connections ago, resulting in an
> "Address already in use" exception.


After how many seconds does this happen?

> My question is if there are any other methods of solving this? Maybe
> a socket option of some sort...


If I'm correct (please correct me if I'm not), on Linux you can use
'sysctl -w net.ipv4.tcp_fin_timeout=X' to set the time between closing
the socket and releasing it to be reused. You can also check the
SO_REUSEADDR argument to the setsockopt function. Read 'man 7 socket'
for more info.

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa

Tor Erik 09-03-2006 08:41 AM

Re: Client-side TCP socket receiving "Address already in use" uponconnect
 
Sybren Stuvel wrote:
> Tor Erik enlightened us with:
>> The reason is that my application does about 16 connects and data
>> transfers per second, to the same 16 remote hosts. After approx 200
>> secs there are 4000 sockets waiting to be garbage collected by the
>> OS.

>
> Which OS are we talking about?


Windows XP

>
>> At this point is seems that connect loops and starts using the same
>> local addresses it used 4000 connections ago, resulting in an
>> "Address already in use" exception.

>
> After how many seconds does this happen?


200 seconds approx

>
>> My question is if there are any other methods of solving this? Maybe
>> a socket option of some sort...

>
> If I'm correct (please correct me if I'm not), on Linux you can use
> 'sysctl -w net.ipv4.tcp_fin_timeout=X' to set the time between closing
> the socket and releasing it to be reused. You can also check the
> SO_REUSEADDR argument to the setsockopt function. Read 'man 7 socket'
> for more info.


I've read about SO_REUSEADDR. As far as I understand, this is what
SO_REUSEADDR is for:

1. Allow a listening socket to bind itself to its well-known port even
if previously established connections use it as their local port.
Setting this option should be done between calls to socket and bind, and
hence is only usable for listening sockets, not client sockets like mine.

2. Allow multiple servers on the same host with different ip-adresses to
listen to the same port.

I've tried setting this option, but could not see any notable changes...

>
> Sybren


keegan.csmith@gmail.com 09-03-2006 10:26 AM

Re: Client-side TCP socket receiving "Address already in use" upon connect
 
> I've read about SO_REUSEADDR. As far as I understand, this is what
> SO_REUSEADDR is for:

....
> I've tried setting this option, but could not see any notable changes...


I was having a similiar problem as you, where as soon as my program
exited, it would get started up again, but could not bind to the same
address.
So i added the follow straight after I create my server object:
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

And it worked. Note that my program was running on Linux, so this might
be a windows issue.


Fredrik Lundh 09-03-2006 10:42 AM

Re: Client-side TCP socket receiving "Address already in use" uponconnect
 
Tor Erik wrote:

> The reason is that my application does about 16 connects and data
> transfers per second, to the same 16 remote hosts. After approx 200 secs
> there are 4000 sockets waiting to be garbage collected by the OS.


what does "netstat" say about these sockets ?

</F>


Tor Erik 09-03-2006 11:52 AM

Re: Client-side TCP socket receiving "Address already in use" uponconnect
 
Fredrik Lundh wrote:
> Tor Erik wrote:
>
>> The reason is that my application does about 16 connects and data
>> transfers per second, to the same 16 remote hosts. After approx 200
>> secs there are 4000 sockets waiting to be garbage collected by the OS.

>
> what does "netstat" say about these sockets ?
>
> </F>
>


They are in the TIME_WAIT state... The msn library has an article on how
to solve this:

http://msdn.microsoft.com/library/de...1fc8ba06a4.asp

Summing up one could either:

1. Increase the upper range of ephemeral ports that are dynamically
allocated to client TCP/IP socket connections:

set registry key:
KEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Service s\Tcpip\Parameters\MaxUserPort
to a new DWORD value... (5000 - 65534)
The default in XP is 3976 -> http://support.microsoft.com/kb/Q149532

or

2. Reduce the client TCP/IP socket connection timeout value from the
default value of 240 seconds

set registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servic es\Tcpip\Parameters\TcpTimedWaitDelay
to a new DWORD value (30 - 300)

The TCP RFC (RFC 793) recommends a value of 2*msl(Maximum Segment
Lifetime). The general consensus about the value of msn seems to be 1-2
minutes, depending on the underlying network... (2*2 min = 2*120 sec =
240 sec)


I do not want to alter my registry, so I'm currently testing an idea
where I let the client connect and send its content, appended with my
own "magic" EOF byte-sequence. When the server receives this EOF, it
takes care to close the connection. This should eliminate the problem as
it is the peer closing the connection that enters the TIME_WAIT state...

I will report my experiences...

Steve Holden 09-03-2006 11:53 AM

Re: Client-side TCP socket receiving "Address already in use" uponconnect
 
keegan.csmith@gmail.com wrote:
>>I've read about SO_REUSEADDR. As far as I understand, this is what
>>SO_REUSEADDR is for:

>
> ....
>
>>I've tried setting this option, but could not see any notable changes...

>
>
> I was having a similiar problem as you, where as soon as my program
> exited, it would get started up again, but could not bind to the same
> address.
> So i added the follow straight after I create my server object:
> server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
>
> And it worked. Note that my program was running on Linux, so this might
> be a windows issue.
>

.... and note also that your program was apprently a server, while the OP
was reporting an error on a client program that presumably asks for an
ephemeral port rather than a specifically-numbered one.

Since there are roughly 64,000 ports, the real question seems to be why
his client runs out after about 4,000.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden


Tor Erik 09-03-2006 12:46 PM

Re: Client-side TCP socket receiving "Address already in use" uponconnect
 
Tor Erik wrote:
> Fredrik Lundh wrote:
>> Tor Erik wrote:
>>
>>> The reason is that my application does about 16 connects and data
>>> transfers per second, to the same 16 remote hosts. After approx 200
>>> secs there are 4000 sockets waiting to be garbage collected by the OS.

>>
>> what does "netstat" say about these sockets ?
>>
>> </F>
>>

>
> They are in the TIME_WAIT state... The msn library has an article on how
> to solve this:
>
> http://msdn.microsoft.com/library/de...1fc8ba06a4.asp
>
>
> Summing up one could either:
>
> 1. Increase the upper range of ephemeral ports that are dynamically
> allocated to client TCP/IP socket connections:
>
> set registry key:
> KEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Service s\Tcpip\Parameters\MaxUserPort
>
> to a new DWORD value... (5000 - 65534)
> The default in XP is 3976 -> http://support.microsoft.com/kb/Q149532
>
> or
>
> 2. Reduce the client TCP/IP socket connection timeout value from the
> default value of 240 seconds
>
> set registry key:
> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servic es\Tcpip\Parameters\TcpTimedWaitDelay
>
> to a new DWORD value (30 - 300)
>
> The TCP RFC (RFC 793) recommends a value of 2*msl(Maximum Segment
> Lifetime). The general consensus about the value of msn seems to be 1-2
> minutes, depending on the underlying network... (2*2 min = 2*120 sec =
> 240 sec)
>
>
> I do not want to alter my registry, so I'm currently testing an idea
> where I let the client connect and send its content, appended with my
> own "magic" EOF byte-sequence. When the server receives this EOF, it
> takes care to close the connection. This should eliminate the problem as
> it is the peer closing the connection that enters the TIME_WAIT state...
>
> I will report my experiences...


Well... my idea does not work as expected. Even though the server
(remote host) calls socket.close(), it is the client that executes
TIME_WAIT. My guess is that the subtrates below socket closes the
connection at the peer calling connect regardless of where socket.close
is called.

Thoughts anyone?

Steve Holden 09-03-2006 01:54 PM

Re: Client-side TCP socket receiving "Address already in use" uponconnect
 
Tor Erik wrote:
> Tor Erik wrote:
>
>>Fredrik Lundh wrote:
>>
>>>Tor Erik wrote:
>>>
>>>
>>>>The reason is that my application does about 16 connects and data
>>>>transfers per second, to the same 16 remote hosts. After approx 200
>>>>secs there are 4000 sockets waiting to be garbage collected by the OS.
>>>
>>>what does "netstat" say about these sockets ?
>>>
>>></F>
>>>

>>They are in the TIME_WAIT state... The msn library has an article on how
>>to solve this:
>>
>>http://msdn.microsoft.com/library/de...1fc8ba06a4.asp
>>
>>
>>Summing up one could either:
>>
>>1. Increase the upper range of ephemeral ports that are dynamically
>>allocated to client TCP/IP socket connections:
>>
>>set registry key:
>>KEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servi ces\Tcpip\Parameters\MaxUserPort
>>
>>to a new DWORD value... (5000 - 65534)
>>The default in XP is 3976 -> http://support.microsoft.com/kb/Q149532
>>
>>or
>>
>>2. Reduce the client TCP/IP socket connection timeout value from the
>>default value of 240 seconds
>>
>>set registry key:
>>HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Serv ices\Tcpip\Parameters\TcpTimedWaitDelay
>>
>>to a new DWORD value (30 - 300)
>>
>>The TCP RFC (RFC 793) recommends a value of 2*msl(Maximum Segment
>>Lifetime). The general consensus about the value of msn seems to be 1-2
>>minutes, depending on the underlying network... (2*2 min = 2*120 sec =
>>240 sec)
>>
>>
>>I do not want to alter my registry, so I'm currently testing an idea
>>where I let the client connect and send its content, appended with my
>>own "magic" EOF byte-sequence. When the server receives this EOF, it
>>takes care to close the connection. This should eliminate the problem as
>>it is the peer closing the connection that enters the TIME_WAIT state...
>>
>>I will report my experiences...

>
>
> Well... my idea does not work as expected. Even though the server
> (remote host) calls socket.close(), it is the client that executes
> TIME_WAIT. My guess is that the subtrates below socket closes the
> connection at the peer calling connect regardless of where socket.close
> is called.
>
> Thoughts anyone?


Yes, it's the transport layer that puts the socket into the TIME_WAIT state.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden


Alex Martelli 09-03-2006 03:39 PM

Re: Client-side TCP socket receiving "Address already in use" upon connect
 
Steve Holden <steve@holdenweb.com> wrote:
...
> >>set registry key:

>
>>KEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servi ces\Tcpip\Parameters\M

axUserPort
> >>
> >>to a new DWORD value... (5000 - 65534)
> >>The default in XP is 3976 -> http://support.microsoft.com/kb/Q149532


I wonder why (performance under RAM-costrained conditions? but then why
not have this vary depending on available RAM -- complications?)

> Yes, it's the transport layer that puts the socket into the TIME_WAIT state.


Yes, there's a good explanation at
<http://www.developerweb.net/forum/showthread.php?t=2941> (though one
should really study Stevens' "TCP-IP Illustrated" for better
understanding in depth). Playing with SO_LINGER and/or the MSL is not
recommended unless you're operating only on a network that you entirely
control (particularly in terms of round-trip times and router behavior).

As debated at
<http://www.sunmanagers.org/pipermail...ry/007068.html
>, you may be able to have your clients go into CLOSE_WAIT (instead of

TIME_WAIT) by playing around with "who closes the socket first", and
CLOSE_WAIT might be more transient than the 2*MSL (240 seconds...)
needed for TIME_WAIT on possibly-unreliable networks. But it's far from
sure that this would gain you much.

Reflecting on the OP's use case, since all connections are forever being
made to the same 16 servers, why not tweak thinks a bit to hold those
connections open for longer periods of time, using a connection for many
send/receive "transactions" instead of opening and closing such
connections all of the time? That might well work better...


Alex


All times are GMT. The time now is 04:32 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.