Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Strange Socket problem

Reply
Thread Tools

Strange Socket problem

 
 
Knute Johnson
Guest
Posts: n/a
 
      03-01-2012
I'm having a problem in some production code that I can't figure out.
I'll post the complete actual code below. This code is running in three
places and has the same problem in two of them at the same time. The
other I'm not sure, it may be that the personnel operating it are
restarting the program and so don't complain. This piece of code is a
simple client that connects via a Socket to a server. The server
supplies some data and the client reads that data and files it away. It
is supposed to restart itself if there is a connection failure or fault
for whatever reason. The problem is that at some random point in time
the Socket disconnects, the code logs the disconnect but never restarts.
It does print the "SportsWinClient Disconnected" message but never
executes the "fireconnectionEvent()" method after creating a new Socket.
It doesn't print any Exception message. I'm not sure how it gets out
of the try block without printing the "End of Stream" message or an
exception message.

The crazy part is that all night long when there is no activity from the
server it times out and restarts with no problems.

I'm hoping that somebody will see a fault in my code that could cause
the failure. It is not a compile problem so I left the formatting as it is.

Thanks for looking.

package com.knutejohnson.xyzcasinos.translux;

import java.io.*;
import java.net.*;
import java.util.*;

import com.knutejohnson.classes.*;

import static com.knutejohnson.xyzcasinos.translux.Constants.*;

public class SportsWinClient implements Runnable {
private final Thread thread;

private volatile boolean isConnected;
private volatile boolean runFlag = true;

private volatile Socket socket;

public SportsWinClient() {
thread = new Thread(this,"SportsWinClient");
}

public void start() {
thread.start();
}

public void run() {
// boolean serverFlag = true;

System.out.println("SportsWinClient: Started");
while (runFlag) {
// String serverAddress = serverFlag ? SPORTS_WIN_IP_PRIMARY :
// SPORTS_WIN_IP_SECONDARY;
try {
// socket = new Socket(serverAddress,SPORTS_WIN_PORT,
socket = new Socket(SPORTS_WIN_IP,SPORTS_WIN_PORT,
InetAddress.getByName(REMOTE_IP),0);
socket.setKeepAlive(true);
isConnected = true;

********* I know that the line below is not being executed **********

fireConnectionEvent(ConnectionEvent.CONNECTED);
socket.setSoTimeout(3600000); // one hour timeout
System.out.println("SportsWinClient: Connected");
InputStream is = socket.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);

String str;
while ((str = br.readLine()) != null) {
if (!str.matches("\\d+.*")) // not a sports record
continue;
SportsBet sb = new SportsBet(str);
SPORTS_BET_MAP.put(sb.betNumber,sb);
}

System.out.println("SportsWinClient: End of Stream");
} catch (IOException ioe) {
System.out.println("SportsWinClient: " + ioe.toString());
} finally {
isConnected = false;
if (socket != null)
try {
socket.close();
} catch (IOException ioe) {
ioe.printStackTrace();
}
fireConnectionEvent(ConnectionEvent.DISCONNECTED);
// serverFlag = !serverFlag;

*********** I know that the line below is being executed *************

System.out.println("SportsWinClient: Disconnected");
}
// stop interrupts this thread so this will be bypassed on
a stop
try {
Thread.sleep(10000);
} catch (InterruptedException ie) { }
}
System.out.println("SportsWinClient: Stopping");
}

public void disconnect() {
if (isConnected())
if (socket != null)
try {
socket.close();
} catch (IOException ioe) {
ioe.printStackTrace();
}
}

public void stop() {
runFlag = false;
thread.interrupt();
if (socket != null)
try {
socket.close();
} catch (IOException ioe) {
ioe.printStackTrace();
}
}

public boolean isConnected() {
return isConnected;
}

private final java.util.List<ConnectionListener> connectionListeners =
new ArrayList<ConnectionListener>();

public synchronized void addConnectionListener(ConnectionListener cl) {
connectionListeners.add(cl);
}

public synchronized void
removeConnectionListener(ConnectionListener cl) {
connectionListeners.remove(cl);
}

private synchronized void fireConnectionEvent(int id) {
ConnectionEvent ce = new ConnectionEvent(this,id);

for (ConnectionListener listener : connectionListeners)
listener.connState(ce);
}
}



--

Knute Johnson
 
Reply With Quote
 
 
 
 
Steven Simpson
Guest
Posts: n/a
 
      03-01-2012
On 01/03/12 19:49, Knute Johnson wrote:
> It does print the "SportsWinClient Disconnected" message but never
> executes the "fireconnectionEvent()" method after creating a new
> Socket. It doesn't print any Exception message. I'm not sure how it
> gets out of the try block without printing the "End of Stream" message
> or an exception message.


Can you provide an exact trace of the messages that are visible in the
code you've shown (around the problematic event, of course), just so
there's no ambiguity about what you're seeing?

> System.out.println("SportsWinClient: End of Stream");
> } catch (IOException ioe) {
> System.out.println("SportsWinClient: " + ioe.toString());


Catch Throwable here too, print it out, and rethrow, so we can be sure
no other exception is slipping through.

> } finally {
> isConnected = false;
> if (socket != null)
> try {



--
ss at comp dot lancs dot ac dot uk

 
Reply With Quote
 
 
 
 
Knute Johnson
Guest
Posts: n/a
 
      03-01-2012
On 3/1/2012 12:59 PM, Steven Simpson wrote:
> On 01/03/12 19:49, Knute Johnson wrote:
>> It does print the "SportsWinClient Disconnected" message but never
>> executes the "fireconnectionEvent()" method after creating a new
>> Socket. It doesn't print any Exception message. I'm not sure how it
>> gets out of the try block without printing the "End of Stream" message
>> or an exception message.

>
> Can you provide an exact trace of the messages that are visible in the
> code you've shown (around the problematic event, of course), just so
> there's no ambiguity about what you're seeing?
>
>> System.out.println("SportsWinClient: End of Stream");
>> } catch (IOException ioe) {
>> System.out.println("SportsWinClient: " + ioe.toString());

>
> Catch Throwable here too, print it out, and rethrow, so we can be sure
> no other exception is slipping through.
>
>> } finally {
>> isConnected = false;
>> if (socket != null)
>> try {

>
>


When the problem arrives, it prints the SportsWinClient: Disconnected
message and nothing else. It doesn't reconnect to the server.

I'll try putting in a catch for Throwable but it could take some time
before I see the problem again.

Thanks,

--

Knute Johnson
 
Reply With Quote
 
markspace
Guest
Posts: n/a
 
      03-01-2012
On 3/1/2012 11:49 AM, Knute Johnson wrote:
> I'm having a problem in some production code that I can't figure out.



This code below strikes me as possibly being in the wrong order. If you
interrupt a thread, it should bail. But if you close the socket, it
should do the same thing, and close the socket too (I'm pretty sure
there's a handshake for TCP for "close me".)

If you interrupt the thread, then close the socket, the "close" might
never happen. I'd *just* close the socket, if that passes your testing.
Try to let the thread actually execute the close, and then unwind
naturally.

(I didn't look to see if your thread/runnable has other places it waits
besides the socket. If it does, those should be eliminated, replaced
with other sockets which are also closed, etc.)



 
Reply With Quote
 
Martin Gregorie
Guest
Posts: n/a
 
      03-01-2012
On Thu, 01 Mar 2012 11:49:30 -0800, Knute Johnson wrote:

> I'm having a problem in some production code that I can't figure out.
> I'll post the complete actual code below. This code is running in three
> places and has the same problem in two of them at the same time. The
> other I'm not sure, it may be that the personnel operating it are
> restarting the program and so don't complain. This piece of code is a
> simple client that connects via a Socket to a server. The server
> supplies some data and the client reads that data and files it away. It
> is supposed to restart itself if there is a connection failure or fault
> for whatever reason. The problem is that at some random point in time
> the Socket disconnects, the code logs the disconnect but never restarts.
> It does print the "SportsWinClient Disconnected" message but never
> executes the "fireconnectionEvent()" method after creating a new Socket.
> It doesn't print any Exception message. I'm not sure how it gets out
> of the try block without printing the "End of Stream" message or an
> exception message.
>
> The crazy part is that all night long when there is no activity from the
> server it times out and restarts with no problems.
>

I notice that InetAddress.getByName() can throw UnknownHostException (and
this exception isn't explicitly handled. Is it possible that this call
can time out when the network is busy?

However, as UnknownHostException is a subclass of IOException I don't see
how it avoids being caught, unless a timeout causes a null to be
returned: the description of getByName() isn't at all clear about when a
null is returned instead of throwing an exception. Pulling this out of
the new Socket() statement so it can be tested for a null return would at
least show whether this is happening.

Also, I don't see why you're using InetAddress to pass what is apparently
a remote address to the Socket constructor as localAddress, or why you're
using port zero as the local port.


--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
 
Reply With Quote
 
Steven Simpson
Guest
Posts: n/a
 
      03-01-2012
On 01/03/12 21:30, Knute Johnson wrote:
> When the problem arrives, it prints the SportsWinClient: Disconnected
> message and nothing else. It doesn't reconnect to the server.


What about previous messages (Started, Connected, End of Stream),
especially for the cycle before the problem? Do you get this?:

Started
Connected
End of Stream
Disconnected
...
Connected
End of Stream
Disconnected
Connected
Disconnected

Or this?:

Started
Connected
End of Stream
Disconnected
...
Connected
End of Stream
Disconnected
Disconnected

Or is there no previous (successful) cycle?

I'm thinking that maybe a listener is throwing an unchecked exception.

--
ss at comp dot lancs dot ac dot uk

 
Reply With Quote
 
Knute Johnson
Guest
Posts: n/a
 
      03-02-2012
On 3/1/2012 2:54 PM, Steven Simpson wrote:
> On 01/03/12 21:30, Knute Johnson wrote:
>> When the problem arrives, it prints the SportsWinClient: Disconnected
>> message and nothing else. It doesn't reconnect to the server.

>
> What about previous messages (Started, Connected, End of Stream),
> especially for the cycle before the problem? Do you get this?:
>
> Started
> Connected
> End of Stream
> Disconnected
> ...
> Connected
> End of Stream
> Disconnected
> Connected
> Disconnected
>
> Or this?:
>
> Started
> Connected
> End of Stream
> Disconnected
> ...
> Connected
> End of Stream
> Disconnected
> Disconnected
>
> Or is there no previous (successful) cycle?
>
> I'm thinking that maybe a listener is throwing an unchecked exception.
>


It cycles fine until it hangs. All three sites stop within a few
seconds of each other so it is something that happens on the server end
I think. But I have no idea what it could be.

--

Knute Johnson
 
Reply With Quote
 
Knute Johnson
Guest
Posts: n/a
 
      03-02-2012
On 3/1/2012 2:15 PM, Martin Gregorie wrote:
> On Thu, 01 Mar 2012 11:49:30 -0800, Knute Johnson wrote:
>
>> I'm having a problem in some production code that I can't figure out.
>> I'll post the complete actual code below. This code is running in three
>> places and has the same problem in two of them at the same time. The
>> other I'm not sure, it may be that the personnel operating it are
>> restarting the program and so don't complain. This piece of code is a
>> simple client that connects via a Socket to a server. The server
>> supplies some data and the client reads that data and files it away. It
>> is supposed to restart itself if there is a connection failure or fault
>> for whatever reason. The problem is that at some random point in time
>> the Socket disconnects, the code logs the disconnect but never restarts.
>> It does print the "SportsWinClient Disconnected" message but never
>> executes the "fireconnectionEvent()" method after creating a new Socket.
>> It doesn't print any Exception message. I'm not sure how it gets out
>> of the try block without printing the "End of Stream" message or an
>> exception message.
>>
>> The crazy part is that all night long when there is no activity from the
>> server it times out and restarts with no problems.
>>

> I notice that InetAddress.getByName() can throw UnknownHostException (and
> this exception isn't explicitly handled. Is it possible that this call
> can time out when the network is busy?


That's possible I guess but I would think it would still throw an
UnknownHostException. Also, since I pass it a string with the IP
address specified with digits, it doesn't have to do any DNS lookup.

> However, as UnknownHostException is a subclass of IOException I don't see
> how it avoids being caught, unless a timeout causes a null to be
> returned: the description of getByName() isn't at all clear about when a
> null is returned instead of throwing an exception. Pulling this out of
> the new Socket() statement so it can be tested for a null return would at
> least show whether this is happening.
>
> Also, I don't see why you're using InetAddress to pass what is apparently
> a remote address to the Socket constructor as localAddress, or why you're
> using port zero as the local port.


The computer has two NICs. The LOCAL_IP is the address of the NIC that
I want to use to connect to the server. Having to use InetAddress is a
limitation of the Socket constructor. Port 0 is the ephemeral port. It
allows the socket implementation to select any available port for the
client end. I'm not sure how InetAddress.getByName() can return a null.

--

Knute Johnson
 
Reply With Quote
 
Knute Johnson
Guest
Posts: n/a
 
      03-02-2012
On 3/1/2012 2:08 PM, markspace wrote:
> On 3/1/2012 11:49 AM, Knute Johnson wrote:
>> I'm having a problem in some production code that I can't figure out.

>
>
> This code below strikes me as possibly being in the wrong order. If you
> interrupt a thread, it should bail. But if you close the socket, it
> should do the same thing, and close the socket too (I'm pretty sure
> there's a handshake for TCP for "close me".)


Interrupting a thread just sets a flag. It doesn't do anything unless
it runs across a method call that will throw an InterruptedException
(ie. Thread.sleep()).

> If you interrupt the thread, then close the socket, the "close" might
> never happen. I'd *just* close the socket, if that passes your testing.
> Try to let the thread actually execute the close, and then unwind
> naturally.
>
> (I didn't look to see if your thread/runnable has other places it waits
> besides the socket. If it does, those should be eliminated, replaced
> with other sockets which are also closed, etc.)


Not sure what you mean here. The thread waits on the BufferedReader but
closing the socket closes the stream which should cause an IOException.
The server end closing the stream should cause the read to return null
and the code should print the End of Stream message which it doesn't so
I don't think that is where it is getting hung up.

--

Knute Johnson
 
Reply With Quote
 
markspace
Guest
Posts: n/a
 
      03-02-2012
On 3/1/2012 4:48 PM, Knute Johnson wrote:

> Interrupting a thread just sets a flag. It doesn't do anything unless it
> runs across a method call that will throw an InterruptedException (ie.
> Thread.sleep()).


I seem to recall that interrupt() would also cause many types of IO
operations to abort and exit. I don't know for sure and I didn't double
check it.

If it happens at the same time for all clients, have you tried running a
client locally on your own system so you can observe it directly? Can
you run a network trace at the same time? Can you tell what was being
sent over the wire immediately before the clients hang? (Put
data/commands into a circular buffer then dump the buffer manually.)

What OS does the server run on? What info/statistics have you tried
looking at after a hang on the server?


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: socket.unbind or socket.unlisten? - socket.error: (48, 'Addressalready in use') Steve Holden Python 1 02-03-2009 06:20 AM
Re: socket.unbind or socket.unlisten? - socket.error: (48, 'Addressalready in use') Steve Holden Python 0 02-01-2009 12:45 PM
Re: socket.unbind or socket.unlisten? - socket.error: (48, 'Addressalready in use') Laszlo Nagy Python 0 02-01-2009 07:37 AM
socket.unbind or socket.unlisten? - socket.error: (48, 'Addressalready in use') Laszlo Nagy Python 1 01-27-2009 05:05 PM
Re: socket.unbind or socket.unlisten? - socket.error: (48,'Address already in use') Jean-Paul Calderone Python 0 01-27-2009 01:41 PM



Advertisments