Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > NIO read() SocketChannel EOF End of Stream

Reply
Thread Tools

NIO read() SocketChannel EOF End of Stream

 
 
Fritz Bayer
Guest
Posts: n/a
 
      07-12-2004
Hello,

I have read in the javadoc of the NIO API that the read() method of
SockeChannel returns a -1 when the end of stream is reached.

I can confirm this. After a connection has been idle for a while and
no data has been sent successive calls to the read() method of
SocketChannel yield a -1.

My question now is what the -1 excatly means. I know that it for sure
means that I can't read any data from the SocketChannel anymore. So
far, so good.

However, I'm not certain what it means for the output channel. Can I
still write data using the put() method to the SocketChannel?

Does the the -1 signal a closed connection in both directions? Or does
it only mean that you can't read anymore but can still write data?

The later would also mean that a -1 does not signal a closed
connection in the sense that there is not network connection at all
anymore.

It would be great if a -1 would signal a close of the connection in
both directions, but I'm not sure about it and it makes a big
difference.

What's you opinion on this?

Fritz
 
Reply With Quote
 
 
 
 
iksrazal
Guest
Posts: n/a
 
      07-13-2004
http://www.velocityreviews.com/forums/(E-Mail Removed) (Fritz Bayer) wrote in message news:<(E-Mail Removed). com>...
> Hello,
>
> My question now is what the -1 excatly means. I know that it for sure
> means that I can't read any data from the SocketChannel anymore. So
> far, so good.
>
> However, I'm not certain what it means for the output channel. Can I
> still write data using the put() method to the SocketChannel?
>
> Does the the -1 signal a closed connection in both directions? Or does
> it only mean that you can't read anymore but can still write data?
>


Step back a second and think not about a single SocketChannel, but
that by using a Selector with a SocketChannel, you can create other
helper objects from the java API such as ReadableByteChannel. You then
close only the ReadableByteChannel instance on EOF.

For example, I have a cache of open client socket connections. They do
read and write operations. On a read, I pass the socket channel
instance to a method that does something like the code below, allowing
for subsequent other read/write operations:

public static String readFromChannel (SocketChannel sChannel,
final int size) throws IOException
{
int numBytes = -1;
boolean hasRead = false;
CharsetDecoder decoder = ascii.newDecoder();
ByteBuffer inBuffer = ByteBuffer.allocateDirect(size);
CharBuffer charBuffer = CharBuffer.allocate(size+100);
StringBuffer dataRead = new StringBuffer();
//Use Selector for activity readiness
Selector selector = Selector.open();
// Register read activity on socket
sChannel.register(selector, SelectionKey.OP_READ);
// Read response with 20000 mililiseconds timeout
//IMPORTANT! this time limit can be overridden by
java.utl.timer,
//set in the invoking class, which therefore could interrupt the
channel
//and cause a read timeout less that 20000 milliseconds
//This behavior is more predictable if this read timeout is
_LESS_
//then the java.util.Timer timeout used by the current thread of
the
//invoking object of this method
while (selector.select(20000) > 0)
{
// Get set of ready objects
Set readyKeys = selector.selectedKeys();
Iterator readyItor = readyKeys.iterator();
// Walk through set
while (readyItor.hasNext())
{
// Get key from set
SelectionKey key = (SelectionKey)readyItor.next();
// Remove current entry
readyItor.remove();
// Get channel
ReadableByteChannel keyChannel =
(SocketChannel)key.channel();
if (key.isReadable())
{
// Read what's ready in response
// Think this little loop is odd? Well it is. The reason
for it
// is because TCP stacks have a tendency to activate with
only 1 byte
// available, and a subsequent read immediately gets the
rest of the data.
// Very annoying, and this leads to us in the common sense
scenario with
// non-blocking to have to inject multiple READ events for
no good reason.
// This little loop tries the read again if it didn't
complete the first
// time around, and provides a noticable performance
boost.
for (int i = 0; ; i++)
{
numBytes += keyChannel.read (inBuffer);
if (numBytes < 0)
{
keyChannel.close();
throw new IOException ("Socket lost connection during
read operation");
}
//loop until buffer full
if (!inBuffer.hasRemaining())
{
//just in case - good for debugging
hasRead = true;
System.out.println("has read");
System.out.println("SUCCESSFUL, length of data read: "
+ numBytes);
// Make buffer readable
inBuffer.flip();
// Decode buffer
decoder.decode(inBuffer, charBuffer, false);
// Make buffer readable
charBuffer.flip();
// Build and return string recieved
return dataRead.append(charBuffer).toString();
}
//one byte trick as described above
if (i >= 2)
{
break;
}
}//end for loop
}//end key.isReadable()
}//end while iterator
}//end while selector

if (false == hasRead)
{
System.err.println("has _NOT_ read");
System.err.println("length of data read: " + numBytes);
System.err.println("length of data expected to read: " +
size);
System.err.println("ERROR, data read: " +
dataRead.append(charBuffer).toString());

throw new IOException ("Socket read operation timed out");
}
else
{
throw new IOException ("Invalid Read Socket state");
}
}


OK so notice I only close the ReadableByteChannel object keyChannel on
an unexpected condition, but that still wouldn't close the
SocketChannel object passed into the method. Its more common in books
like the O'reilly NIO one to get a SelectionKey from a SocketChannel
like I did above, pass that into a method, create another
SocketChannel from that, and then close only that helper SocketChannel
on EOF. You might be able to find a clearer example from googling on
Selector and SocketChannel.

As a side note, I graciously lifted the 'funky for loop' above from
the emberio project liscensed under bsd. The performance is indeed
quite good and works well if you know how many bytes to read. I
suppose with small modifications using a terminating string or simply
an EOF would work as well.

HTH

Outsource to an American programmer living in brazil!
http://www.braziloutsource.com/
iksrazal
 
Reply With Quote
 
 
 
 
iksrazal
Guest
Posts: n/a
 
      07-13-2004
(E-Mail Removed) (Fritz Bayer) wrote in message news:<(E-Mail Removed). com>...
> Hello,
>
> I have read in the javadoc of the NIO API that the read() method of
> SockeChannel returns a -1 when the end of stream is reached.
>
> I can confirm this. After a connection has been idle for a while and
> no data has been sent successive calls to the read() method of
> SocketChannel yield a -1.
>
> My question now is what the -1 excatly means. I know that it for sure
> means that I can't read any data from the SocketChannel anymore. So
> far, so good.
>
> However, I'm not certain what it means for the output channel. Can I
> still write data using the put() method to the SocketChannel?
>
> Does the the -1 signal a closed connection in both directions? Or does
> it only mean that you can't read anymore but can still write data?
>
> The later would also mean that a -1 does not signal a closed
> connection in the sense that there is not network connection at all
> anymore.
>
> It would be great if a -1 would signal a close of the connection in
> both directions, but I'm not sure about it and it makes a big
> difference.
>
> What's you opinion on this?
>
> Fritz


OK, there was a couple of problems with my last post. I guess we both
(hopefully) benefit because I learned something along the way.

1) Closing the ReadableByteChannel in the example code (right before
the return) does indeed close the original SocketChannel. I tried it
and any read/write operations after that will throw
ClosedChannelException. In my case I never recieve an EOF due to the
protocol I'm working with.

2) The counter of numBytes was wrong. Looking at it after the post, it
should be:

int count = 0;
if ((count = keyChannel.read (inBuffer)) < 0) // EOF
{
keyChannel.close();
throw new IOException ("Socket lost connection during read
operation");
}
numBytes += count;

As I said I don't have an easy way to test that. I'll try to test it a
bit later today and I'll post any changes.

HTH

Outsource to an American programmer living in brazil!
http://www.braziloutsource.com/
iksrazal
 
Reply With Quote
 
Fritz Bayer
Guest
Posts: n/a
 
      07-14-2004
(E-Mail Removed) (iksrazal) wrote in message news:<(E-Mail Removed). com>...
> (E-Mail Removed) (Fritz Bayer) wrote in message news:<(E-Mail Removed). com>...
> > Hello,
> >
> > I have read in the javadoc of the NIO API that the read() method of
> > SockeChannel returns a -1 when the end of stream is reached.
> >
> > I can confirm this. After a connection has been idle for a while and
> > no data has been sent successive calls to the read() method of
> > SocketChannel yield a -1.
> >
> > My question now is what the -1 excatly means. I know that it for sure
> > means that I can't read any data from the SocketChannel anymore. So
> > far, so good.
> >
> > However, I'm not certain what it means for the output channel. Can I
> > still write data using the put() method to the SocketChannel?
> >
> > Does the the -1 signal a closed connection in both directions? Or does
> > it only mean that you can't read anymore but can still write data?
> >
> > The later would also mean that a -1 does not signal a closed
> > connection in the sense that there is not network connection at all
> > anymore.
> >
> > It would be great if a -1 would signal a close of the connection in
> > both directions, but I'm not sure about it and it makes a big
> > difference.
> >
> > What's you opinion on this?
> >
> > Fritz

>
> OK, there was a couple of problems with my last post. I guess we both
> (hopefully) benefit because I learned something along the way.
>
> 1) Closing the ReadableByteChannel in the example code (right before
> the return) does indeed close the original SocketChannel. I tried it
> and any read/write operations after that will throw
> ClosedChannelException. In my case I never recieve an EOF due to the
> protocol I'm working with.
>
> 2) The counter of numBytes was wrong. Looking at it after the post, it
> should be:
>
> int count = 0;
> if ((count = keyChannel.read (inBuffer)) < 0) // EOF
> {
> keyChannel.close();
> throw new IOException ("Socket lost connection during read
> operation");
> }
> numBytes += count;
>
> As I said I don't have an easy way to test that. I'll try to test it a
> bit later today and I'll post any changes.
>
> HTH
>
> Outsource to an American programmer living in brazil!
> http://www.braziloutsource.com/
> iksrazal



Thanks for the comments. However, I'm still not sure what the -1
excatly signals. Does it mean we can't read AND write anymore or does
it mean that we can't read BUT still can write data, which would mean
that the network connection is still established?
 
Reply With Quote
 
Gordon Beaton
Guest
Posts: n/a
 
      07-15-2004
On 14 Jul 2004 00:53:51 -0700, Fritz Bayer wrote:
> Thanks for the comments. However, I'm still not sure what the -1
> excatly signals. Does it mean we can't read AND write anymore or
> does it mean that we can't read BUT still can write data, which
> would mean that the network connection is still established?


A TCP connection consists of two independent streams, one in each
direction. If read() indicates EOF, it does not necessarily mean that
the connection has been terminated, although that is often the case.

It may still be possible to write to the remaining stream. Note that
write() can indicate EOF too.

Look up e.g. Socket.shutdownOutput(), or the shutdown() system call.

A typical use of this feature is when a client sends a request to a
server, then uses shutdown() to indicate EOF to the server. The server
can then send the response on the same connection, then use close() to
indicate EOF to the client.

/gordon

--
[ do not email me copies of your followups ]
g o r d o n + n e w s @ b a l d e r 1 3 . s e
 
Reply With Quote
 
iksrazal
Guest
Posts: n/a
 
      07-16-2004
Gordon Beaton <(E-Mail Removed)> wrote in message news:<40f6566c$(E-Mail Removed)>...
> On 14 Jul 2004 00:53:51 -0700, Fritz Bayer wrote:
> > Thanks for the comments. However, I'm still not sure what the -1
> > excatly signals. Does it mean we can't read AND write anymore or
> > does it mean that we can't read BUT still can write data, which
> > would mean that the network connection is still established?

>
> A TCP connection consists of two independent streams, one in each
> direction. If read() indicates EOF, it does not necessarily mean that
> the connection has been terminated, although that is often the case.
>
> It may still be possible to write to the remaining stream. Note that
> write() can indicate EOF too.
>
> Look up e.g. Socket.shutdownOutput(), or the shutdown() system call.
>
> A typical use of this feature is when a client sends a request to a
> server, then uses shutdown() to indicate EOF to the server. The server
> can then send the response on the same connection, then use close() to
> indicate EOF to the client.
>
> /gordon



This clarification actually helped me quite alot with a problem I was
having - thank you. I glanced over the javadoc of shutdownOutput() but
failed to realize its significance. It incidently led led to a very
interesting relationship between java.net.Socket and
java.nio.SocketChannel I was not aware of.

My problem is that I am caching open socketChannels for re-use on the
client side. The protocol I'm working with allows it. However, calling
SocketChannel.close() somehow still left the server thinking the
connection was open. Your explanation led to this code which suprised
me:

SocketChannel target = (SocketChannel) socketCache.get(ii);
//ArrayList
try
{
if (null != target)
{
Socket socket = target.socket();
socket.shutdownOutput();
socket.shutdownInput();
socket.close();
target.close();
target = null;
socket = null;
}
}

I was thinking SocketChannel and nio was completely seperate from the
old Socket class. Guess again.

As a side note, the code I posted before had a nasty file descriptor
leak due to not closing the Selector. Even the shutdownOutput() code
above didn't fix it. Here's the revised code in case anyone googles on
it:

public static String readFromChannel (SocketChannel sChannel,
final int size) throws IOException
{
int numBytes = 0;
boolean hasRead = false;
CharsetDecoder decoder = ascii.newDecoder();
ByteBuffer inBuffer = ByteBuffer.allocateDirect(size);
CharBuffer charBuffer = CharBuffer.allocate(size+100);
StringBuffer dataRead = new StringBuffer();
//Use Selector for activity readiness
Selector selector = Selector.open();
// Register read activity on socket
sChannel.register(selector, SelectionKey.OP_READ);
try
{
// Read response with 20000 mililiseconds timeout
//IMPORTANT! this time limit can be overridden by
java.utl.timer,
//set in the invoking class, which therefore could interrupt
the channel
//and cause a read timeout less that 20000 milliseconds
//This behavior is more predictable if this read timeout is
_LESS_
//then the java.util.Timer timeout used by the current thread
of the
//invoking object of this method
while (selector.select(20000) > 0)
{
// Get set of ready objects
Set readyKeys = selector.selectedKeys();
Iterator readyItor = readyKeys.iterator();
// Walk through set
while (readyItor.hasNext())
{
// Get key from set
SelectionKey key = (SelectionKey)readyItor.next();
// Remove current entry
readyItor.remove();
// Get channel
ReadableByteChannel keyChannel =
(SocketChannel)key.channel();
if (key.isReadable())
{
// Read what's ready in response
// Think this little loop is odd? Well it is. The
reason for it
// is because TCP stacks have a tendency to activate
with only 1 byte
// available, and a subsequent read immediately gets the
rest of the data.
// Very annoying, and this leads to us in the common
sense scenario with
// non-blocking to have to inject multiple READ events
for no good reason.
// This little loop tries the read again if it didn't
complete the first
// time around, and provides a noticable performance
boost.
for (int i = 0; ; i++)
{
int count = 0;
if ((count = keyChannel.read (inBuffer)) < 0) // EOF
{
keyChannel.close();
throw new IOException ("Socket lost connection
during read operation");
}
numBytes += count;
//loop until buffer full
if (!inBuffer.hasRemaining())
{
//just in case - good for debugging
hasRead = true;
System.out.println("has read");
System.out.println("SUCCESSFUL, length of data read:
" + numBytes);
// Make buffer readable
inBuffer.flip();
// Decode buffer
decoder.decode(inBuffer, charBuffer, false);
// Make buffer readable
charBuffer.flip();
// VERY IMPORTANT!!! This Selector _MUST_ be
cancelled, or it will
// create a file descriptor leak
selector.close();
// Build and return string recieved
return dataRead.append(charBuffer).toString();
}
//one byte trick as described above
if (i >= 2)
{
break;
}
}//end for loop
}//end key.isReadable()
}//end while iterator
}//end while selector

if (false == hasRead)
{
System.err.println("has _NOT_ read");
System.err.println("length of data read: " + numBytes);
System.err.println("length of data read: " + numBytes);
if (numBytes > 0)
{
// Make buffer readable
inBuffer.flip();
// Decode buffer
decoder.decode(inBuffer, charBuffer, false);
// Make buffer readable
charBuffer.flip();
System.err.println("ERROR, data read: " +
dataRead.append(charBuffer).toString());
}
throw new IOException ("Socket read operation timed out");
}
else
{
throw new IOException ("Invalid Read Socket state");
}
}//end try
finally
{
// Guarantee the closure of the Selector, preventing a file
descriptor leak
// If already closed this will have no effect. If a "too many
open files"
// error occurs, a linux 'lsof | wc -l' command should show a
growing
// list of files open, possibly indicating an unclosed
selector with Keys
// that have not been 'deregistered' according to the Selector
javadocs
// Note that a simple SocketChannel.close() will _not_
deregister the keys
selector.close();
}
}

iksrazal
 
Reply With Quote
 
Fritz Bayer
Guest
Posts: n/a
 
      07-17-2004
Gordon Beaton <(E-Mail Removed)> wrote in message news:<40f6566c$(E-Mail Removed)>...
> On 14 Jul 2004 00:53:51 -0700, Fritz Bayer wrote:
> > Thanks for the comments. However, I'm still not sure what the -1
> > excatly signals. Does it mean we can't read AND write anymore or
> > does it mean that we can't read BUT still can write data, which
> > would mean that the network connection is still established?

>
> A TCP connection consists of two independent streams, one in each
> direction. If read() indicates EOF, it does not necessarily mean that
> the connection has been terminated, although that is often the case.
>
> It may still be possible to write to the remaining stream. Note that
> write() can indicate EOF too.
>
> Look up e.g. Socket.shutdownOutput(), or the shutdown() system call.
>
> A typical use of this feature is when a client sends a request to a
> server, then uses shutdown() to indicate EOF to the server. The server
> can then send the response on the same connection, then use close() to
> indicate EOF to the client.
>
> /gordon


I'm writting a HTTP proxy and for this purpose I trying to figure out
what the -1 really means.

Until now I have been closing the connection of the client when I read
a -1. If the above said would be true, I could not close the
connection then, because the client could send a minus -1 after a HTTP
"Connection Header: Close", which signals the last HTTP Transaction.
If then close the connection I could not write the reponse from the
server back to the client.

But how could I ever notice a connection close by the client, if the
-1 does not signal a close? I mean then I would have no way of knowing
if the connection is established or not?
 
Reply With Quote
 
Gordon Beaton
Guest
Posts: n/a
 
      07-18-2004
On 17 Jul 2004 02:10:38 -0700, Fritz Bayer wrote:
> I'm writting a HTTP proxy and for this purpose I trying to figure
> out what the -1 really means.
>
> Until now I have been closing the connection of the client when I
> read a -1. If the above said would be true, I could not close the
> connection then, because the client could send a minus -1 after a
> HTTP "Connection Header: Close", which signals the last HTTP
> Transaction. If then close the connection I could not write the
> reponse from the server back to the client.


I fail to see how your comments show that what I wrote isn't true. Of
course if you close the connection, you won't be able to write the
subsequent response.

If read() indicates EOF, it indicates end of file on that particular
stream only (which will be the case if shutdown() was used). If the
connection was closed, then both streams will be at EOF, and also
write() will indicate EOF.

> But how could I ever notice a connection close by the client, if the
> -1 does not signal a close? I mean then I would have no way of
> knowing if the connection is established or not?


As I said in my earlier response: if read() signals EOF, the client
may have used shutdown (e.g. Socket.shutdownOutput()). That does not
terminate the connection, it only terminates his outgoing stream. If
that's the case, you can still write in the reverse direction.

If the client sent "Connection Header: Close", then close the
connection *after* sending the response.

If the client actually closed the connection after sending the
request, then you can only assume that he isn't interested in the
reqponse. You'll detect that when you attempt to write.

/gordon

--
[ do not email me copies of your followups ]
g o r d o n + n e w s @ b a l d e r 1 3 . s e
 
Reply With Quote
 
Fritz Bayer
Guest
Posts: n/a
 
      07-18-2004
Gordon Beaton <(E-Mail Removed)> wrote in message news:<40fa33b4$(E-Mail Removed)>...
> On 17 Jul 2004 02:10:38 -0700, Fritz Bayer wrote:
> > I'm writting a HTTP proxy and for this purpose I trying to figure
> > out what the -1 really means.
> >
> > Until now I have been closing the connection of the client when I
> > read a -1. If the above said would be true, I could not close the
> > connection then, because the client could send a minus -1 after a
> > HTTP "Connection Header: Close", which signals the last HTTP
> > Transaction. If then close the connection I could not write the
> > reponse from the server back to the client.

>
> I fail to see how your comments show that what I wrote isn't true. Of
> course if you close the connection, you won't be able to write the
> subsequent response.
>
> If read() indicates EOF, it indicates end of file on that particular
> stream only (which will be the case if shutdown() was used). If the
> connection was closed, then both streams will be at EOF, and also
> write() will indicate EOF.
>
> > But how could I ever notice a connection close by the client, if the
> > -1 does not signal a close? I mean then I would have no way of
> > knowing if the connection is established or not?

>
> As I said in my earlier response: if read() signals EOF, the client
> may have used shutdown (e.g. Socket.shutdownOutput()). That does not
> terminate the connection, it only terminates his outgoing stream. If
> that's the case, you can still write in the reverse direction.
>


This actually seems not to be correct. I checked it out by putting
some assert statements into the code. You might wanna try this as
well:

if (bytesRead == -1)
{
assert !socketChannel.isConnected() : " The client's SocketChannel
is still connected!" ;

assert socketChannel.socket().isInputShutdown() : " The client's
Input Stream is still open";

assert socketChannel.socket().isOutputShutdown() : " The client's
Output Stream is still open";
}

The last assert never fails. This confirms what you have said before:
A EOF does not necessarily shutdown the output stream.

This is also confirmed by the second assert statement. I sometimes
failes - the connection is still there.

However, now something really unexpected happens. Also the first
assert statement sometimes fails!

So I get a -1 from the SocketChannel.read() and then afterwards
calling socketChannel.socket().isInputShutdown() returns false!

What do you think? How can this be - it also is to the contrary of
what you have explained before?


> If the client sent "Connection Header: Close", then close the
> connection *after* sending the response.
>
> If the client actually closed the connection after sending the
> request, then you can only assume that he isn't interested in the
> reqponse. You'll detect that when you attempt to write.
>
> /gordon

 
Reply With Quote
 
Gordon Beaton
Guest
Posts: n/a
 
      07-19-2004
On 18 Jul 2004 13:34:47 -0700, Fritz Bayer wrote:
> This actually seems not to be correct. I checked it out by putting
> some assert statements into the code. You might wanna try this as
> well:
>
> if (bytesRead == -1)
> {
> assert !socketChannel.isConnected() : " The client's SocketChannel
> is still connected!" ;
>
> assert socketChannel.socket().isInputShutdown() : " The client's
> Input Stream is still open";
>
> assert socketChannel.socket().isOutputShutdown() : " The client's
> Output Stream is still open";
> }
>
> The last assert never fails. This confirms what you have said
> before: A EOF does not necessarily shutdown the output stream.
>
> This is also confirmed by the second assert statement. I sometimes
> failes - the connection is still there.
>
> However, now something really unexpected happens. Also the first
> assert statement sometimes fails!
>
> So I get a -1 from the SocketChannel.read() and then afterwards
> calling socketChannel.socket().isInputShutdown() returns false!


Don't be confused by the behaviour of isConnected() and
isInputShutdown(). They don't actually say anything about the state of
the underlying connection, they only tell you about the state of the
Socket object itself, i.e. whether connect() or shutdownInput() have
been called.

In particular, these methods won't tell you what the application at
the *other* end of the connection has done with his socket.

Have a look at the source code for the Socket class, it's provided
with the JDK.

> What do you think? How can this be - it also is to the contrary of
> what you have explained before?


It is *only* by reading or writing that you can determine whether it
is possible to read or write. That's the nature of TCP, and it has
nothing to do with Java.

/gordon

--
[ do not email me copies of your followups ]
g o r d o n + n e w s @ b a l d e r 1 3 . s e
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
NIO, SocketChannel and packet processing Qu0ll Java 10 11-12-2007 07:59 PM
if EOF = -1, can't a valid character == EOF and cause problems? Kobu C Programming 10 03-04-2005 10:40 PM
Telnet and EOF with SocketChannel.read() Fritz Bayer Java 1 12-08-2004 11:37 AM
Writting data to a SocketChannel using NIO Fritz Bayer Java 3 11-14-2004 11:28 PM
NIO with timeouts != NIO? iksrazal Java 1 06-18-2004 02:28 PM



Advertisments