Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Reading binary socket data and copying to another socket ?

Reply
Thread Tools

Reading binary socket data and copying to another socket ?

 
 
swebb99@gmail.com
Guest
Posts: n/a
 
      11-13-2007
Hi,

I need to write a java process that listens on a socket for a client
sending binary data (actually compressed messages). This socket will
always be open.

This data will then be written to another socket the other end of
which understands the content of the data. The problem I have is that
the data can be sent at any time and there is nothing in the data (no
characters) I can use to indicate the end of a raw message. So how can
I read the original data and decide that it should now be copied to
the out bound socket ?

If I call read(buff[]) then that blocks until the required number of
bytes is received which doesn't help me as I don't know how many bytes
there are, it could be 1 it could be 1000+. I noticed the socket class
has a setSOTimeout() method which might be useful but could I rely on
a timeout for detecting the end of the binary data associated with a
message !

Any idea's how I can cope with this ?

Thanks for any help

Steve

 
Reply With Quote
 
 
 
 
Gordon Beaton
Guest
Posts: n/a
 
      11-13-2007
On Tue, 13 Nov 2007 17:12:36 -0000, http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> I need to write a java process that listens on a socket for a client
> sending binary data (actually compressed messages). This socket will
> always be open.
>
> This data will then be written to another socket the other end of
> which understands the content of the data. The problem I have is
> that the data can be sent at any time and there is nothing in the
> data (no characters) I can use to indicate the end of a raw message.
> So how can I read the original data and decide that it should now be
> copied to the out bound socket ?


If there is truly *nothing* in the data that tells you whether it
should be copied to the outbound socket, then nobody here can help
you. But from your description here is sounds like *all* data should
be copied to the outbound socket. Is that the case?

If it's important for you to handle "messages" individually, but the
protocol does not provide information about message boundaries, then
you are out of luck. Complain to the protocol designer, and mention
that TCP has no concept of message boundaries.

> If I call read(buff[]) then that blocks until the required number of
> bytes is received which doesn't help me as I don't know how many
> bytes there are, it could be 1 it could be 1000+. I noticed the
> socket class has a setSOTimeout() method which might be useful but
> could I rely on a timeout for detecting the end of the binary data
> associated with a message !


You cannot rely on timing as a way to separate messages.

If you call read(buf) you will only block until *some* data arrives
(often less than the requested amount), or until the socket is closed
at the other end, causing EOF at your end.

If you don't want to block at all, then you should have a look at
java.nio.channels.Selector for non-blocking mechanisms. A Selector can
tell you when the socket can be read without blocking.

/gordon

--
 
Reply With Quote
 
 
 
 
Owen Jacobson
Guest
Posts: n/a
 
      11-13-2007
On Nov 13, 9:12 am, "(E-Mail Removed)" <(E-Mail Removed)> wrote:
> Hi,
>
> I need to write a java process that listens on a socket for a client
> sending binary data (actually compressed messages). This socket will
> always be open.
>
> This data will then be written to another socket the other end of
> which understands the content of the data. The problem I have is that
> the data can be sent at any time and there is nothing in the data (no
> characters) I can use to indicate the end of a raw message. So how can
> I read the original data and decide that it should now be copied to
> the out bound socket ?
>
> If I call read(buff[]) then that blocks until the required number of
> bytes is received which doesn't help me as I don't know how many bytes
> there are, it could be 1 it could be 1000+. I noticed the socket class
> has a setSOTimeout() method which might be useful but could I rely on
> a timeout for detecting the end of the binary data associated with a
> message !
>
> Any idea's how I can cope with this ?


It sounds like you have an app that proxies communication between two
endpoints but does nothing with the communication directly. If that's
the case, then the intermediate app doesn't need to care about message
boundaries: it just needs to copy every byte from the input stream to
the output stream. This will not fragment or damage messages any more
than TCP itself does, and will be transparent to correctly-written
endpoints (that make no assumptions about message boundaries with
respect to read calls).

At a high level, then, all you need is this:

prepare a buffer
while there is data:
read input into the buffer
write buffer to output

Hope that helps,
Owen

 
Reply With Quote
 
swebb99@gmail.com
Guest
Posts: n/a
 
      11-13-2007
On 13 Nov, 18:30, Owen Jacobson <(E-Mail Removed)> wrote:
> On Nov 13, 9:12 am, "(E-Mail Removed)" <(E-Mail Removed)> wrote:
>
>
>
> > Hi,

>
> > I need to write a java process that listens on a socket for a client
> > sending binary data (actually compressed messages). This socket will
> > always be open.

>
> > This data will then be written to another socket the other end of
> > which understands the content of the data. The problem I have is that
> > the data can be sent at any time and there is nothing in the data (no
> > characters) I can use to indicate the end of a raw message. So how can
> > I read the original data and decide that it should now be copied to
> > the out bound socket ?

>
> > If I call read(buff[]) then that blocks until the required number of
> > bytes is received which doesn't help me as I don't know how many bytes
> > there are, it could be 1 it could be 1000+. I noticed the socket class
> > has a setSOTimeout() method which might be useful but could I rely on
> > a timeout for detecting the end of the binary data associated with a
> > message !

>
> > Any idea's how I can cope with this ?

>
> It sounds like you have an app that proxies communication between two
> endpoints but does nothing with the communication directly. If that's
> the case, then the intermediate app doesn't need to care about message
> boundaries: it just needs to copy every byte from the input stream to
> the output stream. This will not fragment or damage messages any more
> than TCP itself does, and will be transparent to correctly-written
> endpoints (that make no assumptions about message boundaries with
> respect to read calls).
>
> At a high level, then, all you need is this:
>
> prepare a buffer
> while there is data:
> read input into the buffer
> write buffer to output
>
> Hope that helps,
> Owen


Yes thats exactly my problem Basically all I am doing is taking a
binary stream of data that is actually meaningful messages to other
clients/servers and wrapping the binary in another protocol for one
hop of the connection so it can get through a firewall in a known
protocol. Then once through I have another process that strips the
binary back out and sends it on to the final destination. So basically
I don't know the contents of the binary data. I just know that I have
to read it, wrap it, unwrap it again and then pass it on again as
binary. The socket from the source of the message to my first process
should stay open all the time. The socket however from my 2nd process
to the destination is supposed to disconnect after 60secs of idle (may
use setSOTimeout for this).

I thought that if I used read(data[x]) the read would block until the
socket goes down or x bytes were received, is this not the case ? If
it is how cn I write the while loop without potentially blocking on
the last bit of data ?

Thanks for the help

Steve

 
Reply With Quote
 
Roedy Green
Guest
Posts: n/a
 
      11-14-2007
On Tue, 13 Nov 2007 17:12:36 -0000, "(E-Mail Removed)"
<(E-Mail Removed)> wrote, quoted or indirectly quoted someone who
said :

>I can use to indicate the end of a raw message.


either you put a count on a message, or you reserve some end marker
that can't exist naturally in the message.

One scheme for short messages reserves the byte 0 as an end marker.
Any accidental 0 is replaced by the offset to the next accidental 0.
The byte you reserve is not actually 0, but a rarely used one.

With lots of RAM a buffering up the message before you send it makes
the count method easiest.
--
Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com
 
Reply With Quote
 
Esmond Pitt
Guest
Posts: n/a
 
      11-14-2007
(E-Mail Removed) wrote:
> I thought that if I used read(data[x]) the read would block until the
> socket goes down or x bytes were received, is this not the case ?


No. It blocks until at least 1 byte is received.
 
Reply With Quote
 
Gordon Beaton
Guest
Posts: n/a
 
      11-14-2007
On Tue, 13 Nov 2007 20:54:10 -0000, (E-Mail Removed) wrote:
> Basically all I am doing is taking a binary stream of data that is
> actually meaningful messages to other clients/servers and wrapping
> the binary in another protocol for one hop of the connection so it
> can get through a firewall in a known protocol. Then once through I
> have another process that strips the binary back out and sends it on
> to the final destination.


If you can use SSH through your firewall, then this can be solved
without writing any code, using the port redirection feature (of
OpenSSL for example).

/gordon

--
 
Reply With Quote
 
swebb99@gmail.com
Guest
Posts: n/a
 
      11-14-2007
On 14 Nov, 11:55, Gordon Beaton <(E-Mail Removed)> wrote:
> On Tue, 13 Nov 2007 20:54:10 -0000, (E-Mail Removed) wrote:
> > Basically all I am doing is taking a binary stream of data that is
> > actually meaningful messages to other clients/servers and wrapping
> > the binary in another protocol for one hop of the connection so it
> > can get through a firewall in a known protocol. Then once through I
> > have another process that strips the binary back out and sends it on
> > to the final destination.

>
> If you can use SSH through your firewall, then this can be solved
> without writing any code, using the port redirection feature (of
> OpenSSL for example).
>
> /gordon
>
> --


Thanks for that but the stuff I'm working on has its own protocol that
is allow through the firewall (via various registration routines) so
the data can only be packaged in the payload of messages within this
protocol.

I'm working on the code at the moment and will be testing in the next
few days so will report back when I get somewhere with it.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
psycopg2 insertion and reading binary data to PostgreSQL database(bytea datatype) romap@libero.it Python 0 03-01-2011 10:57 AM
Fast reading and unpacking of binary data (struct module) Daniel Platz Python 2 07-22-2009 11:12 AM
Re: How to write binary data to file and socket Steve Holden Python 0 12-12-2008 01:22 PM
Suggestions for reading binary data from a connected socket. My Name Java 9 07-21-2004 05:40 PM
URGENT: Reading from socket and writing into binary... HOW? Louis Java 6 10-15-2003 07:30 PM



Advertisments