Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Quick compression

Reply
Thread Tools

Quick compression

 
 
kk
Guest
Posts: n/a
 
      02-09-2008
Hello everybody!
I want to build a distributed application, and I want to compress the
packets that are sent between the system's nodes.

Is there a class that I could use that provides quick compression/
decompression?
 
Reply With Quote
 
 
 
 
Arne Vajh°j
Guest
Posts: n/a
 
      02-09-2008
kk wrote:
> I want to build a distributed application, and I want to compress the
> packets that are sent between the system's nodes.
>
> Is there a class that I could use that provides quick compression/
> decompression?


One obvious solution was the classes in java.util.zip.

Arne
 
Reply With Quote
 
 
 
 
rossum
Guest
Posts: n/a
 
      02-10-2008
On Sat, 9 Feb 2008 15:37:55 -0800 (PST), kk <(E-Mail Removed)> wrote:

>Hello everybody!
>I want to build a distributed application, and I want to compress the
>packets that are sent between the system's nodes.
>
>Is there a class that I could use that provides quick compression/
>decompression?

How big are the packets? How many possible different packets are
there?

For large packets then Zip might well be useful as Arne said. For a
reasonably small number of possible different packets then a code book
would suffice:

1 -> armadillo
2 -> aardvark
3 -> pangolin
etc.

You could also use a code book if a high percentage of the packets are
taken from a small set of all possible packets. In this case you need
to be sure that all allowed codes cannot be mistaken for one of the
uncommon packets.

For a lot of small packets then you have a difficult problem.

rossum

 
Reply With Quote
 
Roedy Green
Guest
Posts: n/a
 
      02-10-2008
On Sat, 9 Feb 2008 15:37:55 -0800 (PST), kk <(E-Mail Removed)> wrote,
quoted or indirectly quoted someone who said :

>
>Is there a class that I could use that provides quick compression/
>decompression?


see http://mindprod.com/jgloss/compression.html

See http://mindprod.com/applet/fileio.html to teach you how to add
compression to any file operation.
--

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com
 
Reply With Quote
 
kk
Guest
Posts: n/a
 
      02-10-2008
Thank you all for your answers!
I use udp packets (<1 kb) (in a similar way that rossum described) to
regulate the way that the nodes interact, but at certain times I need
to exchange some data between two nodes. The amount of data that I may
need to transfer could be from 1 kb to 1mb (pure text), and I was
wondering weather there was a quicker way (from zip) for the
compression/decompression.
A friend of mine recommended jzlib (http://www.jcraft.com/jzlib/) but
I will look first at what Roedy suggested.

Thanks!
 
Reply With Quote
 
Arne Vajh°j
Guest
Posts: n/a
 
      02-10-2008
kk wrote:
> A friend of mine recommended jzlib (http://www.jcraft.com/jzlib/) but


If you read the page then you can see that it basically provides
the same functionality as java.util.zip just adding some more
flexibility.

Unless you are absolutely sure that you need that extra flexibility
then you should stick with the standard.

Arne
 
Reply With Quote
 
Roedy Green
Guest
Posts: n/a
 
      02-11-2008
On Sun, 10 Feb 2008 08:30:36 -0800 (PST), kk <(E-Mail Removed)> wrote,
quoted or indirectly quoted someone who said :

>I use udp packets (<1 kb) (in a similar way that rossum described) to
>regulate the way that the nodes interact, but at certain times I need
>to exchange some data between two nodes. The amount of data that I may
>need to transfer could be from 1 kb to 1mb (pure text), and I was
>wondering weather there was a quicker way (from zip) for the
>compression/decompression.


Compression can't get much traction unless it has a fair size chunk to
work on. There is not as much repetition to find within an tiny
packet.

You might want to look into some type of dictionary compression where
you convert tokens to ints and DON'T transmit the meaning of the
tokens with each packet, but rather pre-transmit an entire dictionary
of what they mean.

In the simplest case, if your packets consisted of words separated by
a single space, create a dictionary of all the words you ever use,
sorted by frequency. Then assign numbers. The low number can get
8-bit codes, the next lowest 16-bit, the next lowest 24-bit.

Then encode your message as a string of ints. Each word includes the
space.

You can then turn GZIP compress on top of that.

see http://mindprod.com/project/supercompressor.html
--

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com
 
Reply With Quote
 
Arne Vajh°j
Guest
Posts: n/a
 
      02-12-2008
Roedy Green wrote:
> On Sun, 10 Feb 2008 08:30:36 -0800 (PST), kk <(E-Mail Removed)> wrote,
> quoted or indirectly quoted someone who said :
>> I use udp packets (<1 kb) (in a similar way that rossum described) to
>> regulate the way that the nodes interact, but at certain times I need
>> to exchange some data between two nodes. The amount of data that I may
>> need to transfer could be from 1 kb to 1mb (pure text), and I was
>> wondering weather there was a quicker way (from zip) for the
>> compression/decompression.

>
> Compression can't get much traction unless it has a fair size chunk to
> work on. There is not as much repetition to find within an tiny
> packet.


1 KB is enough to give reasonable compression with ZIP in most cases.

> You might want to look into some type of dictionary compression where
> you convert tokens to ints and DON'T transmit the meaning of the
> tokens with each packet, but rather pre-transmit an entire dictionary
> of what they mean.
>
> In the simplest case, if your packets consisted of words separated by
> a single space, create a dictionary of all the words you ever use,
> sorted by frequency. Then assign numbers. The low number can get
> 8-bit codes, the next lowest 16-bit, the next lowest 24-bit.


I serious doubt that would create better compression than ZIP in
general cases like english and source code.

> You can then turn GZIP compress on top of that.


It is very rarely a good idea to double compress.

Or to put another way: one definition of a good compression
algorithm is that compressing the output with another algorithm
will not shrink it additionally.

> see http://mindprod.com/project/supercompressor.html


#Ordinary ZIP (Lempel, Ziv Welch) compression

ZIP is LZ77

LZW is LZ78

ZIP is not LZW

Arne
 
Reply With Quote
 
Roedy Green
Guest
Posts: n/a
 
      02-12-2008
On Mon, 11 Feb 2008 21:49:55 -0500, Arne Vajh°j <(E-Mail Removed)>
wrote, quoted or indirectly quoted someone who said :

>I serious doubt that would create better compression than ZIP in
>general cases like english and source code.


Obviously you can to better. Zip has to pack the dictionary with each
packet. I did some experiments on this compacting HTML this way some
years ago. It does buy you much for big documents, but it would for
little chunks.
--

Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com
 
Reply With Quote
 
Christian
Guest
Posts: n/a
 
      02-12-2008
Roedy Green schrieb:
> On Mon, 11 Feb 2008 21:49:55 -0500, Arne Vajh°j <(E-Mail Removed)>
> wrote, quoted or indirectly quoted someone who said :
>
>> I serious doubt that would create better compression than ZIP in
>> general cases like english and source code.

>
> Obviously you can to better. Zip has to pack the dictionary with each
> packet. I did some experiments on this compacting HTML this way some
> years ago. It does buy you much for big documents, but it would for
> little chunks.
> --
>
> Roedy Green Canadian Mind Products
> The Java Glossary
> http://mindprod.com


Why should it have to pass the dictionary?
Dictionary is build up online while decompressing.

If you don't use such a dictionary method you would have to create a
table with probabilitys of characters and words (like in Huffman) and
transmit that. Or what ever compression you want otherwise you could not
make use of letters or words coming more often than others.

Christian
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Adding quick-release to a non-quick-release tripod head ste7esmith@hotmail.com Digital Photography 4 11-20-2006 03:19 PM
Quick question, hopefully quick answer. ~misfit~ NZ Computing 114 01-06-2005 01:36 PM
Quick Question Quick Answer JKop C++ 11 05-24-2004 09:46 PM
Quick Restore for a Compaq not so quick! Croos Bustamunky Computer Support 2 05-15-2004 04:17 AM
PanasonicBQ390 "quick" charger - How quick? Ol' Bab Digital Photography 1 01-17-2004 06:54 AM



Advertisments