Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Ruby Newbie Problems with deflate, base64...

Reply
Thread Tools

Ruby Newbie Problems with deflate, base64...

 
 
Pat Patterson
Guest
Posts: n/a
 
      03-13-2007
I'm implementing a spec that calls for messages to be deflated, then
base64 encoded, then URL encoded, so they can be passed as parameters to
an HTTP GET. In PHP, this is achieved with:

$encodedMsg = urlencode( base64_encode( gzdeflate( $msg ) ) )

Looking at the docs, the equivalent in Ruby would be something like:

require "cgi"
require "base64"
require "zlib"

encodedMsg = CGI::escape( Base64.encode64( Zlib:eflate.deflate( msg ) ) )

But this gives me completely different output to my (working) PHP model.

Some googling later, it appears that Zlib:eflate.deflate prepends and
appends 'stuff' to the deflated data. Investigating, the PHP

$deflated = gzdeflate( "Hello world" );
echo bin2hex( $deflated );

Gives me

f348cdc9c95728cf2fca490100

While the Ruby

deflated = Zlib:eflate.deflate( "Hello world" )
myhex = ""
1.upto(deflated.length) { |i| myhex << "%02x" % deflated[i] }
puts myhex

Shows

9cf348cdc9c95728cf2fca49010018ab043d00

(BTW - if anyone knows a more succinct way to hex encode a string in
Ruby, that would be useful)

Notice the additional '9c' at the start of the string and '18ab043d00'
at the end. OK - there seems to be a consistent amount of data prepended
and appended, so I can slice those off.

But now Base 64 is acting weird. In PHP:

$base64Encoded = base64_encode( $deflated );
echo $base64Encoded;

Shows

80jNyclXKM8vykkBAA==

While, in Ruby

# Remove extra stuff from deflated string
deflated = deflated[1,deflated.length-6]
base64encoded = Base64.encode64( deflated )
puts base64encoded

Shows

nPNIzcnJVyjPL8pJAQ==

Completely different! Now, if I feed that back through PHP's
base64_unencode, I get

9cf348cdc9c95728cf2fca4901

Notice, again '9c' prepended, but this time the trailing '00' has been
removed.

What is going on here? Any ideas??? Am I missing something really
obvious about string handling in Ruby?

(BTW - this is on ruby 1.8.4 (2005-12-24) [i486-linux])

Cheers,

Pat

--
Pat Patterson - http://www.velocityreviews.com/forums/(E-Mail Removed)
Federation Architect,
Sun Microsystems, Inc.
http://blogs.sun.com/superpat


 
Reply With Quote
 
 
 
 
Brian Candler
Guest
Posts: n/a
 
      03-13-2007
On Tue, Mar 13, 2007 at 01:21:52PM +0900, Pat Patterson wrote:
> While the Ruby
>
> deflated = Zlib:eflate.deflate( "Hello world" )
> myhex = ""
> 1.upto(deflated.length) { |i| myhex << "%02x" % deflated[i] }
> puts myhex
>
> Shows
>
> 9cf348cdc9c95728cf2fca49010018ab043d00
>
> (BTW - if anyone knows a more succinct way to hex encode a string in
> Ruby, that would be useful)


str = "\001\377"
puts str.unpack("H*")

 
Reply With Quote
 
 
 
 
Pat Patterson
Guest
Posts: n/a
 
      03-14-2007
OK - so now I know what is happening...

Zlib:eflate.deflate implements ZLIB compression according to RFC 1950.
ZLIB defines a 2 byte header containing a variety of flags and a 4 byte
trailer containing an Adler-32 checksum. Just out of interest, the
header that I'm seeing translates as 'compression method = deflate,
windows size = 32k, no preset dictionary, default compression level',
which makes perfect sense. Deflate compression itself is defined by RFC
1951.

So - if you want 'raw' deflated data (which is called for in many
situations), cutting off the leading 2 and trailing 4 bytes is exactly
what you need to do.

Cheers,

Pat

Pat Patterson wrote:
> Thanks, Brian! That revealed an obvious bug in the code I was using to
> examine the deflated data (should have been 0.upto(deflated.length-1)).
>
> So Base64 /is/ working correctly. Deflate prepends 2 bytes (seems to
> be constant 0x789c for default deflate level) and appends 4 bytes
> (rather than 1 and 5 as I thought) to the deflated data. When I cut
> those off, I can get Ruby to work the same as PHP.
>
> Still - it would be nice if deflate worked the same as on Java, PHP, ...
>
> Cheers,
>
> Pat
>
> Brian Candler wrote:
>> On Tue, Mar 13, 2007 at 01:21:52PM +0900, Pat Patterson wrote:
>>
>>> While the Ruby
>>>
>>> deflated = Zlib:eflate.deflate( "Hello world" )
>>> myhex = ""
>>> 1.upto(deflated.length) { |i| myhex << "%02x" % deflated[i] }
>>> puts myhex
>>>
>>> Shows
>>>
>>> 9cf348cdc9c95728cf2fca49010018ab043d00
>>>
>>> (BTW - if anyone knows a more succinct way to hex encode a string in
>>> Ruby, that would be useful)
>>>

>>
>> str = "\001\377"
>> puts str.unpack("H*")
>>


--
Pat Patterson - (E-Mail Removed)
Federation Architect,
Sun Microsystems, Inc.
http://blogs.sun.com/superpat


 
Reply With Quote
 
eden li
Guest
Posts: n/a
 
      03-14-2007
You can prevent deflate from generating the header by passing in -
MAX_WBITS to the options for Deflate.new. The following method
emulates gzdeflate from php:

def gzdeflate(s)
Zlib:eflate.new(nil, -Zlib::MAX_WBITS).deflate(s, Zlib::FINISH)
end

puts gzdeflate("Hello World").unpack('H*').first
# => f348cdc9c95708cf2fca490100

On Mar 14, 8:16 am, Pat Patterson <(E-Mail Removed)> wrote:
> OK - so now I know what is happening...
>
> Zlib:eflate.deflate implements ZLIB compression according to RFC 1950.
> ZLIB defines a 2 byte header containing a variety of flags and a 4 byte
> trailer containing an Adler-32 checksum. Just out of interest, the
> header that I'm seeing translates as 'compression method = deflate,
> windows size = 32k, no preset dictionary, default compression level',
> which makes perfect sense. Deflate compression itself is defined by RFC
> 1951.
>
> So - if you want 'raw' deflated data (which is called for in many
> situations), cutting off the leading 2 and trailing 4 bytes is exactly
> what you need to do.
>
> Cheers,
>
> Pat
>
>
>
> Pat Patterson wrote:
> > Thanks, Brian! That revealed an obvious bug in the code I was using to
> > examine the deflated data (should have been 0.upto(deflated.length-1)).

>
> > So Base64 /is/ working correctly. Deflate prepends 2 bytes (seems to
> > be constant 0x789c for default deflate level) and appends 4 bytes
> > (rather than 1 and 5 as I thought) to the deflated data. When I cut
> > those off, I can get Ruby to work the same as PHP.

>
> > Still - it would be nice if deflate worked the same as on Java, PHP, ...

>
> > Cheers,

>
> > Pat

>
> > Brian Candler wrote:
> >> On Tue, Mar 13, 2007 at 01:21:52PM +0900, Pat Patterson wrote:

>
> >>> While the Ruby

>
> >>> deflated = Zlib:eflate.deflate( "Hello world" )
> >>> myhex = ""
> >>> 1.upto(deflated.length) { |i| myhex << "%02x" % deflated[i] }
> >>> puts myhex

>
> >>> Shows

>
> >>> 9cf348cdc9c95728cf2fca49010018ab043d00

>
> >>> (BTW - if anyone knows a more succinct way to hex encode a string in
> >>> Ruby, that would be useful)

>
> >> str = "\001\377"
> >> puts str.unpack("H*")

>
> --
> Pat Patterson - (E-Mail Removed)
> Federation Architect,
> Sun Microsystems, Inc.http://blogs.sun.com/superpat



 
Reply With Quote
 
Pat Patterson
Guest
Posts: n/a
 
      03-14-2007
Thanks, Eden - that works great and is much cleaner than cutting the
header and checksum off of the deflated data.

Cheers,

Pat

eden li wrote:
> You can prevent deflate from generating the header by passing in -
> MAX_WBITS to the options for Deflate.new. The following method
> emulates gzdeflate from php:
>
> def gzdeflate(s)
> Zlib:eflate.new(nil, -Zlib::MAX_WBITS).deflate(s, Zlib::FINISH)
> end
>
> puts gzdeflate("Hello World").unpack('H*').first
> # => f348cdc9c95708cf2fca490100
>
> On Mar 14, 8:16 am, Pat Patterson <(E-Mail Removed)> wrote:
>
>> OK - so now I know what is happening...
>>
>> Zlib:eflate.deflate implements ZLIB compression according to RFC 1950.
>> ZLIB defines a 2 byte header containing a variety of flags and a 4 byte
>> trailer containing an Adler-32 checksum. Just out of interest, the
>> header that I'm seeing translates as 'compression method = deflate,
>> windows size = 32k, no preset dictionary, default compression level',
>> which makes perfect sense. Deflate compression itself is defined by RFC
>> 1951.
>>
>> So - if you want 'raw' deflated data (which is called for in many
>> situations), cutting off the leading 2 and trailing 4 bytes is exactly
>> what you need to do.
>>
>> Cheers,
>>
>> Pat
>>
>>
>>
>> Pat Patterson wrote:
>>
>>> Thanks, Brian! That revealed an obvious bug in the code I was using to
>>> examine the deflated data (should have been 0.upto(deflated.length-1)).
>>>
>>> So Base64 /is/ working correctly. Deflate prepends 2 bytes (seems to
>>> be constant 0x789c for default deflate level) and appends 4 bytes
>>> (rather than 1 and 5 as I thought) to the deflated data. When I cut
>>> those off, I can get Ruby to work the same as PHP.
>>>
>>> Still - it would be nice if deflate worked the same as on Java, PHP, ...
>>>
>>> Cheers,
>>>
>>> Pat
>>>
>>> Brian Candler wrote:
>>>
>>>> On Tue, Mar 13, 2007 at 01:21:52PM +0900, Pat Patterson wrote:
>>>>
>>>>> While the Ruby
>>>>>
>>>>> deflated = Zlib:eflate.deflate( "Hello world" )
>>>>> myhex = ""
>>>>> 1.upto(deflated.length) { |i| myhex << "%02x" % deflated[i] }
>>>>> puts myhex
>>>>>
>>>>> Shows
>>>>>
>>>>> 9cf348cdc9c95728cf2fca49010018ab043d00
>>>>>
>>>>> (BTW - if anyone knows a more succinct way to hex encode a string in
>>>>> Ruby, that would be useful)
>>>>>
>>>> str = "\001\377"
>>>> puts str.unpack("H*")
>>>>

>> --
>> Pat Patterson - (E-Mail Removed)
>> Federation Architect,
>> Sun Microsystems, Inc.http://blogs.sun.com/superpat
>>

>
>
>


--
Pat Patterson - (E-Mail Removed)
Federation Architect,
Sun Microsystems, Inc.
http://blogs.sun.com/superpat


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Newbie in Ruby...Issues while connecting Ruby with Sqlite3 Manish Kalra Ruby 2 12-02-2008 07:05 AM
Problems, problems for newbie Shelly ASP .Net 1 09-03-2007 02:10 AM
#!/usr/bin/ruby , #!/usr/bin/ruby -w , #!/usr/bin/ruby -T?, #!/usr/bin/ruby -T1... anne001 Ruby 1 04-23-2006 03:02 PM
[ruby-gnome2] general memory problems with ruby-gnome2? David Espada Ruby 2 07-01-2004 02:18 PM
Problems with the Ruby MySQL Interface 2.4.4a and Ruby 1.8.0 on OS X David Heinemeier Hansson Ruby 0 08-09-2003 01:37 PM



Advertisments