Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > strange calculation results...?

Reply
Thread Tools

strange calculation results...?

 
 
Jeremy
Guest
Posts: n/a
 
      02-18-2004
hi,

when performing some calculations for display in a table, I
occasionally get the value '04294967295' which is equivalent to 2^32
-1. This value is incorrect as the numbers I'm dealing with are not of
this order.

I know the above details are pretty sketchy, but this looks like some
kind of overflow/underflow. Does anybody have any idea how this number
could crop up?

btw I'm using IBM jdk 1.4.0 on linux.
 
Reply With Quote
 
 
 
 
Jim White
Guest
Posts: n/a
 
      02-18-2004
Jeremy wrote:

> when performing some calculations for display in a table, I
> occasionally get the value '04294967295' which is equivalent to 2^32
> -1. This value is incorrect as the numbers I'm dealing with are not of
> this order.
>
> I know the above details are pretty sketchy, but this looks like some
> kind of overflow/underflow. Does anybody have any idea how this number
> could crop up?


Most likely is that you've taken a Java "int" (32 bit two's complement
integer) -1 and converted it to an unsigned integer for display.
Perhaps some code like: ((long) i) & 0xFFFFFFFFL, or something suitable
in your display formatting code to show unsigned.

Jim
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
what subnet calculation to use for 70-216 jmiddleton67@hotmail.com Microsoft Certification 3 04-07-2005 01:52 PM
CRC polynomal calculation Runar Gjelsvik VHDL 2 04-29-2004 09:08 AM
power calculation in fpga sunil VHDL 2 02-11-2004 12:16 PM
bgp weight calculation Glenn Cisco 4 10-29-2003 02:46 AM
calculation in string Daniel Perl 7 09-10-2003 07:33 PM



Advertisments
 



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57