Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Floating point calculation problem

Reply
Thread Tools

Floating point calculation problem

 
 
Hyun chul Park
Guest
Posts: n/a
 
      07-08-2008
hi.
i'm confused by float point calculation.
like this!

------------------------------------------
irb(main):001:0> 123.6 - 123
=> 0.599999999999994

------------------------------------------

how to i get result 0.6 ?
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
 
Zoran Regvart
Guest
Posts: n/a
 
      07-08-2008
Hi,

On Tue, Jul 8, 2008 at 10:39 AM, Hyun chul Park <(E-Mail Removed)> wrote:
> hi.
> i'm confused by float point calculation.
> like this!
>
> ------------------------------------------
> irb(main):001:0> 123.6 - 123
> => 0.599999999999994
>
> ------------------------------------------
>
> how to i get result 0.6 ?


try using BigDecimal, like:

irb(main):001:0> require 'bigdecimal'
=> true
irb(main):002:0> x = BigDecimal("123.6") - BigDecimal("123")
=> #<BigDecimal:8b13c,'0.6E0',4(16)>
irb(main):003:0> x.to_f
=> 0.6

zoran
--
Human by day user by night

 
Reply With Quote
 
 
 
 
Hyun chul Park
Guest
Posts: n/a
 
      07-08-2008
Zoran Regvart wrote:
> Hi,
>
> On Tue, Jul 8, 2008 at 10:39 AM, Hyun chul Park <(E-Mail Removed)>
> wrote:
>> how to i get result 0.6 ?

> try using BigDecimal, like:
>
> irb(main):001:0> require 'bigdecimal'
> => true
> irb(main):002:0> x = BigDecimal("123.6") - BigDecimal("123")
> => #<BigDecimal:8b13c,'0.6E0',4(16)>
> irb(main):003:0> x.to_f
> => 0.6
>
> zoran


thank you, you're reply.
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Axel Etzold
Guest
Posts: n/a
 
      07-08-2008

-------- Original-Nachricht --------
> Datum: Tue, 8 Jul 2008 17:39:50 +0900
> Von: Hyun chul Park <(E-Mail Removed)>
> An: http://www.velocityreviews.com/forums/(E-Mail Removed)
> Betreff: Floating point calculation problem


Hi --

> hi.
> i'm confused by float point calculation.
> like this!
>
> ------------------------------------------
> irb(main):001:0> 123.6 - 123
> => 0.599999999999994
>
> ------------------------------------------
>
> how to i get result 0.6 ?


you may want to take a look at this:

http://docs.sun.com/source/806-3568/ncg_goldberg.html

If you want to do exact calculations, you can use fractions

http://www.ruby-doc.org/stdlib/libdo.../Rational.html

To convert between the two, for each real number (Float), you can use its continued fraction
approximation.

http://www.math.mtu.edu/mathlab/COUR...dnt/pell1.html

Here's a link to a Ruby program that calculates digits of Pi from its continued fraction representation
(and also a Java one, which is much longer):

http://www.cs.utsa.edu/~wagner/pi/pi_cont.html

Best regards,

Axel

--
Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten
Browser-Versionen downloaden: http://www.gmx.net/de/go/browser

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Floating point calculation problem Chris Angelico Python 16 02-03-2013 04:22 AM
Re: Floating point calculation problem Steven D'Aprano Python 0 02-02-2013 12:05 PM
Re: Floating point calculation problem Chris Rebert Python 0 02-02-2013 10:47 AM
floating point calculation problem bei C++ 4 01-07-2010 09:01 PM
floating point calculation problem bei C++ 0 01-06-2010 02:43 AM



Advertisments