The oddities of floating-point arithmetic
Posted by Tom Moertel Wed, 28 Sep 2005 02:59:00 GMT
Tobias at too-biased asks wtf? is going on with the following unintuitive result from Ruby:
irb(main):001:0> (4.10 * 100).to_i
=> 409
What’s going on is that to_i truncates its floating-point argument, discarding the fractional part. The fractional part, owing to the intricacies of floating-point arithmetic, does not agree with our intuition:
irb(main):002:0> "%3.20g" % (4.10 * 100)
=> "409.99999999999994316"
In this case, the factional part is almost 1. Throwing it away turns our expected result of 410 into the unexpected 409. (That’s why round makes for a better choice than to_i when we want the nearest integer to our float.)
If this behavior seems bizarre, I recommend David Goldberg’s classic What Every Computer Scientist Should Know About Floating Point Arithmetic (1991). The paper was originally published in ACM Computing Surveys and later republished with corrections. The corrected versions, however, were apparently typeset in Microsoft Word, and the formulas are hard to look at (especially if you are accustomed to TeX-quality math typesetting). You might want to get your hands on a PDF of the original version if you care about that kind of thing.
Posted in programming
no comments
no trackbacks
readers