Advertisement

weird results

Started by January 02, 2003 08:09 PM
12 comments, last by pi_equals_3 22 years, 1 month ago
http://www.research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html might make for interesting reading. It is concerned with floats rather than doubles, but the principle remains the same (I'm sure you could find the IEEE double standard by Googling, if you're concerned).

[edited by - Miserable on January 2, 2003 10:44:40 PM]
floor() returns the largest integral value not greater than the argument. Maybe you should try to round the value instead, ie get the integral value closest to the argument. This can be done by floor(x + 0.5).

Double precision floats should give you at least 15 accurate digits. Your debugger rounds the value to 14 digits. That''s why you don''t see the difference there.
Advertisement
I haven''t checked this particular number, but it looks like it is the usual case. The problem is that a fraction of .89 cannot be represented in binary; it''s like (1 / 3) = 0.33333333... in the decimal system. You''ll probably find that:
double a = double(1234567.89*100.0);  // == 123456788.9999999...double b = floor(a);                  // == 123456788.0 ("correct")
In this case the "error" is introduced by the compiler, which when converting to binary it will make 1234567.89 => 1234567.8899999999... and then multiply that with 100. The reason the debugger shows them equal, is that it might do what you should; sense this "error" and correct it, to make 0.99999999... => 1.


Thanks guys. I''m already working on a way around this problem. Miserable, I read that article, and it''s helped a lot. CWizard, I see what you mean that it can''t be represented properly in binary. That answers some of my questions, and now I now why only certain cases produce this error. Thanks for all the helpful comments!

This topic is closed to new replies.

Advertisement