Pliant talk forum

Pliant talk forum

Bug report: Floating point round off error

(cast 2.4 Int) = 2 ok
(cast 2.5 Int) = 2 wrong, should be 3
(cast 2.51 Int) = 3 ok
(cast 2.6 Int) = 3 ok
Message posted by thomasb on 2002/12/04 20:51:05
See abstract. 
Message posted by maybe Hubert Tonneau on 2002/12/04 21:41:56
It's implemented in function 'cast Int' in module /pliant/language/type/number/float.pli

I'm not going to change it because it relies on the 'FRNDINT' i387
processor instruction, and I'm not responsible of it.
Any change would significantly slow things down.

Also another possible explaination could be that I've not setup the i387
properly, or that it should be done using another i387 instruction.
Message posted by hubert.tonneau on 2002/12/04 22:04:40
Here is an even stranger behaviour:

pliant command 'console (cast 1/2+1 Int) eol'
2
pliant command 'console (cast 1/2+2 Int) eol'
2

So the explaination might be related to that way the i387 is handling the last
last bit when computing.
Message posted by maybe pom on 2002/12/05 07:41:28
It seems to me that you might not initialize the i387 properly: The FINIT and FNINIT instructions that have to be called to initialize the i387 fix its rounding control bits to "To nearest or even" strategy, which seems to be the one followed here.

The other accessible rounding strategies are "round up", "round down" and "truncate".

Please, see i387 FPU instruction set page for more details.

Message posted by thomasb on 2002/12/05 23:40:10
The document What Every Computer Scientist Should Know about Floating-Point Arithmetic describes why round to nearest or even is used (page 185), plus alot of other interesting issues about floating-point numbers.

So the current rounding is not a bug and should probarly be kept like it is. However a function that does ┬źnormal┬╗ rounding and the floor, ceiling and truncate functions would be nice.