Page 1 of 1
the calculator
Posted: Tue Aug 22, 2006 6:19 am
by lemon535
i was using the calculator in dhpos and i done the following equation...
217.30
-
216.40
now the answer is obviously .90 i honestly didnt need the calc for that! but the answer i got was like .90000003
Rounding
Posted: Tue Aug 22, 2006 8:56 am
by Dale Harris
lemon535,
Yep, that is just the way BASIC does floating point math. I can correct for that when calulating the money because I know the rounding factor you have set for currency, but the calculator does not know if you are doing math with money so I have to let the floating point rounding errror go through.
However the rounding error is usually a really tiny amount so I am sure that it you get an answer like 256.2199999999998 that you can figure out that the real answer is 256.22