int() versus round()

[skiman]

Hi,

I had problems with getting values with 2 decimals after some calculations. SOmetimes an invoice isn't calculated correctly. After a long time searching, I found that there is a problem with the following code:

Code: Select all

set decimals to 6
s1 := 22.90
@ 1,1 dcsay "Value of s1: " get s1 saysize 40 sayright getsize 20 when .F.
s2 := s1 * (100-15) / 100
@ 2,1 dcsay "s2 := s1 * 85 / 100" get s2 saysize 40 sayright getsize 20 when .F.
s3 := round(s2,2)
@ 3,1 dcsay "s3 := round(s2,2)" get s3 saysize 40 sayright getsize 20 when .F.
s4 := s2*100+0.5
@ 4,1 dcsay "s4 := s2*100+0.5" get s4 saysize 40 sayright getsize 20 when .F.
s5 := int(s4)
@ 5,1 dcsay "s5 := int(s4)" get s5 saysize 40 sayright getsize 20 when .F.
s6 := int(s4)/100
@ 5,1 dcsay "s6 := int(s4)/100" get s6 saysize 40 sayright getsize 20 when .F.

dcread gui fit addbuttons to lok

This gives me the following:

int2.PNG (19.06 KiB) Viewed 8054 times

I would expect that s3 and s6 should be the same? Also s5 which is the int(s4) looks wrong to me. Someone else who already had this kind of strange calculation?

[rdonnay]

You are showing 2 gets at the same screen location.

Change the 2nd one to row 6.

This will show you that s5 seems to be the problem because the int() of s4 should be 1947 not 1946.

If you change s4 from s2*100+0.5 to Round(s2*100+0.5,6) you will get the desired result.
This is apparently due to a floating point problem.

[skiman]

Hi Roger,

You are showing 2 gets at the same screen location.

The screen shot is correct, I modified the code before I made the screenshot.

This will show you that s5 seems to be the problem because the int() of s4 should be 1947 not 1946.

Yes, that's indeed where there is a problem. However with other values sometimes the problem is in the round().

If you change s4 from s2*100+0.5 to Round(s2*100+0.5,6) you will get the desired result.

I tested this with some values, and it looks as this is working. I will implement this and do a test with a lot of invoices.