[OASIS Issue Tracker] Commented: (OFFICE-2385) 5.4.7 Infix Operator"="

[OASIS Issues Tracker <workgroup_mailer@lists.oasis-open.org>]

    [ http://tools.oasis-open.org/issues/browse/OFFICE-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17586#action_17586 ] 

Dennis Hamilton commented on OFFICE-2385:
-----------------------------------------

There's a rather compact discussion of the problem of comparing values of Number types that use floating-point representations on pp.233-235 in Donald E. Knuth's Art of Computer Programming, vol. 2: Seminumerical Algorithms, ed.3 (Addison-Wesley, Reading MA: 1998).  Start with the paragraph beginning "Programmers who use floating point arithmetic almost never want to test if two computed values are exactly equal (or at least they hardly ever should try to do so) ..."  [There is an interesting discussion about how not to get caught by failures of traditional algebra starting at the bottom of p.321.  This applies to functions like MEAN and STDDEV, with much more in the exercises that Knuth chooses.]

The value epsilon in formulas 4.2.2(21-24) can be obtained using an integer tolerance number with

   epsilon = tolerance*b^(1-p)

where b is the base of the floating point representation and p is the number of base digits of precision.  b^(1-p) is the scaling required to place the tolerance at the the last place of the fraction part in Knuth's model for floating-point numbers.  In this case, epsilon is always exactly derivable from the tolerance number.



> 5.4.7 Infix Operator "="
> ------------------------
>
>                 Key: OFFICE-2385
>                 URL: http://tools.oasis-open.org/issues/browse/OFFICE-2385
>             Project: OASIS Open Document Format for Office Applications (OpenDocument) TC
>          Issue Type: Bug
>          Components: OpenFormula
>    Affects Versions: ODF 1.2 Part 2 CD 2
>            Reporter: Patrick Durusau
>
> General commentary under 5.4.7 Infix Operator "=" deleted.
> Created JIRA issue to call attention to it.
> "Note that in most implementations, numbers are computed using fixed-length representations in base 2 or 16, using a matissa and an exponent. This means that values such as 0.1 cannot be exactly represented (just as 1/3 cannot be exactly represented in base 10 using decimal notation). As a result, (0.1*10) will not have the same bit representation as 1. Since many spreadsheet users do not understand how computers typically represent numbers, applications may attempt to hide these differences by allowing "nearly" equal numbers to be considered equal, but applications are not required to do so.
> Note that in some user interfaces this is displayed or accepted as "=="."

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://tools.oasis-open.org/issues/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira