Re: [office-comment] RSQ - Unstable Algorithm

[Eike Rathke <erack@sun.com>]

Hi Leonard,

On Thursday, 2007-06-07 23:04:10 +0300, Leonard Mada wrote:

> Unfortunately, the formulas/algorithms given for 'a' and 'b' seem to be 
> computationally unstable
> - they both use terms of the form n*sum(x^2) - (sum(x))^2 which are 
> notoriously unstable
> 
> Although I did NOT do a formal calculation of the error term, having 
> *two* occurrences of  this unstable calculation, and both times as 
> *divisors* makes the formula very unreliable.

Thanks for pointing out. Did you try it out in OOoCalc and noticed
shortcomings?

> SUGGESTIONS:
> - remove formula from specification

I think that is not a good option because it would leave the definition
without a mathematical description. Even if the given formula may
numerically be instable at some point it is mathematically correct.
However, we could add a note saying so.

> - consult statistician knowledgeable of stable coding algorithms

That of course would be favourable. As I also know from the OOo mailing
lists that you're interested in that area, could you help us or are you
in contact with one?

  Eike

-- 
 OpenOffice.org Engineering at Sun: http://blogs.sun.com/GullFOSS