On Fri, Aug 7, 2009 at 3:08 PM, Kevin
Grittner<Kevin.Grittner@wicourts.gov> wrote:
> Sam Mason <sam@samason.me.uk> wrote:
>
>> All we're saying is that we're less than 90% confident that there's
>> something "significant" going on. All the fiddling with standard
>> deviations and sample sizes is just easiest way (that I know of)
>> that statistics currently gives us of determining this more formally
>> than a hand-wavy "it looks OK to me". Science tells us that humans
>> are liable to say things are OK when they're not, as well as vice
>> versa; statistics gives us a way to work past these limitations in
>> some common and useful situations.
>
> Following up, I took the advice offered in the referenced article, and
> used a spreadsheet with a TDIST function for more accurate results
> than available through the table included in the article. That allows
> what I think is a more meaningful number: the probability that taking
> a sample that big would have resulted in a t-statistic larger than was
> actually achieved if there was no real difference.
>
> With the 20 samples from that last round of tests, the answer (rounded
> to the nearest percent) is 60%, so "probably noise" is a good summary.
> Combined with the 12 samples from earlier comparable runs with the
> prior version of the patch, it goes to a 90% probability that noise
> would generate a difference at least that large, so I think we've
> gotten to "almost certainly noise". :-)
So should we give up on this patch?
...Robert
