On Mon, Aug 03, 2009 at 10:03:47AM -0500, Kevin Grittner wrote:
> That's about 0.52% slower with the patch. Because there was over 10%
> variation in the numbers with the patch, I tried leaving out the four
> highest outliers on both, in case it was the result of some other
> activity on the system (even though this machine should have been
> pretty quiet over the weekend) and the difference fell to 0.09%.
What do people do when testing this? I think I'd look to something like
Student's t-test to check for statistical significance. My working
would go something like:
I assume the variance is the same because it's being tested on the same machine.
samples = 20 stddev = 144.26 avg1 = 4783.13 avg2 = 4758.46 t = 0.54 ((avg1 - avg2) / (stddev *
sqrt(2/samples)))
We then have to choose how certain we want to be that they're actually
different, 90% is a reasonably easy level to hit (i.e. one part in ten,
with 95% being more commonly quoted). For 20 samples we have 19 degrees
of freedom--giving us a cut-off[1] of 1.328. 0.54 is obviously well
below this allowing us to say that there's no "statistical significance"
between the two samples at a 90% level.
-- Sam http://samason.me.uk/
[1] http://en.wikipedia.org/wiki/Student's_t-distribution#Table_of_selected_values
