On Sun, Dec 23, 2012 at 11:11 PM, Jeff Janes <jeff.janes@gmail.com> wrote:
> You could say that benchmarks should run long enough to average out such
> changes
I would say any benchmark needs to be run long enough to reach a
steady state before the measurements are taken. The usual practice is
to run a series groups and observe the aggregate measurements for each
group. For instance run 10 runs with each run including of 1000
repetitions of the transaction. Then you can observe at which point
the averages for individual groups begin to behave consistently. If
the first three are outliers but the remaining 7 are stable then
discard the first three and take the average (or often median) of the
remaining 7.
If you include the early runs which are affected by non-steady-state
conditions such as cache effects or file fragmentation then it can
take a very long time for those effects to be erased by averaging with
later results. Worse, it's very difficult to tell whether you've
waited long enough.
--
greg
