On 22.2.2015 10:59, Andrew Gierth wrote:
>>>>>> "Gavin" == Gavin Flower <GavinFlower@archidevsys.co.nz> writes:
>
> Gavin> Essentially, how consistent are the results, or how great is the
> Gavin> noise? There may be better indicators than the ones I've
> Gavin> suggested above.
>
> This is all rather missing the point.
>
> The relevant metric is not how much noise is introduced between runs
> of the same code, but rather how much noise is introduced as a result
> of non-consequential changes to the code.
>
> I can get variations of several percent - easily more than three
> sigmas of the timing of repeated runs of unchanged code - in the time
> taken to sort a float8 column simply from introducing varying amounts
> of padding into the body of a function which is never called in the
> test. Clearly, the only possible effect here is that the changed
> memory addresses of functions must be resulting in different patterns
> of cache misses / cache replacements, or TLB misses, or similar
> low-level effects which have nothing to do with the code as such.
>
> (That this is a low-level alignment effect is supported by the fact
> that the performance changes are not monotonic in the size of the
> padding; adding more padding may cause either speedups or slowdowns.)
Interesting, but I think Gavin was asking about how much variation was
there for each tested case (e.g. query executed on the same code /
dataset). And in those cases the padding / alignment won't change, and
thus the effects you describe won't influence the results, no?
--
Tomas Vondra http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
