On Tue, Mar 14, 2017 at 3:45 PM, Kevin Grittner <kgrittn@gmail.com> wrote:
>> On 3/14/17 17:34, Mengxing Liu wrote:
>>> There are several alternative benchmarks. Tony suggested that we
>>> should use TPC-E and TPC-DS.
>
> More benchmarks is better, all other things being equal. Keep in
> mind that good benchmarking practice with PostgreSQL generally
> requires a lot of setup time (so that we're starting from the exact
> same conditions for every run), a lot of run time (so that the
> effects of vacuuming, bloat, and page splitting all comes into play,
> like it would in the real world), and a lot of repetitions of each
> run (to account for variation). In particular, on a NUMA machine it
> is not at all unusual to see bifurcated
Sorry I didn't finish that sentence.
On a NUMA machine It is not at all unusual to see bifurcated results
-- with each run coming in very close to one number or a second
number, often at about a 50/50 rate, with no numbers falling
anywhere else. This seems to be based on where the processes and
memory allocations happen to land.
Different people have dealt with this in different ways. If you can
get five or more runs of a given test, perhaps the best is to throw
away the high and low values and average the rest. Other approaches
I've seen are to average all, take the median, or take the best
result. The worst result of a set of runs is rarely interesting, as
it may be due to some periodic maintenance kicking in (perhaps at
the OS level and not related to database activity at all).
--
Kevin Grittner
