Re: performance-test farm - Mailing list pgsql-hackers

Tomas Vondra <tv@fuzzy.cz> wrote:
> Dne 11.5.2011 23:41, Kevin Grittner napsal(a):
>> Tomas Vondra <tv@fuzzy.cz> wrote:
>>  
>>> 1) Is there something that might serve as a model?
>>  
>> I've been assuming that we would use the PostgreSQL Buildfarm as
>> a model.
>>  
>> http://buildfarm.postgresql.org/
> 
> Yes, I was thinking about that too, but
> 
> 1) A buildfarm used for regular building / unit testing IMHO may
>    not be the right place to do performance testing (not sure how
>    isolated the benchmarks can be etc.).
I'm not saying that we should use the existing buildfarm, or expect
current buildfarm machines to support this; just that the pattern of
people volunteering hardware in a similar way would be good.
> 2) Not sure how open this might be for the developers (if they
>    could issue their own builds etc.).
I haven't done it, but I understand that you can create a "local"
buildfarm instance which isn't reporting its results.  Again,
something similar might be good.
> 3) If this should be part of the current buildfarm, then I'm
>    afraid I can't do much about it.
Not part of the current buildfarm; just using a similar overall
pattern.  Others may have different ideas; I'm just speaking for
myself here about what seems like a good idea to me.
>>> 2) How would you use it? What procedure would you expect?
>>  
>> People who had suitable test environments could sign up to
>> periodically build and performance test using the predetermined
>> test suite, and report results back for a consolidated status
>> display. That would spot regressions.
> 
> So it would be a 'distributed farm'? Not sure it that's a good
> idea, as to get reliable benchmark results you need a proper
> environment (not influenced by other jobs, changes of hw etc.).
Yeah, accurate benchmarking is not easy.  We would have to make sure
people understood that the machine should be dedicated to the
benchmark while it is running, which is not a requirement for the
buildfarm.  Maybe provide some way to annotate HW or OS changes?
So if one machine goes to a new kernel and performance changes
radically, but other machines which didn't change their kernel
continue on a level graph, we'd know to suspect the kernel rather
than some change in PostgreSQL code.
-Kevin