Re: performance-test farm - Mailing list pgsql-hackers

Tomas Vondra wrote:
> Actually I was not aware of how the buildfarm works, all I
> knew was there's something like that because some of the hackers mention
> a failed build on the mailing list occasionally.
>
> So I guess this is a good opportunity to investigate it a bit ;-)
>
> Anyway I'm not sure this would give us the kind of environment we need
> to do benchmarks ... but it's worth to think of.
>   

The idea is that buildfarm systems that are known to have a) reasonable 
hardware and b) no other concurrent work going on could also do 
performance tests.  The main benefit of this approach is it avoids 
duplicating all of the system management and source code building work 
needed for any sort of thing like this; just leverage the buildfarm 
parts when they solve similar enough problems.  Someone has actually 
done all that already; source code was last sync'd to the build farm 
master at the end of March:  https://github.com/greg2ndQuadrant/client-code

By far the #1 thing needed to move this forward from where it's stuck at 
now is someone willing to dig into the web application side of this.  
We're collecting useful data.  It needs to now be uploaded to the 
server, saved, and then reports of what happened generated.  Eventually 
graphs of performance results over time will be straighforward to 
generate.  But the whole idea requires someone else (not Andrew, who has 
enough to do) sits down and figures out how to extend the web UI with 
these new elements.


> I guess we could run a script that collects all those important
> parameters and then detect changes. Anyway we still need some 'really
> stable' machines that are not changed at all, to get a long-term baseline.
>   

I have several such scripts I use, and know where two very serious ones 
developed by others are at too.  This part is not a problem.  If the 
changes are big enough to matter, they will show up as a difference on 
the many possible "how is the server configured?" reports, we just need 
to pick the most reasonable one.  It's a small details I'm not worried 
about yet.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us