Thread: Quality and Performance
Every release we seem to have the same debates about performance issues. In 8.0 we shipped knowing that bgwriter had serious deficiencies, plus had no way of logging SQL statements for performance tuning. In 8.2 we even ended up tweaking the planner *after* release. What I don't understand is all the words about quality, yet we don't seem to include performance as part of that. Performance always seems to be a "feature" that can be left until the next release and it's never the right time to fix it. I would hope to persuade all that Performance is an integral part of Quality, not a hindrance to it. I've never worked on a software project where either the Users or the Sponsors said "don't worry about performance, it can wait, but I really love the way you coded that". Quality is very, very high with Postgres, but we also need to include performance as one of the Top Level concerns *and* do that without dropping the ball on other concerns. That clearly takes time and effort to balance those concerns. We obviously need a performance build farm and I think everyone accepts that. We just need to do it, so that's a given and is something I hope to be involved in. What I would really like to persuade everybody is that performance needs specific attention. Once we've finished integrating the code, we're in Beta and changes seem to be more difficult then. We must give time and attention both to measuring performance and to fixing the things we find. Sure we've done a lot of that, and I've been very happy with that, but recent events make me think we have lapsed back into thinking that performance is a threat to quality. I'd love to hear people say loud and clear that performance matters and we can't ship when we know about fixable performance holes. Please can we clear some space in the next release schedule for performance, plus give some credence to the thought that performance issues rate our attention just as much as other kinds of bugs? Maybe we should give each Beta a name, such as "Initial Beta", "Performance Beta", "Usability Beta" as a way of encouraging folk to focus onto particular aspects of quality at what we consider to be appropriate times to do so. Not sure whether thats a good idea, but I'd love to hear about ways to include performance as one of the essential behaviours of PostgreSQL. Your thoughts are welcome, -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com
On Tue, Nov 27, 2007 at 05:32:49PM +0000, Simon Riggs wrote: > What I would really like to persuade everybody is that performance needs > specific attention. [. . .] > Your thoughts are welcome, Well, one thing that might help is something of the specifics you mention. I remember mentioning to Jan not long after he started at Afilias that we occasionally saw strange behaviour that looked like "lock up". He was slightly incredulous, and I didn't have time to build a repeatable test case. So it was in the context of testing Slony that he discovered the dual pains of buffer shuffling and checkpoint storms; this is part of what led him to work on those problems in 8.0. The key was to state, at the outset, "Here is the problem I want to fix." By stating precisely and specifically what is to be fixed, the issue moves from "performance needs" to a feature that can be implemented. Perhaps now is the time to list some specific performance areas you want to fix up? A -- Andrew Sullivan Old sigs will return after re-constitution of blue smoke
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Tue, 27 Nov 2007 17:32:49 +0000
Simon Riggs <simon@2ndquadrant.com> wrote:
> Maybe we should give each Beta a name, such as "Initial Beta",
> "Performance Beta", "Usability Beta" as a way of encouraging folk to
> focus onto particular aspects of quality at what we consider to be
> appropriate times to do so. Not sure whether thats a good idea, but
> I'd love to hear about ways to include performance as one of the
> essential behaviours of PostgreSQL.
>
> Your thoughts are welcome,
Well I think that we do take performance into account. I agree
that we should *never* have a regression in performance from release
to release, which is what I believe has inspired this thread.
However if you look at a lot of the items that have gone into this
release they were all about performance:
HOT -- has a direct correlation to performance because without it
we vacuum more. Not only more, but in a lot of cases it resolves a
serious PostgreSQL limitation for high velocity tables.
Phantom Xid - less vacuuming
The sequential scan thing Jeff Davis did
The reduced size of the tuple headers
Sincerely,
Joshua D. Drake
- --
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFHTF0XATb/zqfZUUQRAul2AKCfKToFWWkdNWYZTCLmjJJeDQpysQCfTKnI
kURZp3SdAoyRLScxG5PizDo=
=PTcF
-----END PGP SIGNATURE-----
On Tue, 2007-11-27 at 10:08 -0800, Joshua D. Drake wrote: > Simon Riggs <simon@2ndquadrant.com> wrote: > > Maybe we should give each Beta a name, such as "Initial Beta", > > "Performance Beta", "Usability Beta" as a way of encouraging folk to > > focus onto particular aspects of quality at what we consider to be > > appropriate times to do so. Not sure whether thats a good idea, but > > I'd love to hear about ways to include performance as one of the > > essential behaviours of PostgreSQL. > > > > Your thoughts are welcome, > > Well I think that we do take performance into account. I agree > that we should *never* have a regression in performance from release > to release, which is what I believe has inspired this thread. > > However if you look at a lot of the items that have gone into this > release they were all about performance: Agreed. I either initiated or assisted with most of those items; but that's not really my point, however because those were planned performance "features". My thinking was about how we handle the last minute attention to detail that ensures we have performance everywhere, not just on the main features. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Tue, 27 Nov 2007 18:18:52 +0000
Simon Riggs <simon@2ndquadrant.com> wrote:
> On Tue, 2007-11-27 at 10:08 -0800, Joshua D. Drake wrote:
> > Simon Riggs <simon@2ndquadrant.com> wrote:
> Agreed. I either initiated or assisted with most of those items; but
> that's not really my point, however because those were planned
> performance "features".
>
> My thinking was about how we handle the last minute attention to
> detail that ensures we have performance everywhere, not just on the
> main features.
Well that is certainly a valid point. Unfortunately that is also a lot
of work :). We would have to have some benchmark that actually
attributes itself to common real world load (even the very simple real
world load like was just fixed).
I would also be happy to put some resources behind this.
Sincerely,
Joshua D. Drake
- --
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFHTGEoATb/zqfZUUQRAgk8AKCLNFPpkEqKF44PyXI9D7k5ynZDSgCfbNNm
5JcjMi1+ZxWX4CzQn4y0Bdk=
=hvTR
-----END PGP SIGNATURE-----
Simon Riggs wrote: > > We obviously need a performance build farm and I think everyone accepts > that. We just need to do it, so that's a given and is something I hope > to be involved in. > > > It's on my list ... Had I but world enough and time ... Performance testing can be bolted onto the exiting buildfarm as an option. However, performance test machines have some requirements that pure functional/build test machines don't have: especially stability. A standard buildfarm client can be put on almost any machine and run happily. My main workstation runs four buildfarm members including three in a VM, and I never notice any impact. But a performance test machine probably needs to be dedicated to just that function. And at least some members of the performance test machines would need to be higher end machines. The number of people who can afford such resources is much lower than those who can run a relatively low impact simple buildfarm member. Maybe we also need to talk about running clients elsewhere for performance testing too. We also need to talk about what would be a good set of tests to run. One useful thing this would buy us is a time series of test results so we could easily see sudden degradations in performance. It must have been annoying trying to triangulate performance dropoff recently. cheers andrew
Joshua D. Drake wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Tue, 27 Nov 2007 17:32:49 +0000 > Simon Riggs <simon@2ndquadrant.com> wrote: > > > > Maybe we should give each Beta a name, such as "Initial Beta", > > "Performance Beta", "Usability Beta" as a way of encouraging folk to > > focus onto particular aspects of quality at what we consider to be > > appropriate times to do so. Not sure whether thats a good idea, but > > I'd love to hear about ways to include performance as one of the > > essential behaviours of PostgreSQL. > > Well I think that we do take performance into account. I agree > that we should *never* have a regression in performance from release > to release, which is what I believe has inspired this thread. Hmm. I have developed several features that have driven performance down. Autovacuum enabled by default for one. IIRC the SELECT FOR SHARE stuff (tuple-level share locks) also hurt performance. Savepoints required enlarging tuple headers, which also hurt performance. In all cases we have gotten some other benefit, be it reduced administrative pain, or reduced lock contention, or a new feature. (In fact I think most performance drops have been my fault.) -- Alvaro Herrera http://www.advogato.org/person/alvherre "Some men are heterosexual, and some are bisexual, and some men don't think about sex at all... they become lawyers" (Woody Allen)
Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> Joshua D. Drake wrote:
>> Well I think that we do take performance into account. I agree
>> that we should *never* have a regression in performance from release
>> to release, which is what I believe has inspired this thread.
> Hmm. I have developed several features that have driven performance
> down.
Even changes that are not feature additions but intended solely to
improve performance may have corner cases where they are losses rather
than wins. I think "*never* have a regression in performance" is not
only pie-in-the-sky but would be a bad policy to adopt, because it
would mean for instance that we couldn't intentionally optimize common
cases at the expense of uncommon ones.
However, I think everybody agrees that getting blindsided by unexpected
performance dropoffs is a bad thing. We really need to reinstitute
the sort of daily (or near-daily) performance tracking that Mark Wong
used to be doing, and extend it to cover a wider variety of test cases
than just DBT-2. As an example, I'll bet that this issue of operator
lookup speed would never have been visible at all in DBT-2.
regards, tom lane
On Tue, 2007-11-27 at 13:54 -0500, Tom Lane wrote: > However, I think everybody agrees that getting blindsided by unexpected > performance dropoffs is a bad thing. We really need to reinstitute > the sort of daily (or near-daily) performance tracking that Mark Wong > used to be doing, and extend it to cover a wider variety of test cases > than just DBT-2. As an example, I'll bet that this issue of operator > lookup speed would never have been visible at all in DBT-2. Yeh, we need multiple large benchmarks run on a regular basis. My understanding is the community has two 8-core servers to run benchmarks on, but I'd quite like to have some details on where these are at. One is likely to be running RHEL, one Solaris. We also need performance regression tests, which is a slightly different thing even if they do sound similar. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Tue, 27 Nov 2007 20:32:57 +0000
Simon Riggs <simon@2ndquadrant.com> wrote:
> On Tue, 2007-11-27 at 13:54 -0500, Tom Lane wrote:
>
> > However, I think everybody agrees that getting blindsided by
> > unexpected performance dropoffs is a bad thing. We really need to
> > reinstitute the sort of daily (or near-daily) performance tracking
> > that Mark Wong used to be doing, and extend it to cover a wider
> > variety of test cases than just DBT-2. As an example, I'll bet
> > that this issue of operator lookup speed would never have been
> > visible at all in DBT-2.
>
> Yeh, we need multiple large benchmarks run on a regular basis.
>
> My understanding is the community has two 8-core servers to run
> benchmarks on, but I'd quite like to have some details on where these
> are at. One is likely to be running RHEL, one Solaris.
The RHEL one as I know it, is the MyYearbook donated one. We are
currently unaware of the status of that machine except to say it is
currently running Gentoo.
I don't know the status of the Solaris machine except that I think we
had IO issues with it.
Sincerely,
Joshua D. Drake
>
> We also need performance regression tests, which is a slightly
> different thing even if they do sound similar.
>
- --
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFHTH+7ATb/zqfZUUQRAhlYAJ0QqG5CzDmQfi2ynj/i9xJoe1nWUACeOyJQ
gSOnZhVhp61HbOlfRjfDzvM=
=MEIW
-----END PGP SIGNATURE-----
On Tue, 2007-11-27 at 13:32 -0500, Andrew Dunstan wrote: > We also need to talk about what would be a good set of tests to run. I think we should develop a series of performance regression tests that can be run as an option on the buildfarm. We'd want a separate page for that with graphs etc, as you suggest. My vision for that is a set of tests that test very specific aspects of code, much the same way as the regression tests attempt feature coverage. Examples would be - 10000 INSERTs - 10000 INSERTs using multi-VALUEs clauses - 100000 rows inserted by COPY - 100000 rows inserted by CTAS We would need a way to compare results between releases, so we can see which aspects have regressed/improved, just as we have with the buildfarm. That will also be food for release notes, where we can mention all actions that are >5% faster, or anything we must regrettably report as being slower. Sounds like it's waiting on somebody to make the first move, so maybe I should do that, then let everybody else chip into the framework. Should we do this as part of core, or as a separate pgfoundry project? -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com
On Tue, 2007-11-27 at 12:36 -0800, Joshua D. Drake wrote: > The RHEL one as I know it, is the MyYearbook donated one. We are > currently unaware of the status of that machine except to say it is > currently running Gentoo. > > I don't know the status of the Solaris machine except that I think we > had IO issues with it. Might I enquire who has these machines, so I can ask them how can I get access to them? Are they really Community systems? In what sense? -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Tue, 27 Nov 2007 21:00:03 +0000
Simon Riggs <simon@2ndquadrant.com> wrote:
> On Tue, 2007-11-27 at 12:36 -0800, Joshua D. Drake wrote:
>
> > The RHEL one as I know it, is the MyYearbook donated one. We are
> > currently unaware of the status of that machine except to say it is
> > currently running Gentoo.
> >
> > I don't know the status of the Solaris machine except that I think
> > we had IO issues with it.
>
> Might I enquire who has these machines, so I can ask them how can I
> get access to them? Are they really Community systems? In what sense?
They are community systems in the sense that JoshB told the community
Sun donated a box and MyYearbook said they donated a box. MyYearbook
had stated that we (cmd) were going to manage the MyYearbook box but
nothing has come of it as of yet. I haven't heard anything on the Sun
machine in a *long* time.
My guess is that everyone is busy.
So I have a question :)... The community does have money. We have IO
available, what we don't have is a machine. Would the community be
interested in purchasing a box?
We could fairly reasonably get an HP or Dell (the new Dell's aren't
nearly as bad as they used to be) or Sun 8 core box with reasonable ram
and just a raid 1 for the OS. Then use the external IO we have available
with that machine.
Joshua D. Drake
- --
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFHTIh5ATb/zqfZUUQRArJ1AJ4jf6S12CY4duUTYzlSQlF+YTYRvwCgmu/i
Dj36cknbGDX2Bzh6rmIkNw8=
=jl1/
-----END PGP SIGNATURE-----
On Tue, 2007-11-27 at 15:33 -0300, Alvaro Herrera wrote: > Joshua D. Drake wrote: > I agree > > that we should *never* have a regression in performance from release > > to release, which is what I believe has inspired this thread. > > Hmm. I have developed several features that have driven performance > down. I think performance reductions as a result of additional functionality are acceptable, but we should aim to minimise them. It's the small things that crop up along the way that must be fixed, with the same vigour we fix other bugs. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com
Simon Riggs wrote: > On Tue, 2007-11-27 at 13:32 -0500, Andrew Dunstan wrote: > > >> We also need to talk about what would be a good set of tests to run. >> > > Sounds like it's waiting on somebody to make the first move, so maybe I > should do that, then let everybody else chip into the framework. > If you start with a set of tests and send it to me I will start work on a benchmarking step in the buildfarm client. > Should we do this as part of core, or as a separate pgfoundry project? > > Core, please. This is mainline -hackers material. cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> Simon Riggs wrote:
>> Should we do this as part of core, or as a separate pgfoundry project?
> Core, please. This is mainline -hackers material.
Huh? The buildfarm isn't in core, why would a performfarm be?
regards, tom lane
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Tue, 27 Nov 2007 21:00:03 +0000
Simon Riggs <simon@2ndquadrant.com> wrote:
> On Tue, 2007-11-27 at 12:36 -0800, Joshua D. Drake wrote:
>
> > The RHEL one as I know it, is the MyYearbook donated one. We are
> > currently unaware of the status of that machine except to say it is
> > currently running Gentoo.
> >
> > I don't know the status of the Solaris machine except that I think
> > we had IO issues with it.
>
> Might I enquire who has these machines, so I can ask them how can I
> get access to them? Are they really Community systems? In what sense?
>
We just spoke with MyYearbook and my suspicions were correct. Everyone
is just busy. Next ETA on that machine is 3 weeks (on the outside).. so
expect the beginning of the year.
Joshua D. Drake
- --
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFHTKN4ATb/zqfZUUQRAoTiAKCC6+RNPFihWyos4s7vCmwt2K3C1gCfdFkV
z0ht4nKhelQ+UUwrHnMLXkA=
=2JWW
-----END PGP SIGNATURE-----
On Nov 27, 2007 11:45 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > If you start with a set of tests and send it to me I will start work on > a benchmarking step in the buildfarm client. Are you sure it shouldn't be a separate client? I don't think neither the prerequisites nor the results wanted have something in common with the build farm. Another idea is that we should find a way to let people run their specific benchmarks and use the client to report their results. Some people may want to run read only benchmarks, some I/O heavy benchmarks, and others benchmarks which runs for more than a day. -- Guillaume
Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: > >> Simon Riggs wrote: >> >>> Should we do this as part of core, or as a separate pgfoundry project? >>> > > >> Core, please. This is mainline -hackers material. >> > > Huh? The buildfarm isn't in core, why would a performfarm be? > > > It's the tests I think belong in core, not the farm software. Currently buildfarm performs functionality tests that are also in core. cheers andrew
On Nov 27, 2007 7:32 PM, Andrew Dunstan <andrew@dunslane.net> wrote: > But a performance test machine > probably needs to be dedicated to just that function. And at least some > members of the performance test machines would need to be higher end > machines. The number of people who can afford such resources is much > lower than those who can run a relatively low impact simple buildfarm > member. As I already indicated it in another thread, we will dedicate 7 servers donated by Continuent to exactly that in a near future (they are waiting in their boxes atm). So we will have dedicated boxes to run benchmarks night & day. They will be accessible to the community to set up as much benchmarks as necessary and I'll help to set up them and make sure everything is OK. They are not high end servers as the MyYearBook or Sun ones but they are pretty decent. They are all i386 so we'll have to get other clients but I think we can set up a variety of benchmarks to get them busy and produce valuable results. If we can get a benchmark client/results server infrastructure by the end of january, it will be right on time. I'm pretty sure all vendors of PostgreSQL are aware that a bench farm is really important right now so I'm pretty sure we'll get more servers if we can get something as easy to setup as the build farm. -- Guillaume
Andrew, > It's the tests I think belong in core, not the farm software. Currently > buildfarm performs functionality tests that are also in core. Jignesh and I were talking about writing a Pole Position-style test which measures peformance on each of a couple dozen specific operations. There are limits to what we could ship with PostgreSQL ... DW operations aren't really testable without 18 hours to generate data ... but we could test a lot of things. -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco
Josh Berkus wrote: > Andrew, > > >> It's the tests I think belong in core, not the farm software. Currently >> buildfarm performs functionality tests that are also in core. >> > > Jignesh and I were talking about writing a Pole Position-style test which > measures peformance on each of a couple dozen specific operations. There > are limits to what we could ship with PostgreSQL ... DW operations aren't > really testable without 18 hours to generate data ... but we could test a > lot of things. > > I think we're going to need several sets of tests, and some settable scaling factors. Performance isn't just about humungous DW apps. We need to have testing on small to medium sized apps/machinery as well. cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> Josh Berkus wrote:
>> ... DW operations aren't
>> really testable without 18 hours to generate data ... but we could test a
>> lot of things.
> Performance isn't just about humungous DW apps.
Indeed. I think the real take-home lesson from these past few days'
discussion is that *any* particular view of performance is going to
miss things that don't affect that case, but do affect somebody else.
What I find most worrisome about the notion of setting up a
performance-farm is that it will encourage us to optimize with blinkers
on --- that is, that we will consider only the specific cases measured
by whatever tests are included in the farm, and will happily pessimize
other cases. We can ameliorate that a bit if we can get a sufficiently
wide variety of test cases, but it will always be a concern. And
dogmatic positions like "only cases involving terabytes of data are
worth testing" are definitely not going to help.
regards, tom lane
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Wed, 28 Nov 2007 00:15:48 -0500
Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
> > Josh Berkus wrote:
> >> ... DW operations aren't
> >> really testable without 18 hours to generate data ... but we could
> >> test a lot of things.
>
> > Performance isn't just about humungous DW apps.
>
> Indeed. I think the real take-home lesson from these past few days'
> discussion is that *any* particular view of performance is going to
> miss things that don't affect that case, but do affect somebody else.
>
> What I find most worrisome about the notion of setting up a
> performance-farm is that it will encourage us to optimize with
> blinkers on --- that is, that we will consider only the specific
> cases measured by whatever tests are included in the farm, and will
> happily pessimize other cases. We can ameliorate that a bit if we
> can get a sufficiently wide variety of test cases, but it will always
> be a concern. And dogmatic positions like "only cases involving
> terabytes of data are worth testing" are definitely not going to help.
Well I certainly agree with that, especially considering that although
we do have installations with that type of data, the percentage is
microscopic compared to those that don't.
I think it may be interested to host a series of different test from
pgbench, odbcbench, dbt2, custom scripts etc... to provide trending for
a particular host. That way if one host does 50tps continuously and
then changes one way or the other, there is at least a flag...
Sincerely,
Joshua D. Drake
- --
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFHTP3iATb/zqfZUUQRAgQ3AJ9N71Xgl+O4H4dr/IFagzjxu0HvAwCfZVkB
ReQmGxacCqilRstLGhoqBU4=
=03BG
-----END PGP SIGNATURE-----
Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: >> Josh Berkus wrote: >>> ... DW operations aren't >>> really testable without 18 hours to generate data ... but we could test a >>> lot of things. > >> Performance isn't just about humungous DW apps. > > Indeed. I think the real take-home lesson from these past few days' > discussion is that *any* particular view of performance is going to > miss things that don't affect that case, but do affect somebody else. yep - but the "do not affect somebody else" might be one that could be actually catched by something like a benchfarm. > > What I find most worrisome about the notion of setting up a > performance-farm is that it will encourage us to optimize with blinkers > on --- that is, that we will consider only the specific cases measured > by whatever tests are included in the farm, and will happily pessimize > other cases. We can ameliorate that a bit if we can get a sufficiently > wide variety of test cases, but it will always be a concern. And > dogmatic positions like "only cases involving terabytes of data are > worth testing" are definitely not going to help. agreed - I don't think having the tests itself in core (at least initially) is such a good idea(neither am I sure tacking it on top of the buildfarm really is). There are a LOT of things we could do with such a farm/infrastructure but it will take time to exactly figure out what we can reasonably do on an automated/regular base and in a common framework. Stefan
On Tue, 27 Nov 2007, Simon Riggs wrote: > My vision for that is a set of tests that test very specific aspects of > code, much the same way as the regression tests attempt feature > coverage. Examples would be > - 10000 INSERTs > - 10000 INSERTs using multi-VALUEs clauses > - 100000 rows inserted by COPY > - 100000 rows inserted by CTAS In addition to these, there's actually a nice set of sample primitive operations to test included in the "Advanced transactional" section of sysbench: http://sysbench.sourceforge.net/docs/#database_mode I have code that does this sort of thing, including tests at multiple client loads, that even spits out HTML for the results. It's based on pgbench and I sent out hundreds of results generated by it when I was doing the background writer optimization. The results are thoroughly boring on just my system, and I'm just waiting for a larger server to be available before it's worth my trouble to revive that project. Once such a system becomes available, I could have a basic set of tests like these ready the day after I get a login. That's not to say what you're talking about isn't needed, because using pgbench introduces some limitations and buildfarm integration is a whole different topic. But I have made a first move here and only need the hardware to become available before I can produce something useful with it. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD