Thread: Re: [PATCHES] Some minor changes to pgbench
"Joshua D. Drake" <jd@commandprompt.com> writes:
> * The schema now uses foreign keys to more accurately reflect a finacial DDL
Addition of foreign key checking will certainly impact performance
significantly.
> * The history table now has a primary key that uses a serial
Ditto.
> * The respective balance columns have been increased to int8 to deal
> with larger values
Ditto.
> * Initalization will be done in a new schema/namespace, pgbench will
> exit if this schema/namespace exists
OK, maybe that doesn't matter.
> * The new DDL should allow both Mammoth Replicator and Slony to be
> tested using pgbench (at least basic replication)
Erm ... exactly why couldn't you do that before?
pgbench doesn't have all that many things to recommend it, but what
it does have is that it's been a stable testbed across quite a few
PG releases. Arbitrarily whacking around the tested functionality
will destroy that continuity. I fell into this trap before myself
... I have a local copy of pgbench that produces TPS numbers quite
a lot better than the standard pgbench, against exactly the same
server. What's wrong with that picture?
regards, tom lane
Tom Lane wrote:
> "Joshua D. Drake" <jd@commandprompt.com> writes:
>> * The schema now uses foreign keys to more accurately reflect a finacial DDL
>
> Addition of foreign key checking will certainly impact performance
> significantly.
That is kind of the point. Without foreign keys it is a flawed test
because you wouldn't be running in production without them and thus you
can't bench without them.
>
>> * The history table now has a primary key that uses a serial
>
> Ditto.
Again, part of the point :)
>
>> * The respective balance columns have been increased to int8 to deal
>> with larger values
>
> Ditto.
This was done because you can easily break pgbench without the increase
in data type.
pgbench -c 850 -t 1000 pgbench
gave a stream of errors like this before ending:
Client 18 aborted in state 8: ERROR: integer out of range
Client 429 aborted in state 8: ERROR: integer out of range
Client 168 aborted in state 8: ERROR: integer out of range
PG error log showed:
2006-08-22 15:45:19 PDT-[local]STATEMENT: UPDATE branches SET bbalance
= bbalance + 4209228 WHERE bid = 679;
2006-08-22 15:45:19 PDT-[local]ERROR: integer out of range
>> * Initalization will be done in a new schema/namespace, pgbench will
>> exit if this schema/namespace exists
>
> OK, maybe that doesn't matter.
Yeah I did it just so we wouldn't stomp on somebody on accident.
>
>> * The new DDL should allow both Mammoth Replicator and Slony to be
>> tested using pgbench (at least basic replication)
>
> Erm ... exactly why couldn't you do that before?
history was missing a primary key. It could be done before. I just
removed a step in getting it to work.
> pgbench doesn't have all that many things to recommend it, but what
> it does have is that it's been a stable testbed across quite a few
> PG releases. Arbitrarily whacking around the tested functionality
> will destroy that continuity.
Well to be fair, I wasn't doing it arbitrarily. I had a specific purpose
which was to have it use a schema that would be closer to a production
schema, without breaking existing behavior.
This patch does that :)
> I fell into this trap before myself
> ... I have a local copy of pgbench that produces TPS numbers quite
> a lot better than the standard pgbench, against exactly the same
> server. What's wrong with that picture?
Well I think we all agree that some of the behavior of pgbench has been
weird.
Sincerely,
Joshua D. Drake
>
> regards, tom lane
>
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
"Joshua D. Drake" <jd@commandprompt.com> writes:
> Tom Lane wrote:
>> Addition of foreign key checking will certainly impact performance
>> significantly.
> That is kind of the point. Without foreign keys it is a flawed test
> because you wouldn't be running in production without them and thus you
> can't bench without them.
pgbench is not about reality, though. If we can't rely on it to give
consistent results across versions then I don't think it's useful at all.
There are many other benchmarks you can run that do speak to reality
(eg OSDL's work).
regards, tom lane
Tom Lane wrote:
> "Joshua D. Drake" <jd@commandprompt.com> writes:
>> Tom Lane wrote:
>>> Addition of foreign key checking will certainly impact performance
>>> significantly.
>
>> That is kind of the point. Without foreign keys it is a flawed test
>> because you wouldn't be running in production without them and thus you
>> can't bench without them.
>
> pgbench is not about reality, though. If we can't rely on it to give
> consistent results across versions then I don't think it's useful at all.
> There are many other benchmarks you can run that do speak to reality
> (eg OSDL's work).
Would it be worthwhile to add a switch so that the foreign key test is
only used "if" they use the switch in conjunction with a -i?
Joshua D. Drake
>
> regards, tom lane
>
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
"Joshua D. Drake" <jd@commandprompt.com> writes:
> Would it be worthwhile to add a switch so that the foreign key test is
> only used "if" they use the switch in conjunction with a -i?
I wouldn't object to providing that as a (non default) option.
The int8 change should be unnecessary in view of Tatsuo's recent fix
to make the random deltas symmetrical about zero. I would counsel
against making the other changes either, as they seem to accomplish
little except make the comparability of results more doubtful.
regards, tom lane
Tom Lane wrote:
> "Joshua D. Drake" <jd@commandprompt.com> writes:
>> Would it be worthwhile to add a switch so that the foreign key test is
>> only used "if" they use the switch in conjunction with a -i?
>
> I wouldn't object to providing that as a (non default) option.
O.k. I will take a look at what that would take..
>
> The int8 change should be unnecessary in view of Tatsuo's recent fix
> to make the random deltas symmetrical about zero.
O.k. that may be the case, my testing was with the 8.1 version pgbench.
I will verify with -HEAD before I change the int8.
> I would counsel
> against making the other changes either, as they seem to accomplish
> little except make the comparability of results more doubtful.
I am not interested in really changing the overall internals. I just
wanted the foreign key option.
Sincerely,
Joshua D. Drake
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
> choose an index scan if your joining column's datatypes do not
> match
>
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
"Joshua D. Drake" <jd@commandprompt.com> wrote: > >> Would it be worthwhile to add a switch so that the foreign key test is > >> only used "if" they use the switch in conjunction with a -i? > > > > I wouldn't object to providing that as a (non default) option. > > O.k. I will take a look at what that would take.. If we provide 'external initializer', some of your proposals can be done through more generalized way, at least the foreign key test. It is a replacement of the DDLAFTERs[], which is a hard-codeded list in the source code to create primary keys. It may help other people, for example, who want to test the performance of hash index instead of btree index (primary key). Using it, you can use the below script or something: $ pgbench -i -f init_with_fkey.sql [init_with_fkey.sql] alter table branches add primary key (bid); ... alter table tellers add constraint tellers_bid_fkey foreign key (bid) references branches (bid); ... Are you interested in this idea? Regards, --- ITAGAKI Takahiro NTT Open Source Software Center
ITAGAKI Takahiro <itagaki.takahiro@oss.ntt.co.jp> writes:
> If we provide 'external initializer', some of your proposals can be done
> through more generalized way, at least the foreign key test. ...
> Are you interested in this idea?
Sounds good to me --- this seems to follow along with Tatsuo's recent
work to push pgbench's tests into scriptable form. I've got no problem
with making pgbench able to exercise more test cases; I'm only concerned
that we have a consistent "base case" across versions.
regards, tom lane