Thread: Unfortunate choice of short switch name in pgbench

I just wasted some time puzzling over strange results from pgbench.
I eventually realized that I'd been testing against the wrong server,
because rather than "-p 65432" I'd typed "-P 65432", thereby invoking
the recently added --progress option.  pgbench has no way to know that
that isn't what I meant; the fact that both switches take integer
arguments doesn't help.

To fix this, I propose removing the -P short form and only allowing the
long --progress form.  I won't argue that this feature is completely
useless, but for sure it's not something I'd want more often than once
in a blue moon.  So I think it does not need to have a short form; and
for sure it doesn't need a short form that's so easily confused with a
commonly used switch.

If no objections, I'll go make that change.
        regards, tom lane

On Tue, Feb 25, 2014 at 2:49 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I just wasted some time puzzling over strange results from pgbench.
> I eventually realized that I'd been testing against the wrong server,
> because rather than "-p 65432" I'd typed "-P 65432", thereby invoking
> the recently added --progress option.  pgbench has no way to know that
> that isn't what I meant; the fact that both switches take integer
> arguments doesn't help.
>
> To fix this, I propose removing the -P short form and only allowing the
> long --progress form.  I won't argue that this feature is completely
> useless, but for sure it's not something I'd want more often than once
> in a blue moon.  So I think it does not need to have a short form; and
> for sure it doesn't need a short form that's so easily confused with a
> commonly used switch.
>
> If no objections, I'll go make that change.

Hmm.  I don't have a real specific opinion on the value of this
particular --progress option, but my experience is that most
--progress options get a lot of use.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Feb 25, 2014 at 2:49 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> To fix this, I propose removing the -P short form and only allowing the
>> long --progress form.  I won't argue that this feature is completely
>> useless, but for sure it's not something I'd want more often than once
>> in a blue moon.  So I think it does not need to have a short form; and
>> for sure it doesn't need a short form that's so easily confused with a
>> commonly used switch.

> Hmm.  I don't have a real specific opinion on the value of this
> particular --progress option, but my experience is that most
> --progress options get a lot of use.

Meh.  A progress-reporting feature has use when the tool is working
towards completion of a clearly defined task.  In the case of pgbench,
if you told it to run for -T 60 seconds rather than -T 10 seconds,
that's probably because you don't trust a 10-second average to be
sufficiently reproducible.  So I'm not real sure that reporting averages
over shorter intervals is all that useful; especially not if it takes
cycles out of pgbench, which itself is often a bottleneck.

I could see some value in a feature that computed shorter-interval TPS
averages and then did some further arithmetic, like measuring the standard
deviation of the shorter-interval averages to assess how much noise there
will be in the full-run average.  But that's not what this does, and if it
did do that, "reporting progress" would not be what I'd see as its main
purpose.
        regards, tom lane

On 02/25/2014 11:32 PM, Tom Lane wrote:
> Meh.  A progress-reporting feature has use when the tool is working
> towards completion of a clearly defined task.  In the case of pgbench,
> if you told it to run for -T 60 seconds rather than -T 10 seconds,
> that's probably because you don't trust a 10-second average to be
> sufficiently reproducible.  So I'm not real sure that reporting averages
> over shorter intervals is all that useful; especially not if it takes
> cycles out of pgbench, which itself is often a bottleneck.

It's not useful when doing rigorous benchmarking to publish results, but 
in quick testing of various hacks during development, getting 10-second 
averages is very useful. You quickly see how stable the short averages 
are, and you can just hit CTRL-C when you've seen enough, without having 
to decide the suitable test length before hand.

It's also useful to see how checkpoints or autovacuum affects the 
transaction rate.

That said, no objection to removing the -P shorthand.

- Heikki

Hello Tom.

> I just wasted some time puzzling over strange results from pgbench.
> I eventually realized that I'd been testing against the wrong server,
> because rather than "-p 65432" I'd typed "-P 65432", thereby invoking
> the recently added --progress option.  pgbench has no way to know that
> that isn't what I meant; the fact that both switches take integer
> arguments doesn't help.

ISTM that this is an unfortunate but unlikely mistake, as "-p" is used in 
all postgresql commands to signify the port number (psql, pg_dump, 
pg_basebackup, createdb, ...).

> To fix this, I propose removing the -P short form and only allowing the
> long --progress form.

I do not think that such a "fix" is really needed. This logic would lead 
to remove many short options from many commands in postgresql and 
elsewhere : -t/-T in pgbench, -s/-S in psql, and so on, -l/-L -r/-R -s/-S 
in ls...

Moreover, I use -P more often than -p:-)

-- 
Fabien.

Hello Tom,

> Meh.  A progress-reporting feature has use when the tool is working
> towards completion of a clearly defined task.  In the case of pgbench,
> if you told it to run for -T 60 seconds rather than -T 10 seconds,
> that's probably because you don't trust a 10-second average to be
> sufficiently reproducible.

The motivation for the progress options are:

(1) to check for (not blindly trust) the performance stability, especially 
as warming up time can be very long. See for instance my blog post:
    http://blog.coelho.net/database/2013/08/14/postgresql-warmup/

a scaled 100 read-only pgbench run on a standard HDD requires 18 minutes 
to reach the performance steady state, and the performance is multiplied 
by 120 along the way, mostly in the last 2 minutes. In my experience 10 
and 60 seconds running period are equally ridiculously short running times 
for real benchmarks. When I am running a bench for 30 minutes, I like to 
have some output before the end of the command to know what is going on.

(2) when reporting performance figures, benchmark rules usually require 
that the detailed performance during the whole run are also reported, not 
just the final average, so as to rule out warming up or other unexpected 
and transitional effects.

(3) another use case of the option is to run with --rate (to target some 
tps you expect on your system) and then to run other commands in parallel 
(say pg_dump, pg_basebackup...) to check the impact it has on performance.

I do agree that having report every second on a 10 second run is not very 
useful, but that is not the use case.

> So I'm not real sure that reporting averages over shorter intervals is 
> all that useful; especially not if it takes cycles out of pgbench, which 
> itself is often a bottleneck.

If you do not ask for it, it does not harm the performance significantly.

> I could see some value in a feature that computed shorter-interval TPS
> averages and then did some further arithmetic, like measuring the standard
> deviation of the shorter-interval averages to assess how much noise there
> will be in the full-run average.

I do not understand. "pgbench -P" does report the standard deviation as 
well as the client side latency. Without this option pgbench is a black 
box.

> But that's not what this does, and if it did do that, "reporting 
> progress" would not be what I'd see as its main purpose.

This is for benchmarking. It is really reporting progress towards 
performance steady state, not reporting progress towards task completion.
Maybe a better name could have been thought for.

-- 
Fabien.

Fabien COELHO wrote:

> >I just wasted some time puzzling over strange results from pgbench.
> >I eventually realized that I'd been testing against the wrong server,
> >because rather than "-p 65432" I'd typed "-P 65432", thereby invoking
> >the recently added --progress option.  pgbench has no way to know that
> >that isn't what I meant; the fact that both switches take integer
> >arguments doesn't help.
> 
> ISTM that this is an unfortunate but unlikely mistake, as "-p" is
> used in all postgresql commands to signify the port number (psql,
> pg_dump, pg_basebackup, createdb, ...).

Plus other tools already use -P for progress, such as rsync.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Fabien COELHO wrote:
>>> I just wasted some time puzzling over strange results from pgbench.
>>> I eventually realized that I'd been testing against the wrong server,
>>> because rather than "-p 65432" I'd typed "-P 65432", thereby invoking
>>> the recently added --progress option.  pgbench has no way to know that
>>> that isn't what I meant; the fact that both switches take integer
>>> arguments doesn't help.

>> ISTM that this is an unfortunate but unlikely mistake, as "-p" is
>> used in all postgresql commands to signify the port number (psql,
>> pg_dump, pg_basebackup, createdb, ...).

> Plus other tools already use -P for progress, such as rsync.

Yeah, but they don't make -P take an integer argument.  It's that
little frammish that makes this problem significant.

I don't object to having the --progress switch.  I just think we
could live without a short form for it.
        regards, tom lane

>>> ISTM that this is an unfortunate but unlikely mistake, as "-p" is
>>> used in all postgresql commands to signify the port number (psql,
>>> pg_dump, pg_basebackup, createdb, ...).
>
>> Plus other tools already use -P for progress, such as rsync.
>
> Yeah, but they don't make -P take an integer argument.  It's that
> little frammish that makes this problem significant.

I do not see a strong case to make options with arguments case insensitive 
as a general rule. If this is done for -p/-P, why not -t/-T?

If you really fell you must remove -P, please replace it by another 
one-letter, I use this option nearly everytime a run pgbench.

-- 
Fabien.

Fabien COELHO <coelho@cri.ensmp.fr> writes:
>> Yeah, but they don't make -P take an integer argument.  It's that
>> little frammish that makes this problem significant.

> I do not see a strong case to make options with arguments case insensitive 
> as a general rule. If this is done for -p/-P, why not -t/-T?

I have not proposed fooling with -t/-T; people are used to that one,
and it's a core part of what pgbench does, so it's reasonable to expect
that people are familiar with it.  It helps also that -t and -T do
somewhat related things, ie constrain the length of the test --- so even
if you pick the wrong one, you still get behavior that's somewhat sane.

> If you really fell you must remove -P, please replace it by another 
> one-letter, I use this option nearly everytime a run pgbench.

Meh.  If I thought -P would be that popular, I'd expect people to get
used to the issue.  I don't believe it though.
        regards, tom lane

(2014/02/28 2:39), Tom Lane wrote:
> Fabien COELHO <coelho@cri.ensmp.fr> writes:
>>> Yeah, but they don't make -P take an integer argument.  It's that
>>> little frammish that makes this problem significant.
> 
>> I do not see a strong case to make options with arguments case insensitive
>> as a general rule. If this is done for -p/-P, why not -t/-T?
I'll say the same thing. And if we remove -P short option in pgbench, it means
that -P with integer will be forbided in postgres command. Surely, we don't hope so.

>> If you really fell you must remove -P, please replace it by another
>> one-letter, I use this option nearly everytime a run pgbench.
> 
> Meh.  If I thought -P would be that popular, I'd expect people to get
> used to the issue.  I don't believe it though.
At least, a user which is interested in postgres performance tuning(include
kernel options, etc) will often use this option. I recommended this feature,
because we can see the bottle-neck which we have not seen:) I believe you will
also become to like it more and more, while you use it.

Regards,
--
Mitsumasa KONDO
NTT Open Source Software Center