Thread: Desirability of client-side expressions in psql?

Hello devs,

This is a discussion without actual patch intended for pg12, to be added 
to CF 2018-09. The expected end result is either "returned with feedback", 
meaning proceed to send some big patch(es), or "rejected", meaning the 
project does not want this, no point in submitting something.

Client "psql" has an "\if" which can test a boolean value and has 
":"-prefixed variables, including special presets such as ":VERSION_NUM" 
and ":SERVER_VERSION_NUM".

The features are already usable because one can do server-side expressions 
(if connected), which is a little cumbersome and ugly but nevertheless 
functional, eg:

   SELECT :VERSION_NUM = :SERVER_VERSION_NUM AS "same_version" \gset
   \if :same_version
     ...

However, when the "\if" patch was discussed, there was the underlying idea 
to extend psql so as to add client-side expression. That would allow 
things like:

   \let i <some arithmetic or logical expression...>
   \if :VERSION_NUM = :SERVER_VERSION_NUM
     ...

Before eventually starting on this subject with a loose objective of 
targeting 12.0, I would like to ascertain, especially from committers, but 
also from others, that:

(1) the objective is desirable (i.e. avoid ending with "we do not want
     this feature on principle, the cost-benefit balance is not good
     enough").

(2) maybe have a feedback on the proposed changes (not necessarily
     distinct patches, opinions are welcome), which would be to:
     (a) extend pgbench expressions so that they can handle what
         psql can do (eg variable-exists test which are available in psql)
     (b) maybe do some preliminary refactoring (eg create
         "pgbench/expression.c", "pgbench/variable.c")
     (c) move the pgbench expression engine to fe-utils
         (lexer, parser, execution...),
     (d) do some renaming (eg some "feex" prefix for "front-end
         expressions" to the various functions & types?),
     (e) abstract pgbench and psql variables so that they can be used
         transparently by expressions (i.e. some API alignment)
     (f) connect the engine to "psql"
     (g) create a shared documentation about these expressions,
         referenced from both psql and pgbench documentations.
     (h) provide non-regression tests on psql side as well.

The overall transformation would be quite large (about 2000 lines moved 
around) but "quite" simple (it is moving an existing, working and tested 
feature to allow using it elsewhere), not a lot of new code per se.

-- 
Fabien.

Hello Pavel,

>>> This is a discussion without actual patch intended for pg12, to be added
>>> to CF 2018-09. The expected end result is either "returned with feedback",
>>> meaning proceed to send some big patch(es), or "rejected", meaning the
>>> project does not want this, no point in submitting something.
>
> please, can you port expression evaluation from pgbench to psql? I miss a
> patch.

Indeed, I have not done it yet:-)

This is a pretty large series of patches, which would take a significant 
time for developing (me), rewiewing (not me) and committing (with overload 
committers). I would not like to end in the "we do not want this feature 
at all" state *after* the time has been spent, hence this pre-patch call 
for discussion about the feature itself, before diving into coding.

I take note that you are interested in actually having this feature.

> I am sure so psql expression evaluation has strong benefit - mainly for
> expressions like
>
> \if SERVER_VERSION_NUM >= xxxx
>  ...
> \endif

Note that this can already be done by relying on server-side expressions:

  SELECT :SERVER_VERSION_NUM >= 110000 AS "version_11_plus" \gset
  \if :version_11_plus
    ...
  \endif

Not very elegant, but functional. I'm not sure there is a compelling 
reason to have this feature beyond elegance.

-- 
Fabien.

>> Note that this can already be done by relying on server-side expressions:
>>
>>  SELECT :SERVER_VERSION_NUM >= 110000 AS "version_11_plus" \gset
>>  \if :version_11_plus
>>    ...
>>  \endif
>>
>> Not very elegant, but functional. I'm not sure there is a compelling
>> reason to have this feature beyond elegance.
>
> not elegant, not readable, and not user friendly.

Sure, I can only agree.

> the possibility of simple expression evaluation is just practical.

Ok. Good.

I do not mind spending some time for elegance, but I mind spending time 
for nothing.

I hope some committers will also express their views about the feature.

-- 
Fabien.

On Mon, Jun 25, 2018 at 1:29 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
>
> I do not mind spending some time for elegance, but I mind spending time for
> nothing.
>
> I hope some committers will also express their views about the feature.
>

There's a commitfest entry for this but no patch. So I am marking this
as RWF. The discussion here can continue. Please submit a patch (may
be a PoC) to the commitfest whenever there's one.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

On Tue, Jul 3, 2018 at 8:43 AM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:
> On Mon, Jun 25, 2018 at 1:29 AM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:
>>
>> I do not mind spending some time for elegance, but I mind spending time for
>> nothing.
>>
>> I hope some committers will also express their views about the feature.
>>
>
> There's a commitfest entry for this but no patch. So I am marking this
> as RWF. The discussion here can continue. Please submit a patch (may
> be a PoC) to the commitfest whenever there's one.
>


I set it to Rejected. This seems like a misuse of the CF process,
which is about discussing actual patches.

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hello Andrew,

> I set it to Rejected. This seems like a misuse of the CF process,
> which is about discussing actual patches.

Somehow.

As I have not received feedback from committers about the desirability of 
the feature, I interpret that as "the feature is not desirable", and I 
will not develop it, because of the probability that this would be time 
down the drain.

-- 
Fabien.

Greeting Fabien,

* Fabien COELHO (coelho@cri.ensmp.fr) wrote:
> >I set it to Rejected. This seems like a misuse of the CF process,
> >which is about discussing actual patches.
>
> Somehow.

So, I tend to agree w/ Andrew that while this is a good topic to have on
-hackers, it shouldn't be a CF entry.  I wouldn't draw any conclusions
from Andrew closing it out as "not appropriate for CF".

> As I have not received feedback from committers about the desirability of
> the feature, I interpret that as "the feature is not desirable", and I will
> not develop it, because of the probability that this would be time down the
> drain.

Personally, I'm definitely in favor of having a lot more flexibility and
capability in psql, that's an area which I think we don't focus on
nearly enough.  Having to fight with bash or another language to make
calls to psql to get things done is downright annoying.

So, +1 from me on the overall idea.  The challenge here will almost
certainly be in the details.  I do like the proposal you have of
building out a common set of capabilities which are shared between psql
and pgbench.

The other big challenge here is figuring out how to distinguish between
SQL which should be sent to the server and something which needs to be
client-side processed.  I've never liked the ':var' approach and it
really sucks when you want to combine that variable with something else,
but it's what we've got and therefore has history behind it.  If you can
find a way to improve on that without breaking existing code, that'd be
fantastic.  If not, definitely try to minimize the impact.

Thanks!

Stephen

Hello Stephen,

> [...] So, I tend to agree w/ Andrew that while this is a good topic to 
> have on -hackers, it shouldn't be a CF entry.  I wouldn't draw any 
> conclusions from Andrew closing it out as "not appropriate for CF".

Sure. As I had no committer feedback on the discussion for 3 months, I 
tried this as an ineffective way to get some. It did not work up to now.

>> As I have not received feedback from committers about the desirability of
>> the feature, I interpret that as "the feature is not desirable", and I will
>> not develop it, because of the probability that this would be time down the
>> drain.
>
> Personally, I'm definitely in favor of having a lot more flexibility and
> capability in psql, that's an area which I think we don't focus on
> nearly enough.  Having to fight with bash or another language to make
> calls to psql to get things done is downright annoying.
>
> So, +1 from me on the overall idea.

Good, that is one committer opinion, an infinite improvement over the 
previous status:-)

> The challenge here will almost certainly be in the details.

Yep. I'm fine with "your code is not good and creates problems so it is 
rejected". I'm trying to avoid "your code was a loss from the start, 
whatever you did, because we do not want such a feature".

> I do like the proposal you have of building out a common set of 
> capabilities which are shared between psql and pgbench.

Good.

> The other big challenge here is figuring out how to distinguish between 
> SQL which should be sent to the server and something which needs to be 
> client-side processed.

The current great idea is to use backslash commands to define existing 
variables:

   psql> \let i  1 + 2 * 3
   psql> SELECT :i ;
   psql> \if :i >= 5
   psql>   ...
   psql> \endif

> I've never liked the ':var' approach and it really sucks when you want 
> to combine that variable with something else, but it's what we've got 
> and therefore has history behind it.

Indeed. I do not think that changing this would make much sense.

> If you can find a way to improve on that without breaking existing code, 
> that'd be fantastic.  If not, definitely try to minimize the impact.

I was just planning to set existing :-variables with expressions, I have 
no great other idea. Mixing languages is always a pain.

Thanks for your feedback.

-- 
Fabien.

Hello Corey,

>>    psql> \if :i >= 5
>>
> I think we're ok with that so long as none of the operators or values has a
> \ in it.
> What barriers do you see to re-using the pgbench grammar?

The pgbench expression grammar mimics SQL expression grammar,
on integers, floats, booleans & NULL.

I'm unsure about some special cases in psql (`shell command`,
'text' "identifier"). They can be forbidden on a new commande (\let),
but what happens on "\if ..." which I am afraid allows them is unclear.

-- 
Fabien.