Thread: CALL stmt, ERROR: unrecognized node type: 113 bug

On Thu, Feb 01, 2018 at 05:33:54PM +0100, Pavel Stehule wrote:
> I am playing with procedures little bit
>
> I found few bugs
>
> create procedure test(a int)
> as $$
> begin
>   raise notice '>>>%<<<', a;
> end;
> $$ language plpgsql;
>
> call test(10); -- ok
>
> postgres=# call test((select 10));
> ERROR:  unrecognized node type: 113
>
> postgres=# \sf test
> ERROR:  cache lookup failed for type 0

Peter, Andrew, this is missing some bits related to the conversion of
SubLink nodes to SubPlan nodes for procedures when used as argument of
a procedure as only the latter can be executed after the former is
processed by the latter (see SS_process_sublinks).
--
Michael

On Fri, Feb 02, 2018 at 04:01:13PM +0900, Michael Paquier wrote:
> Peter, Andrew, this is missing some bits related to the conversion of
> SubLink nodes to SubPlan nodes for procedures when used as argument of
> a procedure as only the latter can be executed after the former is
> processed by the latter (see SS_process_sublinks).

Meh-to-self.

You need to read that as "only a SubPlan can be executed after a SubLink
has been processed by the planner", so please replace the last "latter"
by "planner".
--
Michael

On Fri, Feb 02, 2018 at 04:07:28PM +0900, Michael Paquier wrote:
> You need to read that as "only a SubPlan can be executed after a SubLink
> has been processed by the planner", so please replace the last "latter"
> by "planner".

(I forgot to add Peter and Andrew in CC: previously, so done now.)

e4128ee7 is making is clear that SubLink are authorized when
transforming it in transformSubLink(), however I cannot think about a
use case so should we just forbid them, and this is actually untested.
So the patch attached does so.

The second problem involves a cache lookup failure for a type when
trying to use pg_get_functiondef on a procedure.  Luckily, it is
possible to make the difference between a procedure and a function by
checking if prorettype is InvalidOid or not.  There is room for a new
patch which supports pg_get_proceduredef() to generate the definition of
a procedure, with perhaps a dedicated psql shortcut, but that could
always be done later on.
--
Michael

On Fri, Feb 09, 2018 at 12:02:57PM +0100, Pavel Stehule wrote:
> 2018-02-09 7:56 GMT+01:00 Michael Paquier <michael@paquier.xyz>:
> > The second problem involves a cache lookup failure for a type when
> > trying to use pg_get_functiondef on a procedure.  Luckily, it is
> > possible to make the difference between a procedure and a function by
> > checking if prorettype is InvalidOid or not.  There is room for a new
> > patch which supports pg_get_proceduredef() to generate the definition of
> > a procedure, with perhaps a dedicated psql shortcut, but that could
> > always be done later on.
>
> Blocking subqueries in CALL parameters is possible solution.

ExecuteCallStmt has visibly been written so as it is able to handle the
input set of arguments with a minimal infrastructure in place.  SubLink
nodes require more advanced handling as those need to go through the
planner.  There is also additional processing in the rewriter.  At the
end I tend to think that any user would just turn their back on calling
a function for such cases anyway, so it seems to me that the potential
benefits are not worth the code complexity.

> But blocking
> func def for procedures without any substitution doesn't look correct for
> me.

I don't disagree with you here, there is room for such a function, but
on the other side not having it does not make the existing feature less
useful.  As it is Peter's and Andrew's feature, the last word should
come from them.  Here is my opinion for what it's worth:
- Procedures are not functions, the code is pretty clear about that, so
a system function to generate the definition of a procedure should not
happen with pg_get_functiondef().  They share a lot of infrastructure so
you can reuse a lot of the code present.
- A different psql shortcut should be used, and I would assume that \sp
is adapted.
- Aggregates are also in pg_proc, we generate an error on them because
they are of different type, so an error for procedures when trying to
fetch a functoin definition looks like the good answer.

If folks feel that having a way to retrieve the procedure definition
easily via ruleutils.c is a must-have, then we have material for a good
open item :)
--
Michael

"David G. Johnston" <david.g.johnston@gmail.com> writes:
> On Fri, Feb 9, 2018 at 6:23 AM, Michael Paquier <michael@paquier.xyz> wrote:
>> On Fri, Feb 09, 2018 at 12:02:57PM +0100, Pavel Stehule wrote:
>>> Blocking subqueries in CALL parameters is possible solution.

> To me this feels like an interaction between two features that users are
> going to expect to just work.

Meh.  It doesn't look significantly different to me than the restriction
that you can't have sub-selects in CHECK expressions, index expressions,
etc.  Obviously we need a clean failure like you get for those cases.
But otherwise it's an OK restriction that stems from exactly the same
cause: we do not want to invoke the full planner in this context (and
even if we did, we don't want to use the full executor to execute the
result).

            regards, tom lane

On 2018-02-09 09:42:41 -0500, Tom Lane wrote:
> It doesn't look significantly different to me than the restriction
> that you can't have sub-selects in CHECK expressions, index
> expressions, etc.  Obviously we need a clean failure like you get for
> those cases.  But otherwise it's an OK restriction that stems from
> exactly the same cause: we do not want to invoke the full planner in
> this context (and even if we did, we don't want to use the full
> executor to execute the result).

+1

On Fri, Feb 09, 2018 at 08:30:49AM -0800, Andres Freund wrote:
> On 2018-02-09 09:42:41 -0500, Tom Lane wrote:
>> It doesn't look significantly different to me than the restriction
>> that you can't have sub-selects in CHECK expressions, index
>> expressions, etc.  Obviously we need a clean failure like you get for
>> those cases.  But otherwise it's an OK restriction that stems from
>> exactly the same cause: we do not want to invoke the full planner in
>> this context (and even if we did, we don't want to use the full
>> executor to execute the result).
>
> +1

+1.
--
Michael

Michael Paquier <michael@paquier.xyz> writes:
> On Fri, Feb 09, 2018 at 08:30:49AM -0800, Andres Freund wrote:
>> On 2018-02-09 09:42:41 -0500, Tom Lane wrote:
>>> ...  But otherwise it's an OK restriction that stems from
>>> exactly the same cause: we do not want to invoke the full planner in
>>> this context (and even if we did, we don't want to use the full
>>> executor to execute the result).

So I idly looked at ExecuteCallStmt to see how, in fact, it is executing
these things, and I was right to guess that it couldn't possibly support
a sub-select, since it's just using ExecEvalExprSwitchContext.  So we
need to go teach transformSubLink that EXPR_KIND_CALL is *not* okay,
which is simple enough.

However, I also wondered how ExecuteCallStmt works at all for pass-by-
reference datatypes, since it immediately destroys the execution context
for each expression.  And the answer is that it doesn't, as proven here:

regression=# create procedure myp(f1 text) as $$begin
regression$# raise notice 'f1 = %', f1;
regression$# end$$ language plpgsql;
CREATE PROCEDURE
regression=# call myp('xyzzy');
NOTICE:  f1 = xyzzy
CALL
regression=# call myp('xyzzy' || 'x');
NOTICE:  f1 = 
CALL

The call with a literal constant accidentally seems to work, because the
returned Datum is actually pointing into the original expression tree.
But if you have the expression do any actual work, then not so much.

I think this could be fixed by evaluating all the arguments in a single
execution context that is not destroyed till after the call finishes
(using a separate one for each argument seems pretty silly anyway).
However, the code could do with more than zero commentary about how come
this is safe at all --- even if we keep the execution context, it's still
a child of whatever random memory context ExecuteCallStmt was called in,
and it's not very clear that that context will survive a transaction
commit.

This does not leave me with a warm feeling about either the amount
of testing the procedure feature has gotten, or the state of its
internal documentation.

            regards, tom lane

I wrote:
> However, I also wondered how ExecuteCallStmt works at all for pass-by-
> reference datatypes, since it immediately destroys the execution context
> for each expression.  And the answer is that it doesn't, as proven here:

On closer inspection, there are actually three sub-cases involved.
It accidentally works for simple constant arguments, because the
passed Datum will point at a Const node generated during transformExpr.
And it works for fully run-time-evaluated arguments, because those end
up in memory belonging to the standalone ExprContext(s), which
ExecuteCallStmt never bothers to free at all.  (Which is a bug in itself,
although possibly one that wouldn't be exposed in practice given that
we disallow SRFs here; I don't know if there are any other cases that
would expect ExprContext cleanup hooks to get invoked.)  Where it doesn't
work is for expressions that are const-folded during ExecPrepareExpr,
because then the values are in Const nodes that live in the EState's
per-query context, and the code is throwing that away too soon.

I pushed a fix for all that.

The failure in pg_get_functiondef() is still there.  While the immediate
answer probably is to teach that function to emit correct CREATE PROCEDURE
syntax, I continue to think that it's a bad idea to be putting zeroes into
pg_proc.prorettype.

            regards, tom lane

On Sat, Feb 10, 2018 at 01:46:40PM -0500, Tom Lane wrote:
> I pushed a fix for all that.

Shouldn't there be a test case as well?  The patch I sent upthread was
doing the whole set, except that I did not bother

> The failure in pg_get_functiondef() is still there.  While the immediate
> answer probably is to teach that function to emit correct CREATE PROCEDURE
> syntax, I continue to think that it's a bad idea to be putting zeroes into
> pg_proc.prorettype.

Yeah, or an error with a new function dedicated to procedures.  I also
finc confusing the use of prorettype to track this object type.

This brings the amount of objects stored in pg_proc to four.  Perhaps it
would be time to bring more clarity in pg_proc by introducing a prokind
column for functions, aggregates, window functions and procedures?  I
don't feel really hot for an extra boolean column like proisproc.
--
Michael

On Sun, Feb 11, 2018 at 08:17:55AM +0900, Michael Paquier wrote:
> On Sat, Feb 10, 2018 at 01:46:40PM -0500, Tom Lane wrote:
>> I pushed a fix for all that.
>
> Shouldn't there be a test case as well?  The patch I sent upthread was
> doing the whole set, except that I did not bother

... Rename EXPR_KIND_CALL to something else.

This was missing the last part of the sentence.
--
Michael

Michael Paquier <michael@paquier.xyz> writes:
> On Sat, Feb 10, 2018 at 01:46:40PM -0500, Tom Lane wrote:
>> I pushed a fix for all that.

> Shouldn't there be a test case as well?

There was one for the premature-free issue in d02d4a6d4.  I didn't really
see a need for an explicit test for the subselect issue.

> This brings the amount of objects stored in pg_proc to four.  Perhaps it
> would be time to bring more clarity in pg_proc by introducing a prokind
> column for functions, aggregates, window functions and procedures?

Yeah.  I was under the impression that Peter was looking into that ...
[ digs... ] see
https://www.postgresql.org/message-id/80ee1f5c-fa9d-7285-ed07-cff53d4f4858@2ndquadrant.com

            regards, tom lane

On 2/9/18 09:42, Tom Lane wrote:
> Meh.  It doesn't look significantly different to me than the restriction
> that you can't have sub-selects in CHECK expressions, index expressions,
> etc.  Obviously we need a clean failure like you get for those cases.
> But otherwise it's an OK restriction that stems from exactly the same
> cause: we do not want to invoke the full planner in this context (and
> even if we did, we don't want to use the full executor to execute the
> result).

A close analogy is that EXECUTE parameters also don't accept subqueries.
 It would perhaps be nice if that could be made to work, but as
discussed it would require a bunch more work.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2/1/18 11:33, Pavel Stehule wrote:
> postgres=# \sf test
> ERROR:  cache lookup failed for type 0

Here is a patch set that adds procedure support to \ef and \sf.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2/11/18 01:10, Tom Lane wrote:
>> This brings the amount of objects stored in pg_proc to four.  Perhaps it
>> would be time to bring more clarity in pg_proc by introducing a prokind
>> column for functions, aggregates, window functions and procedures?
> 
> Yeah.  I was under the impression that Peter was looking into that ...
> [ digs... ] see
> https://www.postgresql.org/message-id/80ee1f5c-fa9d-7285-ed07-cff53d4f4858@2ndquadrant.com

Yeah that's still pending, but there wasn't a whole lot of reaction in
that thread.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 2/11/18 01:10, Tom Lane wrote:
>> Yeah.  I was under the impression that Peter was looking into that ...
>> [ digs... ] see
>> https://www.postgresql.org/message-id/80ee1f5c-fa9d-7285-ed07-cff53d4f4858@2ndquadrant.com

> Yeah that's still pending, but there wasn't a whole lot of reaction in
> that thread.

I think it's probably a good idea, but you hadn't finished the client
end of the patch.  Since the main argument against doing this would
probably be about client-side breakage, we need to see how big that
impact will be before we make a final decision.

            regards, tom lane

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 2/1/18 11:33, Pavel Stehule wrote:
>> postgres=# \sf test
>> ERROR:  cache lookup failed for type 0

> Here is a patch set that adds procedure support to \ef and \sf.

I've not read in detail, but it looks reasonable offhand, modulo
that I still don't like prorettype == 0 ;-).

I did notice a tiny typo:

-     * with.  Hence prefer "$function$", but extend if needed.
+     * with.  Hence prefer "$function$"/"$procedure", but extend if needed.

I think you want

+     * with.  Hence prefer "$function$"/"$procedure$", but extend if needed.

            regards, tom lane

On Mon, Feb 12, 2018 at 03:19:27PM -0500, Tom Lane wrote:
> I've not read in detail, but it looks reasonable offhand, modulo
> that I still don't like prorettype == 0 ;-).
>
> I did notice a tiny typo:
>
> -     * with.  Hence prefer "$function$", but extend if needed.
> +     * with.  Hence prefer "$function$"/"$procedure", but extend if needed.
>
> I think you want
>
> +     * with.  Hence prefer "$function$"/"$procedure$", but extend if needed.

0001 and 0002 are welcome.  I have a small comment on top of Tom's for 0003.

+   appendStringInfoString(&buf, ")\n");
+   if (proc->prorettype)
+   {
+       appendStringInfoString(&buf, " RETURNS ");
+       print_function_rettype(&buf, proctup);
+       appendStringInfoChar(&buf, '\n');
+   }
Could you use a separate boolean variable which is set as
OidIsValid(prorettype), say called isfunction?  The goal is to avoid the
check on prorettype in more than one place.  If pg_proc's shape is
changed depending on the discussion, the current patch is a recipy to
forget updating all those places.  A comment in pg_get_function_result
to mention that prorettype = InvalidOid is here to track that the call
involves a procedure would be nice.

Should the documentation of pg_function_is_visible also mention
procedures?
--
Michael

On 2/13/18 03:57, Michael Paquier wrote:
> On Mon, Feb 12, 2018 at 03:19:27PM -0500, Tom Lane wrote:
>> I've not read in detail, but it looks reasonable offhand, modulo
>> that I still don't like prorettype == 0 ;-).
>>
>> I did notice a tiny typo:
>>
>> -     * with.  Hence prefer "$function$", but extend if needed.
>> +     * with.  Hence prefer "$function$"/"$procedure", but extend if needed.
>>
>> I think you want
>>
>> +     * with.  Hence prefer "$function$"/"$procedure$", but extend if needed.

done

> 0001 and 0002 are welcome.  I have a small comment on top of Tom's for 0003.
> 
> +   appendStringInfoString(&buf, ")\n");
> +   if (proc->prorettype)
> +   {
> +       appendStringInfoString(&buf, " RETURNS ");
> +       print_function_rettype(&buf, proctup);
> +       appendStringInfoChar(&buf, '\n');
> +   }
> Could you use a separate boolean variable which is set as
> OidIsValid(prorettype), say called isfunction?

done

> Should the documentation of pg_function_is_visible also mention
> procedures?

done

and committed

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Feb 12, 2018 at 3:19 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> modulo that I still don't like prorettype == 0 ;-).

+1.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Feb 13, 2018 at 03:26:20PM -0500, Peter Eisentraut wrote:
> and committed

Thanks, those changes are fine for me.

Perhaps you want to have print_function_rettype() drop a elog(ERROR) if
called with an invalid prorettype?  I tend to be allergic to Asserts
in ruleutils.c since 0daeba0e...
--
Michael