Thread: PL/pgSQL: Incomplete item Allow handling of %TYPE arrays, e.g. tab.col%TYPE[]

Implement TODO item:
PL/pgSQL
Incomplete item Allow handling of %TYPE arrays, e.g. tab.col%TYPE[]

As a first step, deal only with [], such as
xxx.yyy%TYPE[]
xxx%TYPE[]

It can be extended to support multi-dimensional and complex syntax in 
the future.


--
Quan Zongliang

> On 16 Oct 2023, at 12:15, Quan Zongliang <quanzongliang@yeah.net> wrote:

> Implement TODO item:
> PL/pgSQL
> Incomplete item Allow handling of %TYPE arrays, e.g. tab.col%TYPE[]

Cool!  While I haven't looked at the patch yet, I've wanted this myself many
times in the past when working with large plpgsql codebases.

> As a first step, deal only with [], such as
> xxx.yyy%TYPE[]
> xxx%TYPE[]
>
> It can be extended to support multi-dimensional and complex syntax in the future.

Was this omitted due to complexity of implementation or for some other reason?

--
Daniel Gustafsson

Attached new patch
   More explicit error messages based on type.


On 2023/10/16 18:15, Quan Zongliang wrote:
> 
> 
> Implement TODO item:
> PL/pgSQL
> Incomplete item Allow handling of %TYPE arrays, e.g. tab.col%TYPE[]
> 
> As a first step, deal only with [], such as
> xxx.yyy%TYPE[]
> xxx%TYPE[]
> 
> It can be extended to support multi-dimensional and complex syntax in 
> the future.
> 
> 
> -- 
> Quan Zongliang

Error messages still seem ambiguous.
    do not support multi-dimensional arrays in PL/pgSQL

Isn't that better?
    do not support multi-dimensional %TYPE arrays in PL/pgSQL


On 2023/10/17 09:19, Quan Zongliang wrote:
> 
> Attached new patch
>    More explicit error messages based on type.
> 
> 
> On 2023/10/16 18:15, Quan Zongliang wrote:
>>
>>
>> Implement TODO item:
>> PL/pgSQL
>> Incomplete item Allow handling of %TYPE arrays, e.g. tab.col%TYPE[]
>>
>> As a first step, deal only with [], such as
>> xxx.yyy%TYPE[]
>> xxx%TYPE[]
>>
>> It can be extended to support multi-dimensional and complex syntax in 
>> the future.
>>
>>
>> -- 
>> Quan Zongliang

On 2023/10/16 20:05, Pavel Stehule wrote:
> 
> 
> po 16. 10. 2023 v 13:56 odesílatel Daniel Gustafsson <daniel@yesql.se 
> <mailto:daniel@yesql.se>> napsal:
> 
>      > On 16 Oct 2023, at 12:15, Quan Zongliang <quanzongliang@yeah.net
>     <mailto:quanzongliang@yeah.net>> wrote:
> 
>      > Implement TODO item:
>      > PL/pgSQL
>      > Incomplete item Allow handling of %TYPE arrays, e.g. tab.col%TYPE[]
> 
>     Cool!  While I haven't looked at the patch yet, I've wanted this
>     myself many
>     times in the past when working with large plpgsql codebases.
> 
>      > As a first step, deal only with [], such as
>      > xxx.yyy%TYPE[]
>      > xxx%TYPE[]
>      >
>      > It can be extended to support multi-dimensional and complex
>     syntax in the future.
> 
>     Was this omitted due to complexity of implementation or for some
>     other reason?
> 
Because of complexity.

> 
> There is no reason for describing enhancement. The size and dimensions 
> of postgresql arrays are dynamic, depends on the value, not on 
> declaration. Now, this information is ignored, and can be compatibility 
> break to check and enforce this info.
> 
Yes. I don't think it's necessary.
If anyone needs it, we can continue to enhance it in the future.

> 
>     --
>     Daniel Gustafsson
> 
> 
> 

On 2023/10/17 12:15, Pavel Stehule wrote:
> 
> 
> út 17. 10. 2023 v 3:30 odesílatel Quan Zongliang <quanzongliang@yeah.net 
> <mailto:quanzongliang@yeah.net>> napsal:
> 
> 
> 
>     On 2023/10/16 20:05, Pavel Stehule wrote:
>      >
>      >
>      > po 16. 10. 2023 v 13:56 odesílatel Daniel Gustafsson
>     <daniel@yesql.se <mailto:daniel@yesql.se>
>      > <mailto:daniel@yesql.se <mailto:daniel@yesql.se>>> napsal:
>      >
>      >      > On 16 Oct 2023, at 12:15, Quan Zongliang
>     <quanzongliang@yeah.net <mailto:quanzongliang@yeah.net>
>      >     <mailto:quanzongliang@yeah.net
>     <mailto:quanzongliang@yeah.net>>> wrote:
>      >
>      >      > Implement TODO item:
>      >      > PL/pgSQL
>      >      > Incomplete item Allow handling of %TYPE arrays, e.g.
>     tab.col%TYPE[]
>      >
>      >     Cool!  While I haven't looked at the patch yet, I've wanted this
>      >     myself many
>      >     times in the past when working with large plpgsql codebases.
>      >
>      >      > As a first step, deal only with [], such as
>      >      > xxx.yyy%TYPE[]
>      >      > xxx%TYPE[]
>      >      >
>      >      > It can be extended to support multi-dimensional and complex
>      >     syntax in the future.
>      >
>      >     Was this omitted due to complexity of implementation or for some
>      >     other reason?
>      >
>     Because of complexity.
> 
>      >
>      > There is no reason for describing enhancement. The size and
>     dimensions
>      > of postgresql arrays are dynamic, depends on the value, not on
>      > declaration. Now, this information is ignored, and can be
>     compatibility
>      > break to check and enforce this info.
>      >
>     Yes. I don't think it's necessary.
>     If anyone needs it, we can continue to enhance it in the future.
> 
> 
> I don't think it is possible to do it.  But there is another missing 
> functionality, if I remember well. There is no possibility to declare 
> variables for elements of array.
The way it's done now is more like laziness.

Is it possible to do that?
If the parser encounters %TYPE[][]. It can be parsed. Then let 
parse_datatype do the rest.

For example, partitioned_table.a%TYPE[][100][]. Parse the type 
name(int4) of partitioned_table.a%TYPE and add the following [][100][]. 
Passing "int4[][100][]" to parse_datatype will give us the array 
definition we want.

Isn't this code a little ugly?

> 
> I propose syntax xxx.yyy%ELEMENTTYPE and xxx%ELEMENTTYPE
> 
> What do you think about it?
No other relational database can be found with such an implementation. 
But it seems like a good idea. It can bring more convenience to write 
stored procedure.

> 
> Regards
> 
> Pavel
> 
> 
>      >
>      >     --
>      >     Daniel Gustafsson
>      >
>      >
>      >
> 

On 2023/11/20 17:33, Pavel Stehule wrote:

> 
> 
> I did some deeper check:
> 
> - I don't like too much parser's modification (I am sending alternative 
> own implementation) - the SQL parser allows richer syntax, and for full 
> functionality is only few lines more
Agree.

> 
> - original patch doesn't solve %ROWTYPE
> 
> (2023-11-20 10:04:36) postgres=# select * from foo;
> ┌────┬────┐
> │ a  │ b  │
> ╞════╪════╡
> │ 10 │ 20 │
> │ 30 │ 40 │
> └────┴────┘
> (2 rows)
> 
> (2023-11-20 10:08:29) postgres=# do $$
> declare v foo%rowtype[];
> begin
>    v := array(select row(a,b) from foo);
>    raise notice '%', v;
> end;
> $$;
> NOTICE:  {"(10,20)","(30,40)"}
> DO
> 
two little fixes
1. spelling mistake
   ARRAY [ icons ] --> ARRAY [ iconst ]
2. code bug
   if (!OidIsValid(dtype->typoid)) --> if (!OidIsValid(array_typeid))


> - original patch doesn't solve type RECORD
> the error message should be more intuitive, although the arrays of 
> record type can be supported, but it probably needs bigger research.
> 
> (2023-11-20 10:10:34) postgres=# do $$
> declare r record; v r%type[];
> begin
>    v := array(select row(a,b) from foo);
>    raise notice '%', v;
> end;
> $$;
> ERROR:  syntax error at or near "%"
> LINE 2: declare r record; v r%type[];
>                               ^
> CONTEXT:  invalid type name "r%type[]"
> 
Currently only scalar variables are supported.
This error is consistent with the r%type error. And record arrays are 
not currently supported.
Support for r%type should be considered first. For now, let r%type[] 
report the same error as record[].
I prefer to implement it with a new patch.

> - missing documentation
My English is not good. I wrote it down, please correct it. Add a note 
in the "Record Types" documentation that arrays and "Copying Types" are 
not supported yet.

> 
> - I don't like using the word "partitioned" in the regress test name 
> "partitioned_table". It is confusing
fixed

> 
> Regards
> 
> Pavel

On 2023/11/24 03:39, Pavel Stehule wrote:

> 
> I modified the documentation a little bit - we don't need to extra 
> propose SQL array syntax, I think.
> I rewrote regress tests - we don't need to test unsupported 
> functionality (related to RECORD).
> 
> - all tests passed
> 
I wrote two examples of errors:
   user_id users.user_id%ROWTYPE[];
   user_id users.user_id%ROWTYPE ARRAY[4][3];

Fixed.

> Regards
> 
> Pavel
> 
> 
>      >
>      > Regards
>      >
>      > Pavel
>

Pavel Stehule <pavel.stehule@gmail.com> writes:
> Now, I think so this simple patch is ready for committers

I pushed this with some editorialization -- mostly, rewriting the
documentation and comments.  I found that the existing docs for %TYPE
were not great.  There are two separate use-cases, one for referencing
a table column and one for referencing a previously-declared variable,
and the docs were about as clear as mud about explaining that.

I also looked into the problem Pavel mentioned that it doesn't work
for RECORD.  If you just write "record[]" you get an error message
that at least indicates it's an unsupported case:

regression=# do $$declare r record[]; begin end$$;
ERROR:  variable "r" has pseudo-type record[]
CONTEXT:  compilation of PL/pgSQL function "inline_code_block" near line 1

Maybe we could improve on that, but it would be a lot of work and
I'm not terribly excited about it.  However, %TYPE fails entirely
for both "record" and named composite types, and the reason turns
out to be just that plpgsql_parse_wordtype fails to handle the
PLPGSQL_NSTYPE_REC case.  So that's easily fixed.

I also wonder what the heck the last half of plpgsql_parse_wordtype
is for at all.  It looks for a named type, which means you can do

regression=# do $$declare x float8%type; begin end$$;
DO

but that's just stupid.  You could leave off the %TYPE and get
the same result.  Moreover, it is inconsistent because
plpgsql_parse_cwordtype has no equivalent behavior:

regression=# do $$declare x pg_catalog.float8%type; begin end$$;
ERROR:  syntax error at or near "%"
LINE 1: do $$declare x pg_catalog.float8%type; begin end$$;
                                        ^
CONTEXT:  invalid type name "pg_catalog.float8%type"

It's also undocumented and untested (the code coverage report
shows this part is never reached).  So I propose we remove it.

That leads me to the attached proposed follow-on patch.

Another thing we could think about, but I've not done it here,
is to make plpgsql_parse_wordtype and friends throw error
instead of just returning NULL when they don't find the name.
Right now, if NULL is returned, we end up passing the whole
string to parse_datatype, leading to unhelpful errors like
the one shown above.  We could do better than that I think,
perhaps like "argument of %TYPE is not a known variable".

            regards, tom lane

diff --git a/src/pl/plpgsql/src/expected/plpgsql_record.out b/src/pl/plpgsql/src/expected/plpgsql_record.out
index afb922df29..36d65e4286 100644
--- a/src/pl/plpgsql/src/expected/plpgsql_record.out
+++ b/src/pl/plpgsql/src/expected/plpgsql_record.out
@@ -306,6 +306,32 @@ NOTICE:  r1 = (1,2)
 ERROR:  record "r1" has no field "nosuchfield"
 CONTEXT:  SQL expression "r1.nosuchfield"
 PL/pgSQL function inline_code_block line 9 at RAISE
+-- check that type record can be passed through %type
+do $$
+declare r1 record;
+        r2 r1%type;
+begin
+  r2 := row(1,2);
+  raise notice 'r2 = %', r2;
+  r2 := row(3,4,5);
+  raise notice 'r2 = %', r2;
+end$$;
+NOTICE:  r2 = (1,2)
+NOTICE:  r2 = (3,4,5)
+-- arrays of record are not supported at the moment
+do $$
+declare r1 record[];
+begin
+end$$;
+ERROR:  variable "r1" has pseudo-type record[]
+CONTEXT:  compilation of PL/pgSQL function "inline_code_block" near line 2
+do $$
+declare r1 record;
+        r2 r1%type[];
+begin
+end$$;
+ERROR:  variable "r2" has pseudo-type record[]
+CONTEXT:  compilation of PL/pgSQL function "inline_code_block" near line 3
 -- check repeated assignments to composite fields
 create table some_table (id int, data text);
 do $$
diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgsql/src/pl_comp.c
index b745eaa3f8..c63719c796 100644
--- a/src/pl/plpgsql/src/pl_comp.c
+++ b/src/pl/plpgsql/src/pl_comp.c
@@ -1596,8 +1596,8 @@ plpgsql_parse_tripword(char *word1, char *word2, char *word3,


 /* ----------
- * plpgsql_parse_wordtype    The scanner found word%TYPE. word can be
- *                a variable name or a basetype.
+ * plpgsql_parse_wordtype    The scanner found word%TYPE. word should be
+ *                a pre-existing variable name.
  *
  * Returns datatype struct, or NULL if no match found for word.
  * ----------
@@ -1605,10 +1605,7 @@ plpgsql_parse_tripword(char *word1, char *word2, char *word3,
 PLpgSQL_type *
 plpgsql_parse_wordtype(char *ident)
 {
-    PLpgSQL_type *dtype;
     PLpgSQL_nsitem *nse;
-    TypeName   *typeName;
-    HeapTuple    typeTup;

     /*
      * Do a lookup in the current namespace stack
@@ -1623,39 +1620,13 @@ plpgsql_parse_wordtype(char *ident)
         {
             case PLPGSQL_NSTYPE_VAR:
                 return ((PLpgSQL_var *) (plpgsql_Datums[nse->itemno]))->datatype;
-
-                /* XXX perhaps allow REC/ROW here? */
-
+            case PLPGSQL_NSTYPE_REC:
+                return ((PLpgSQL_rec *) (plpgsql_Datums[nse->itemno]))->datatype;
             default:
                 return NULL;
         }
     }

-    /*
-     * Word wasn't found in the namespace stack. Try to find a data type with
-     * that name, but ignore shell types and complex types.
-     */
-    typeName = makeTypeName(ident);
-    typeTup = LookupTypeName(NULL, typeName, NULL, false);
-    if (typeTup)
-    {
-        Form_pg_type typeStruct = (Form_pg_type) GETSTRUCT(typeTup);
-
-        if (!typeStruct->typisdefined ||
-            typeStruct->typrelid != InvalidOid)
-        {
-            ReleaseSysCache(typeTup);
-            return NULL;
-        }
-
-        dtype = build_datatype(typeTup, -1,
-                               plpgsql_curr_compile->fn_input_collation,
-                               typeName);
-
-        ReleaseSysCache(typeTup);
-        return dtype;
-    }
-
     /*
      * Nothing found - up to now it's a word without any special meaning for
      * us.
@@ -1689,8 +1660,8 @@ plpgsql_parse_cwordtype(List *idents)
     {
         /*
          * Do a lookup in the current namespace stack. We don't need to check
-         * number of names matched, because we will only consider scalar
-         * variables.
+         * number of names matched, because field references aren't supported
+         * here.
          */
         nse = plpgsql_ns_lookup(plpgsql_ns_top(), false,
                                 strVal(linitial(idents)),
@@ -1703,6 +1674,11 @@ plpgsql_parse_cwordtype(List *idents)
             dtype = ((PLpgSQL_var *) (plpgsql_Datums[nse->itemno]))->datatype;
             goto done;
         }
+        else if (nse != NULL && nse->itemtype == PLPGSQL_NSTYPE_REC)
+        {
+            dtype = ((PLpgSQL_rec *) (plpgsql_Datums[nse->itemno]))->datatype;
+            goto done;
+        }

         /*
          * First word could also be a table name
diff --git a/src/pl/plpgsql/src/sql/plpgsql_record.sql b/src/pl/plpgsql/src/sql/plpgsql_record.sql
index db655335b1..f0fd05ba48 100644
--- a/src/pl/plpgsql/src/sql/plpgsql_record.sql
+++ b/src/pl/plpgsql/src/sql/plpgsql_record.sql
@@ -199,6 +199,29 @@ begin
   raise notice 'r1.nosuchfield = %', r1.nosuchfield;
 end$$;

+-- check that type record can be passed through %type
+do $$
+declare r1 record;
+        r2 r1%type;
+begin
+  r2 := row(1,2);
+  raise notice 'r2 = %', r2;
+  r2 := row(3,4,5);
+  raise notice 'r2 = %', r2;
+end$$;
+
+-- arrays of record are not supported at the moment
+do $$
+declare r1 record[];
+begin
+end$$;
+
+do $$
+declare r1 record;
+        r2 r1%type[];
+begin
+end$$;
+
 -- check repeated assignments to composite fields
 create table some_table (id int, data text);