Thread: Proposal to CREATE FOREIGN TABLE LIKE
Hi, all
I wanted to bring up an idea that could really help out.
Our DBA team uses foreign tables for ETL processes in Greenplum and Cloudberry,
and we often need to create foreign tables that match the column definitions of local tables.
When dealing with wide tables and lots of those foreign tables, it can get pretty tedious and mistakes happen easily.
We end up having to troubleshoot errors when querying, which is a hassle.
Sure, we could use
It would work similarly to
And since Postgres doesn’t enforce constraints on foreign tables, it’s up to the user to make sure the constraints match the actual data.
https://www.postgresql.org/docs/current/sql-createforeigntable.html
This means that enabling
I haven’t rush with the codes yet, but it seems like it could be straightforward to implement by tweaking the existing limitations:
```
static void
transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_clause)
{
...
if (cxt->isforeign)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIKE is not supported for creating foreign tables")));
}
```
with some test cases and Documents changes.
I wanted to bring up an idea that could really help out.
Our DBA team uses foreign tables for ETL processes in Greenplum and Cloudberry,
and we often need to create foreign tables that match the column definitions of local tables.
When dealing with wide tables and lots of those foreign tables, it can get pretty tedious and mistakes happen easily.
We end up having to troubleshoot errors when querying, which is a hassle.
Sure, we could use
pg_dump
to get the table DDL and modify the name, but that just adds more busywork.CREATE FOREIGN TABLE LIKE
command could save a lot of time and reduce errors in the long run.It would work similarly to
CREATE TABLE LIKE
, copying the column definitions and constraints from the source table.And since Postgres doesn’t enforce constraints on foreign tables, it’s up to the user to make sure the constraints match the actual data.
https://www.postgresql.org/docs/current/sql-createforeigntable.html
This means that enabling
CREATE FOREIGN TABLE LIKE
shouldn’t introduce more issues with constraintsI haven’t rush with the codes yet, but it seems like it could be straightforward to implement by tweaking the existing limitations:
```
static void
transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_clause)
{
...
if (cxt->isforeign)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg("LIKE is not supported for creating foreign tables")));
}
```
with some test cases and Documents changes.
www.hashdata.xyz
On 2025-Feb-01, Zhang Mingli wrote: > For example, we use kafka_fdw to produce and consume data from a Kafka > server. In our scenario, we sometimes need to write records from a > local table into Kafka. Here’s a brief outline of our process: > > 1. We already have a wide table, local_wide_table in Postgres. > 2. We need to create a foreign table, foreign_table, with the same > definition as local_wide_table. > 3. Insert records into foreign_table by selecting > from local_wide_table with the some quals. > > In step 2, we currently have to manually create the foreign table > using CREATE FOREIGN TABLE and copy the column definitions one by one. Eh yeah, I guess for this use case it makes sense to allow a LIKE clause on CREATE FOREIGN TABLE. Were you going to submit a patch? -- Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/ "Having your biases confirmed independently is how scientific progress is made, and hence made our great society what it is today" (Mary Gardiner)
On 2025-Feb-06, Zhang Mingli wrote: > Added support for CREATE FOREIGN TABLE LIKE to enable the creation of > foreign tables based on the column definitions, constraints of > existing source tables. > This feature mirrors the behavior of CREATE TABLE LIKE, but ignores > inapplicable options such as INCLUDING INDEXES and INCLUDING > COMPRESSION for foreign tables. I think it'd be better to throw errors if they are given -- but INCLUDING ALL should be made to work in a different way than today so that it doesn't raise errors uselessly. Right now it works by setting all the bits in the value, um. -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
On 2025-Feb-06, Álvaro Herrera wrote: > On 2025-Feb-06, Zhang Mingli wrote: > > > Added support for CREATE FOREIGN TABLE LIKE to enable the creation of > > foreign tables based on the column definitions, constraints of > > existing source tables. > > This feature mirrors the behavior of CREATE TABLE LIKE, but ignores > > inapplicable options such as INCLUDING INDEXES and INCLUDING > > COMPRESSION for foreign tables. > > I think it'd be better to throw errors if they are given -- but > INCLUDING ALL should be made to work in a different way than today so > that it doesn't raise errors uselessly. Right now it works by setting > all the bits in the value, um. Ah, but our fine manual already says The LIKE clause can also be used to copy column definitions from views, foreign tables, or composite types. Inapplicable options (e.g., INCLUDING INDEXES from a view) are ignored. so what you implemented seems to be okay from that POV. -- Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/ "Investigación es lo que hago cuando no sé lo que estoy haciendo" (Wernher von Braun)
On Feb 6, 2025 at 18:31 +0800, Álvaro Herrera <alvherre@alvh.no-ip.org>, wrote:
Ah, but our fine manual already says
The LIKE clause can also be used to copy column definitions from views,
foreign tables, or composite types. Inapplicable options (e.g.,
INCLUDING INDEXES from a view) are ignored.
so what you implemented seems to be okay from that POV.
Hi, Yeah,
Our current convention is to ignore any inapplicable options without throwing errors.
As you mentioned, we use bits to identify the options, which does add some complexity to the codes if we try to track the origin of the option bits.
--
Zhang Mingli
HashData
Our current convention is to ignore any inapplicable options without throwing errors.
As you mentioned, we use bits to identify the options, which does add some complexity to the codes if we try to track the origin of the option bits.
--
Zhang Mingli
HashData
On Feb 6, 2025 at 18:09 +0800, Zhang Mingli <zmlpostgres@gmail.com>, wrote:
On Feb 3, 2025 at 08:29 +0800, Michael Paquier <michael@paquier.xyz>, wrote:On Mon, Feb 03, 2025 at 06:22:13AM +0800, Mingli Zhang wrote:Hi,Yes, I would like to provide a patch.
Glad to see we have come to an agreement on this.
Just adding my +1 here. FWIW.
Patch added.
> Patch V2 addressed the comments. Overall this LGTM. I still see a "no real storage" in v2 that should be removed from the documentation. + Foreign tables have no real storage in PostgreSQL. + Inapplicable options: <literal>INCLUDING INDEXES</literal>, <literal>INCLUDING STORAGE</literal>, I think the test coverage to check for the negative conditions only is enough. Regards, Sami
On Feb 18, 2025 at 08:49 +0800, Michael Paquier <michael@paquier.xyz>, wrote:
On Mon, Feb 17, 2025 at 07:14:59PM +0800, Zhang Mingli wrote:On Feb 17, 2025 at 15:24 +0800, Michael Paquier <michael@paquier.xyz>, wrote:+ * For foreign tables, they have no storage in Postgres.
+ * Inapplicable options are ignored.
Wording is a bit strange here.
* Foreign tables do not store data in Postgres.
* Any options that are not applicable for foreign tables will be ignored:
I would do something like that, perhaps, though I could get that
people don't like this suggestion:
"Some options are ignored. For example, as foreign tables have no
storage, these options have no effect: storage, compression, identity
and indexes. Similarly, INCLUDING INDEXES is ignored from a view."
OK.
I also didn't realize this until I wrote this patch. This could be
useful for the planner?
Constraints can be used as hints in the planner when working on
foreign tables. I'm pretty sure that this is the same reason here,
seeing that this is supported since v10 where statistics have been
introduced. I would need to dig more into the code, but that's not
really the point for this thread..
Agree.
+ Inapplicable options: <literal>INCLUDING INDEXES</literal>, <literal>INCLUDING STORAGE</literal>,
+ <literal>INCLUDING COMPRESSION</literal>, <literal>INCLUDING IDENTITY</literal> are ignored.
I would remove this paragraph, actually. The options supported are
listed by your patch, and that would be one area less to update if a
new INCLUDING flavor is added.
OK.
Copy-pasting the details of how the LIKE options work to the
create_foreign_table.sgml page is OK for me, and perhaps this will
diverge a bit from the CREATE TABLE part. One thing is that LIKE is
not part of the SQL specification for CREATE FOREIGN TABLE. Perhaps
this should be mentioned at the bottom of the page under the
"compatibility" section?
Good point.
Will address the comments later, thanks for review!
--
Zhang Mingli
HashData
Will address the comments later, thanks for review!
--
Zhang Mingli
HashData
On Feb 19, 2025 at 14:53 +0800, Michael Paquier <michael@paquier.xyz>, wrote:
There was a hole in the tests for the option LIKE_STORAGE. Removing
the check for it in transformTableLikeClause() did now show a diff in
the tests. In the case of foreign tables, extended for storage is a
correct choice when using a text type for an attribute. It makes more
sense to use something like "main" on the origin table, then check
that the foreign table uses "extended", for example.
You're right.
That was my mistake when I squashed the independent `like_options` cases into the two cases (`INCLUDING ALL`/`EXCLUDING ALL`) ,
particularly where there is an `ALTER STORAGE` before creating the foreign table, which shows the STORAGE difference.
Thanks for the correction.
That was my mistake when I squashed the independent `like_options` cases into the two cases (`INCLUDING ALL`/`EXCLUDING ALL`) ,
particularly where there is an `ALTER STORAGE` before creating the foreign table, which shows the STORAGE difference.
Thanks for the correction.
\d+ for a foreign table has no compression field, so using
HIDE_TOAST_COMPRESSION has no meaning. Removing the check for the
option LIKE_COMPRESSION leads to no diffs in the regression tests.
The other two restrictions for indexes and identity were OK.
The docs are fine after a closer look, relying mostly on the clauses
supported by the CREATE FOREIGN TABLE command, tweaked a bit the part
at the bottom where LIKE is not part of the standard.
And applied.
Thanks.
--
Zhang Mingli
HashData
--
Zhang Mingli
HashData
On Wed, Feb 19, 2025 at 03:40:56PM +0800, Zhang Mingli wrote: > Thanks for the correction. No problem. If you see anything else that you think is misleading, please feel free to ping me on this thread. -- Michael