Thread: Support logical replication of global object commands
Hello, Logical replication of DDL commands support is being worked on in [1]. However, global object commands are quite different from other non-global object DDL commands and need to be handled differently. For example, global object commands include ROLE statements, DATABASE statements, TABLESPACE statements and a subset of GRANT/REVOKE statements if the object being modified is a global object. These commands are different from other DDL commands in that: 1. Global object commands can be executed in any database. 2. Global objects are not schema qualified. 3. Global object commands are not captured by event triggers. I’ve put together a prototype to support logical replication of global object commands in the attached patch. This patch builds on the DDL replication patch from ZJ in [2] and must be applied on top of it. Here is a list of global object commands that the patch replicate, you can find more details in function LogGlobalObjectCommand: /* ROLE statements */ CreateRoleStmt AlterRoleStmt AlterRoleSetStmt DropRoleStmt ReassignOwnedStmt GrantRoleStmt /* Database statements */ CreatedbStmt AlterDatabaseStmt AlterDatabaseRefreshCollStmt AlterDatabaseSetStmt DropdbStmt /* TableSpace statements */ CreateTableSpaceStmt DropTableSpaceStmt AlterTableSpaceOptionsStmt /* GrantStmt and RevokeStmt if objtype is a global object determined by EventTriggerSupportsObjectType() */ GrantStmt RevokeStmt The idea with this patch is to support global objects commands replication by WAL logging the command using the same function for DDL logging - LogLogicalDDLMessage towards the end of standard_ProcessUtility. Because global objects are not schema qualified, we can skip the deparser invocation and directly log the original command string for replay on the subscriber. A key problem to address is that global objects can become inconsistent between the publisher and the subscriber if a command modifying the global object gets executed in a database (on the source side) that doesn't replicate the global object commands. I think we can work on the following two aspects in order to avoid such inconsistency: 1. Introduce a publication option for global object commands replication and document that logical replication of global object commands is preferred to be enabled on all databases. Otherwise inconsistency can happen if a command modifies the global object in a database that doesn't replicate global object commands. For example, we could introduce the following publication option publish_global_object_command : CREATE PUBLICATION mypub FOR ALL TABLES WITH (publish = 'insert, delete, update', publish_global_object_command = true); We may consider other fine tuned global command options such as “publish_role_statements”, “publish_database_statements”, “publish_tablespace_statements” and "publish_grant_statements", i.e. you pick which global commands you want replicated. For example, you can do this if you need a permission or tablespace to be set up differently on the target cluster. In addition, we may need to adjust the syntax once the DDL replication syntax finalizes. 2. Introduce the following database cluster level logical replication commands to avoid such inconsistency, this is especially handy when there is a large number of databases to configure for logical replication. CREATE PUBLICATION GROUP mypub_ FOR ALL DATABASES WITH (publish = 'insert, delete, update', publish_global_object_command = true); CREATE SUBSCRIPTION GROUP mysub_ CONNECTION 'dbnames = \“path to file\” host=hostname user=username port=5432' PUBLICATION GROUP mypub_; Under the hood, the CREATE PUBLICATION GROUP command generates one CREATE PUBLICATION mypub_n sub-command for each database in the cluster where n is a monotonically increasing integer from 1. The command outputs the (dbname, publication name) pairs which can be saved in a file and then used on the subscription side. Similarly, the CREATE SUBSCRIPTION GROUP command will generate one CREATE SUBSCRIPTION mysub_n sub-command for each database in the dbnames file. The dbnames file contains the (dbname, publication name) pairs which come from the output of the CREATE PUBLICATION GROUP command. Notice the connection string doesn’t have the dbname field, During execution the connection string will be appended the dbname retrieved from the dbnames file. By default the target DB name is the same as the source DB name, optionally user can specify the source_db to target_db mapping in the dbnames file. In addition, we might want to create dependencies for the publications/subscriptions created by the above commands in order to guarantee the group consistency. Also we need to enforce that there is only one group of publications/subscriptions for database cluster level replication. Logical replication of all commands across an entire cluster (instead of on a per-database basis) is a separate topic. We can start another thread after implementing a prototype. Please let me know your thoughts. [1] https://www.postgresql.org/message-id/flat/CAAD30U%2BpVmfKwUKy8cbZOnUXyguJ-uBNejwD75Kyo%3DOjdQGJ9g%40mail.gmail.com [2]https://www.postgresql.org/message-id/OS0PR01MB5716009FDCCC0B50BCB14A99949D9%40OS0PR01MB5716.jpnprd01.prod.outlook.com With Regards, Zheng Li Amazon RDS/Aurora for PostgreSQL
Attachment
On Tue, Aug 9, 2022 at 1:31 AM Zheng Li <zhengli10@gmail.com> wrote: > > Hello, > > Logical replication of DDL commands support is being worked on in [1]. > However, global object commands are quite different from other > non-global object DDL commands and need to be handled differently. For > example, global object commands include ROLE statements, DATABASE > statements, TABLESPACE statements and a subset of GRANT/REVOKE > statements if the object being modified is a global object. These > commands are different from other DDL commands in that: > > 1. Global object commands can be executed in any database. > 2. Global objects are not schema qualified. > 3. Global object commands are not captured by event triggers. > > I’ve put together a prototype to support logical replication of global > object commands in the attached patch. This patch builds on the DDL > replication patch from ZJ in [2] and must be applied on top of it. > Here is a list of global object commands that the patch replicate, you > can find more details in function LogGlobalObjectCommand: > > /* ROLE statements */ > CreateRoleStmt > AlterRoleStmt > AlterRoleSetStmt > DropRoleStmt > ReassignOwnedStmt > GrantRoleStmt > > /* Database statements */ > CreatedbStmt > AlterDatabaseStmt > AlterDatabaseRefreshCollStmt > AlterDatabaseSetStmt > DropdbStmt > > /* TableSpace statements */ > CreateTableSpaceStmt > DropTableSpaceStmt > AlterTableSpaceOptionsStmt > > /* GrantStmt and RevokeStmt if objtype is a global object determined > by EventTriggerSupportsObjectType() */ > GrantStmt > RevokeStmt > > The idea with this patch is to support global objects commands > replication by WAL logging the command using the same function for DDL > logging - LogLogicalDDLMessage towards the end of > standard_ProcessUtility. Because global objects are not schema > qualified, we can skip the deparser invocation and directly log the > original command string for replay on the subscriber. > > A key problem to address is that global objects can become > inconsistent between the publisher and the subscriber if a command > modifying the global object gets executed in a database (on the source > side) that doesn't replicate the global object commands. I think we > can work on the following two aspects in order to avoid such > inconsistency: > > 1. Introduce a publication option for global object commands > replication and document that logical replication of global object > commands is preferred to be enabled on all databases. Otherwise > inconsistency can happen if a command modifies the global object in a > database that doesn't replicate global object commands. > > For example, we could introduce the following publication option > publish_global_object_command : > CREATE PUBLICATION mypub > FOR ALL TABLES > WITH (publish = 'insert, delete, update', publish_global_object_command = true); > Tying global objects with FOR ALL TABLES seems odd to me. One possible idea could be to introduce publications FOR ALL OBJECTS. However, I am not completely sure whether tying global objects with database-specific publications is what users would expect but OTOH I don't have any better ideas here. -- With Regards, Amit Kapila.
On Fri, Aug 12, 2022 at 5:41 PM Amit Kapila <amit.kapila16@gmail.com> wrote: > > On Tue, Aug 9, 2022 at 1:31 AM Zheng Li <zhengli10@gmail.com> wrote: > > > > Hello, > > > > Logical replication of DDL commands support is being worked on in [1]. > > However, global object commands are quite different from other > > non-global object DDL commands and need to be handled differently. For > > example, global object commands include ROLE statements, DATABASE > > statements, TABLESPACE statements and a subset of GRANT/REVOKE > > statements if the object being modified is a global object. These > > commands are different from other DDL commands in that: > > > > 1. Global object commands can be executed in any database. > > 2. Global objects are not schema qualified. > > 3. Global object commands are not captured by event triggers. > > > > I’ve put together a prototype to support logical replication of global > > object commands in the attached patch. This patch builds on the DDL > > replication patch from ZJ in [2] and must be applied on top of it. > > Here is a list of global object commands that the patch replicate, you > > can find more details in function LogGlobalObjectCommand: > > > > /* ROLE statements */ > > CreateRoleStmt > > AlterRoleStmt > > AlterRoleSetStmt > > DropRoleStmt > > ReassignOwnedStmt > > GrantRoleStmt > > > > /* Database statements */ > > CreatedbStmt > > AlterDatabaseStmt > > AlterDatabaseRefreshCollStmt > > AlterDatabaseSetStmt > > DropdbStmt > > > > /* TableSpace statements */ > > CreateTableSpaceStmt > > DropTableSpaceStmt > > AlterTableSpaceOptionsStmt > > > > /* GrantStmt and RevokeStmt if objtype is a global object determined > > by EventTriggerSupportsObjectType() */ > > GrantStmt > > RevokeStmt > > > > The idea with this patch is to support global objects commands > > replication by WAL logging the command using the same function for DDL > > logging - LogLogicalDDLMessage towards the end of > > standard_ProcessUtility. Because global objects are not schema > > qualified, we can skip the deparser invocation and directly log the > > original command string for replay on the subscriber. > > > > A key problem to address is that global objects can become > > inconsistent between the publisher and the subscriber if a command > > modifying the global object gets executed in a database (on the source > > side) that doesn't replicate the global object commands. I think we > > can work on the following two aspects in order to avoid such > > inconsistency: > > > > 1. Introduce a publication option for global object commands > > replication and document that logical replication of global object > > commands is preferred to be enabled on all databases. Otherwise > > inconsistency can happen if a command modifies the global object in a > > database that doesn't replicate global object commands. > > > > For example, we could introduce the following publication option > > publish_global_object_command : > > CREATE PUBLICATION mypub > > FOR ALL TABLES > > WITH (publish = 'insert, delete, update', publish_global_object_command = true); > > > > Tying global objects with FOR ALL TABLES seems odd to me. One possible > idea could be to introduce publications FOR ALL OBJECTS. However, I am > not completely sure whether tying global objects with > database-specific publications is what users would expect but OTOH I > don't have any better ideas here. > Can we think of relying to send WAL of such DDLs just based on whether there is a corresponding publication (ex. publication of ALL OBJECTS)? I mean avoid database-specific filtering in decoding for such DDL commands but not sure how much better it is than the current proposal? The other idea that occurred to me is to have separate event triggers for global objects that we can store in the shared catalog but again it is not clear how to specify the corresponding function as functions are database specific. Thoughts? -- With Regards, Amit Kapila.
> Can we think of relying to send WAL of such DDLs just based on whether > there is a corresponding publication (ex. publication of ALL OBJECTS)? > I mean avoid database-specific filtering in decoding for such DDL > commands but not sure how much better it is than the current proposal? I think a publication of ALL OBJECTS sounds intuitive. Does it mean we'll publish all DDL commands, all commit and abort operations in every database if there is such publication of ALL OBJECTS? Best, Zheng
On Tue, Aug 16, 2022 at 11:35 AM Zheng Li <zhengli10@gmail.com> wrote: > > > Can we think of relying to send WAL of such DDLs just based on whether > > there is a corresponding publication (ex. publication of ALL OBJECTS)? > > I mean avoid database-specific filtering in decoding for such DDL > > commands but not sure how much better it is than the current proposal? > > I think a publication of ALL OBJECTS sounds intuitive. Does it mean we'll > publish all DDL commands, all commit and abort operations in every > database if there is such publication of ALL OBJECTS? > Actually, I intend something for global objects. But the main thing that is worrying me about this is that we don't have a clean way to untie global object replication from database-specific object replication. -- With Regards, Amit Kapila.
> > I think a publication of ALL OBJECTS sounds intuitive. Does it mean we'll > > publish all DDL commands, all commit and abort operations in every > > database if there is such publication of ALL OBJECTS? > > > > Actually, I intend something for global objects. But the main thing > that is worrying me about this is that we don't have a clean way to > untie global object replication from database-specific object > replication. I think ultimately we need a clean and efficient way to publish (and subscribe to) any changes in all databases, preferably in one logical replication slot. -- Regards, Zheng
On Tue, Aug 30, 2022 at 8:09 AM Zheng Li <zhengli10@gmail.com> wrote: > > > > I think a publication of ALL OBJECTS sounds intuitive. Does it mean we'll > > > publish all DDL commands, all commit and abort operations in every > > > database if there is such publication of ALL OBJECTS? > > > > > > > Actually, I intend something for global objects. But the main thing > > that is worrying me about this is that we don't have a clean way to > > untie global object replication from database-specific object > > replication. > > I think ultimately we need a clean and efficient way to publish (and > subscribe to) any changes in all databases, preferably in one logical > replication slot. > Agreed. I was thinking currently for logical replication both walsender and slot are database-specific. So we need a way to distinguish the WAL for global objects and then avoid filtering based on the slot's database during decoding. I also thought about whether we want to have a WALSender that is not connected to a database for the replication of global objects but I couldn't come up with a reason for doing so. Do you have any thoughts on this matter? -- With Regards, Amit Kapila.
On Thu, Feb 16, 2023 at 12:02 PM Amit Kapila <amit.kapila16@gmail.com> wrote: > > On Tue, Aug 30, 2022 at 8:09 AM Zheng Li <zhengli10@gmail.com> wrote: > > > > > > I think a publication of ALL OBJECTS sounds intuitive. Does it mean we'll > > > > publish all DDL commands, all commit and abort operations in every > > > > database if there is such publication of ALL OBJECTS? > > > > > > > > > > Actually, I intend something for global objects. But the main thing > > > that is worrying me about this is that we don't have a clean way to > > > untie global object replication from database-specific object > > > replication. > > > > I think ultimately we need a clean and efficient way to publish (and > > subscribe to) any changes in all databases, preferably in one logical > > replication slot. > > > > Agreed. I was thinking currently for logical replication both > walsender and slot are database-specific. So we need a way to > distinguish the WAL for global objects and then avoid filtering based > on the slot's database during decoding. I also thought about whether > we want to have a WALSender that is not connected to a database for > the replication of global objects but I couldn't come up with a reason > for doing so. Do you have any thoughts on this matter? > Another thing about the patch proposed here is that it LOGs the DDL for global objects without any consideration of whether that is required for logical replication. This is quite unlike what we are planning to do for other DDLs where it will be logged only when the publication has defined an event trigger for it. -- With Regards, Amit Kapila.
> > > Actually, I intend something for global objects. But the main thing > > > that is worrying me about this is that we don't have a clean way to > > > untie global object replication from database-specific object > > > replication. > > > > I think ultimately we need a clean and efficient way to publish (and > > subscribe to) any changes in all databases, preferably in one logical > > replication slot. > > > > Agreed. I was thinking currently for logical replication both > walsender and slot are database-specific. So we need a way to > distinguish the WAL for global objects and then avoid filtering based > on the slot's database during decoding. But which WALSender should handle the WAL for global objects if we don't filter by database? Is there any specific problem you see for decoding global objects commands in a database specific WALSender? > I also thought about whether > we want to have a WALSender that is not connected to a database for > the replication of global objects but I couldn't come up with a reason > for doing so. Do you have any thoughts on this matter? Regards, Zane
On Fri, Feb 17, 2023 at 10:58 AM Zheng Li <zhengli10@gmail.com> wrote: > > > > > Actually, I intend something for global objects. But the main thing > > > > that is worrying me about this is that we don't have a clean way to > > > > untie global object replication from database-specific object > > > > replication. > > > > > > I think ultimately we need a clean and efficient way to publish (and > > > subscribe to) any changes in all databases, preferably in one logical > > > replication slot. > > > > > > > Agreed. I was thinking currently for logical replication both > > walsender and slot are database-specific. So we need a way to > > distinguish the WAL for global objects and then avoid filtering based > > on the slot's database during decoding. > > But which WALSender should handle the WAL for global objects if we > don't filter by database? Is there any specific problem you see for > decoding global objects commands in a database specific WALSender? > I haven't verified but I was concerned about the below check: logicalddl_decode { ... + + if (message->dbId != ctx->slot->data.database || -- With Regards, Amit Kapila.
On Fri, Feb 17, 2023 at 4:48 AM Amit Kapila <amit.kapila16@gmail.com> wrote: > > On Fri, Feb 17, 2023 at 10:58 AM Zheng Li <zhengli10@gmail.com> wrote: > > > > > > > Actually, I intend something for global objects. But the main thing > > > > > that is worrying me about this is that we don't have a clean way to > > > > > untie global object replication from database-specific object > > > > > replication. > > > > > > > > I think ultimately we need a clean and efficient way to publish (and > > > > subscribe to) any changes in all databases, preferably in one logical > > > > replication slot. > > > > > > > > > > Agreed. I was thinking currently for logical replication both > > > walsender and slot are database-specific. So we need a way to > > > distinguish the WAL for global objects and then avoid filtering based > > > on the slot's database during decoding. > > > > But which WALSender should handle the WAL for global objects if we > > don't filter by database? Is there any specific problem you see for > > decoding global objects commands in a database specific WALSender? > > > > I haven't verified but I was concerned about the below check: > logicalddl_decode > { > ... > + > + if (message->dbId != ctx->slot->data.database || OK, let's suppose we don't filter by database for global commands when decoding ddl records, roughly what the following code does: logicalddl_decode { ... if (message->dbId != ctx->slot->data.database || + message->cmdtype != DCT_GlobalObjectCmd But this is not enough, we also need the subsequent commit record of the txn to be decoded in order to replicate the global command. So I think we also need to make DecodeCommit bypass the filter by database if global object replication is turned on and we have decoded a global command in the txn. Regards, Zane
Hi, On Tue, Aug 9, 2022 at 5:01 AM Zheng Li <zhengli10@gmail.com> wrote: > > Hello, > > Logical replication of DDL commands support is being worked on in [1]. > However, global object commands are quite different from other > non-global object DDL commands and need to be handled differently. For > example, global object commands include ROLE statements, DATABASE > statements, TABLESPACE statements and a subset of GRANT/REVOKE > statements if the object being modified is a global object. These > commands are different from other DDL commands in that: > > 1. Global object commands can be executed in any database. > 2. Global objects are not schema qualified. > 3. Global object commands are not captured by event triggers. > > I’ve put together a prototype to support logical replication of global > object commands in the attached patch. This patch builds on the DDL > replication patch from ZJ in [2] and must be applied on top of it. > Here is a list of global object commands that the patch replicate, you > can find more details in function LogGlobalObjectCommand: > > /* ROLE statements */ > CreateRoleStmt > AlterRoleStmt > AlterRoleSetStmt > DropRoleStmt > ReassignOwnedStmt > GrantRoleStmt > > /* Database statements */ > CreatedbStmt > AlterDatabaseStmt > AlterDatabaseRefreshCollStmt > AlterDatabaseSetStmt > DropdbStmt > > /* TableSpace statements */ > CreateTableSpaceStmt > DropTableSpaceStmt > AlterTableSpaceOptionsStmt > > /* GrantStmt and RevokeStmt if objtype is a global object determined > by EventTriggerSupportsObjectType() */ > GrantStmt > RevokeStmt > > The idea with this patch is to support global objects commands > replication by WAL logging the command using the same function for DDL > logging - LogLogicalDDLMessage towards the end of > standard_ProcessUtility. Because global objects are not schema > qualified, we can skip the deparser invocation and directly log the > original command string for replay on the subscriber. > > A key problem to address is that global objects can become > inconsistent between the publisher and the subscriber if a command > modifying the global object gets executed in a database (on the source > side) that doesn't replicate the global object commands. I think we > can work on the following two aspects in order to avoid such > inconsistency: > > 1. Introduce a publication option for global object commands > replication and document that logical replication of global object > commands is preferred to be enabled on all databases. Otherwise > inconsistency can happen if a command modifies the global object in a > database that doesn't replicate global object commands. > > For example, we could introduce the following publication option > publish_global_object_command : > CREATE PUBLICATION mypub > FOR ALL TABLES > WITH (publish = 'insert, delete, update', publish_global_object_command = true); > > We may consider other fine tuned global command options such as > “publish_role_statements”, “publish_database_statements”, > “publish_tablespace_statements” and "publish_grant_statements", i.e. > you pick which global commands you want replicated. For example, you > can do this if you need a permission or tablespace to be set up > differently on the target cluster. In addition, we may need to adjust > the syntax once the DDL replication syntax finalizes. > > 2. Introduce the following database cluster level logical replication > commands to avoid such inconsistency, this is especially handy when > there is a large number of databases to configure for logical > replication. > > CREATE PUBLICATION GROUP mypub_ > FOR ALL DATABASES > WITH (publish = 'insert, delete, update', publish_global_object_command = true); > > CREATE SUBSCRIPTION GROUP mysub_ > CONNECTION 'dbnames = \“path to file\” host=hostname user=username port=5432' > PUBLICATION GROUP mypub_; > > Under the hood, the CREATE PUBLICATION GROUP command generates one > CREATE PUBLICATION mypub_n sub-command for each database in the > cluster where n is a monotonically increasing integer from 1. The > command outputs the (dbname, publication name) pairs which can be > saved in a file and then used on the subscription side. > > Similarly, the CREATE SUBSCRIPTION GROUP command will generate one > CREATE SUBSCRIPTION mysub_n sub-command for each database in the > dbnames file. The dbnames file contains the (dbname, publication name) > pairs which come from the output of the CREATE PUBLICATION GROUP > command. Notice the connection string doesn’t have the dbname field, > During execution the connection string will be appended the dbname > retrieved from the dbnames file. By default the target DB name is the > same as the source DB name, optionally user can specify the source_db > to target_db mapping in the dbnames file. > > In addition, we might want to create dependencies for the > publications/subscriptions created by the above commands in order to > guarantee the group consistency. Also we need to enforce that there is > only one group of publications/subscriptions for database cluster > level replication. > > Logical replication of all commands across an entire cluster (instead > of on a per-database basis) is a separate topic. We can start another > thread after implementing a prototype. > > Please let me know your thoughts. Thank you for working on this item. I think that there are some (possibly) tricky challenges that haven't been discussed yet to support replicating global objects. First, as for publications having global objects (roles, databases, and tablespaces), but storing them in database specific tables like pg_publication doesn't make sense, because it should be at some shared place where all databases can have access to it. Maybe we need to have a shared catalog like pg_shpublication or pg_publication_role to store publications related to global objects or the relationship between such publications and global objects. Second, we might need to change the logical decoding infrastructure so that it's aware of shared catalog changes. Currently we need to scan only db-specific catalogs. Finally, since we process CREATE DATABASE in a different way than other DDLs (by cloning another database such as template1), simply replicating the CREATE DATABASE statement would not produce the same results as the publisher. Also, since event triggers are not fired on DDLs for global objects, always WAL-logging such DDL statements like the proposed patch does is not a good idea. Given that there seems to be some tricky problems and there is a discussion for cutting the scope to make the initial patch small[1], I think it's better to do this work after the first version. Regards, [1] https://www.postgresql.org/message-id/CAA4eK1K3VXfTWXbLADcH81J%3D%3D7ussvNdqLFHN68sEokDPueu7w%40mail.gmail.com -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com
> I think that there are some (possibly) tricky challenges that haven't > been discussed yet to support replicating global objects. > > First, as for publications having global objects (roles, databases, > and tablespaces), but storing them in database specific tables like > pg_publication doesn't make sense, because it should be at some shared > place where all databases can have access to it. Maybe we need to have > a shared catalog like pg_shpublication or pg_publication_role to store > publications related to global objects or the relationship between > such publications and global objects. Second, we might need to change > the logical decoding infrastructure so that it's aware of shared > catalog changes. Thanks for the feedback. This is insightful. > Currently we need to scan only db-specific catalogs. > Finally, since we process CREATE DATABASE in a different way than > other DDLs (by cloning another database such as template1), simply > replicating the CREATE DATABASE statement would not produce the same > results as the publisher. Also, since event triggers are not fired on > DDLs for global objects, always WAL-logging such DDL statements like > the proposed patch does is not a good idea. > Given that there seems to be some tricky problems and there is a > discussion for cutting the scope to make the initial patch small[1], I > think it's better to do this work after the first version. Agreed. Regards, Zane