Thread: Logical Replication of sequences

In the past, we have discussed various approaches to replicate
sequences by decoding the sequence changes from WAL. However, we faced
several challenges to achieve the same, some of which are due to the
non-transactional nature of sequences. The major ones were: (a)
correctness of the decoding part, some of the problems were discussed
at [1][2][3] (b) handling of sequences especially adding certain
sequences automatically (e.g. sequences backing SERIAL/BIGSERIAL
columns) for built-in logical replication is not considered in the
proposed work [1] (c) there were some performance concerns in not so
frequent scenarios [4] (see performance issues), we can probably deal
with this by making sequences optional for builtin logical replication

It could be possible that we can deal with these and any other issues
with more work but as the use case for this feature is primarily major
version upgrades it is not clear that we want to make such a big
change to the code or are there better alternatives to achieve the
same.

This time at pgconf.dev (https://2024.pgconf.dev/), we discussed
alternative approaches for this work which I would like to summarize.
The various methods we discussed are as follows:

1. Provide a tool to copy all the sequences from publisher to
subscriber. The major drawback is that users need to perform this as
an additional step during the upgrade which would be inconvenient and
probably not as useful as some built-in mechanism.
2. Provide a command say Alter Subscription ...  Replicate Sequences
(or something like that) which users can perform before shutdown of
the publisher node during upgrade. This will allow copying all the
sequences from the publisher node to the subscriber node directly.
Similar to previous approach, this could also be inconvenient for
users.
3. Replicate published sequences via walsender at the time of shutdown
or incrementally while decoding checkpoint record. The two ways to
achieve this are: (a) WAL log a special NOOP record just before
shutting down checkpointer. Then allow the WALsender to read the
sequence data and send it to the subscriber while decoding the new
NOOP record. (b) Similar to the previous idea but instead of WAL
logging a new record directly invokes a decoding callback after
walsender receives a request to shutdown which will allow pgoutput to
read and send required sequences. This approach has a drawback that we
are adding more work at the time of shutdown but note that we already
waits for all the WAL records to be decoded and sent before shutting
down the walsender during shutdown of the node.

Any other ideas?

I have added the members I remember that were part of the discussion
in the email. Please feel free to correct me if I have misunderstood
or missed any point we talked about.

Thoughts?

[1] - https://www.postgresql.org/message-id/e4145f77-6f37-40e0-a770-aba359c50b93%40enterprisedb.com
[2] - https://www.postgresql.org/message-id/CAA4eK1Lxt%2B5a9fA-B7FRzfd1vns%3DEwZTF5z9_xO9Ms4wsqD88Q%40mail.gmail.com
[3] - https://www.postgresql.org/message-id/CAA4eK1KR4%3DyALKP0pOdVkqUwoUqD_v7oU3HzY-w0R_EBvgHL2w%40mail.gmail.com
[4] - https://www.postgresql.org/message-id/12822961-b7de-9d59-dd27-2e3dc3980c7e%40enterprisedb.com

-- 
With Regards,
Amit Kapila.

Hi,

On Tue, Jun 4, 2024 at 4:27 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> 3. Replicate published sequences via walsender at the time of shutdown
> or incrementally while decoding checkpoint record. The two ways to
> achieve this are: (a) WAL log a special NOOP record just before
> shutting down checkpointer. Then allow the WALsender to read the
> sequence data and send it to the subscriber while decoding the new
> NOOP record. (b) Similar to the previous idea but instead of WAL
> logging a new record directly invokes a decoding callback after
> walsender receives a request to shutdown which will allow pgoutput to
> read and send required sequences. This approach has a drawback that we
> are adding more work at the time of shutdown but note that we already
> waits for all the WAL records to be decoded and sent before shutting
> down the walsender during shutdown of the node.

Thanks. IIUC, both of the above approaches decode the sequences during
only shutdown. I'm wondering, why not periodically decode and
replicate the published sequences so that the decoding at the shutdown
will not take that longer? I can imagine a case where there are tens
of thousands of sequences in a production server, and surely decoding
and sending them just during the shutdown can take a lot of time
hampering the overall server uptime.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Tue, Jun 4, 2024 at 4:53 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Tue, Jun 4, 2024 at 4:27 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > 3. Replicate published sequences via walsender at the time of shutdown
> > or incrementally while decoding checkpoint record. The two ways to
> > achieve this are: (a) WAL log a special NOOP record just before
> > shutting down checkpointer. Then allow the WALsender to read the
> > sequence data and send it to the subscriber while decoding the new
> > NOOP record. (b) Similar to the previous idea but instead of WAL
> > logging a new record directly invokes a decoding callback after
> > walsender receives a request to shutdown which will allow pgoutput to
> > read and send required sequences. This approach has a drawback that we
> > are adding more work at the time of shutdown but note that we already
> > waits for all the WAL records to be decoded and sent before shutting
> > down the walsender during shutdown of the node.
>
> Thanks. IIUC, both of the above approaches decode the sequences during
> only shutdown. I'm wondering, why not periodically decode and
> replicate the published sequences so that the decoding at the shutdown
> will not take that longer?
>

Even if we decode it periodically (say each time we decode the
checkpoint record) then also we need to send the entire set of
sequences at shutdown. This is because the sequences may have changed
from the last time we sent them.

>
> I can imagine a case where there are tens
> of thousands of sequences in a production server, and surely decoding
> and sending them just during the shutdown can take a lot of time
> hampering the overall server uptime.
>

It is possible but we will send only the sequences that belong to
publications for which walsender is supposed to send the required
data. Now, we can also imagine providing option 2 (Alter Subscription
... Replicate Sequences) so that users can replicate sequences before
shutdown and then disable the subscriptions so that there won't be a
corresponding walsender.

--
With Regards,
Amit Kapila.

On 6/4/24 06:57, Amit Kapila wrote:
> 1. Provide a tool to copy all the sequences from publisher to
> subscriber. The major drawback is that users need to perform this as
> an additional step during the upgrade which would be inconvenient and
> probably not as useful as some built-in mechanism.

Agree, this requires additional steps. Not a preferred approach in my
opinion. When a large set of sequences are present, it will add
additional downtime for upgrade process.

> 2. Provide a command say Alter Subscription ...  Replicate Sequences
> (or something like that) which users can perform before shutdown of
> the publisher node during upgrade. This will allow copying all the
> sequences from the publisher node to the subscriber node directly.
> Similar to previous approach, this could also be inconvenient for
> users.

This is similar to option 1 except that it is a SQL command now. Still
not a preferred approach in my opinion. When a large set of sequences are
present, it will add additional downtime for upgrade process.


> 3. Replicate published sequences via walsender at the time of shutdown
> or incrementally while decoding checkpoint record. The two ways to
> achieve this are: (a) WAL log a special NOOP record just before
> shutting down checkpointer. Then allow the WALsender to read the
> sequence data and send it to the subscriber while decoding the new
> NOOP record. (b) Similar to the previous idea but instead of WAL
> logging a new record directly invokes a decoding callback after
> walsender receives a request to shutdown which will allow pgoutput to
> read and send required sequences. This approach has a drawback that we
> are adding more work at the time of shutdown but note that we already
> waits for all the WAL records to be decoded and sent before shutting
> down the walsender during shutdown of the node.

At the time of shutdown a) most logical upgrades don't necessarily call
for shutdown b) it will still add to total downtime with large set of
sequences. Incremental option is better as it will not require a shutdown.

I do see a scenario where sequence of events can lead to loss of sequence
and generate duplicate sequence values, if subscriber starts consuming
sequences while publisher is also consuming them. In such cases, subscriber
shall not be allowed sequence consumption.



-- 
Kind Regards,
Yogesh Sharma
Open Source Enthusiast and Advocate
PostgreSQL Contributors Team @ RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Tue, Jun 4, 2024 at 7:40 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Tue, Jun 4, 2024 at 4:27 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>>
>> 3. Replicate published sequences via walsender at the time of shutdown
>> or incrementally while decoding checkpoint record. The two ways to
>> achieve this are: (a) WAL log a special NOOP record just before
>> shutting down checkpointer. Then allow the WALsender to read the
>> sequence data and send it to the subscriber while decoding the new
>> NOOP record. (b) Similar to the previous idea but instead of WAL
>> logging a new record directly invokes a decoding callback after
>> walsender receives a request to shutdown which will allow pgoutput to
>> read and send required sequences. This approach has a drawback that we
>> are adding more work at the time of shutdown but note that we already
>> waits for all the WAL records to be decoded and sent before shutting
>> down the walsender during shutdown of the node.
>>
>> Any other ideas?
>>
>
> In case of primary crash the sequence won't get replicated. That is true even with the previous approach in case
walsenderis shut down because of a crash, but it is more serious with this approach. 
>

Right, but if we just want to support a major version upgrade scenario
then this should be fine because upgrades require a clean shutdown.

>
 How about periodically sending this information?
>

Now, if we want to support some sort of failover then probably this
will help. Do you have that use case in mind? If we want to send
periodically then we can do it when decoding checkpoint
(XLOG_CHECKPOINT_ONLINE) or some other periodic WAL record like
running_xacts (XLOG_RUNNING_XACTS).

--
With Regards,
Amit Kapila.

On Tue, Jun 4, 2024 at 8:56 PM Yogesh Sharma
<yogesh.sharma@catprosystems.com> wrote:
>
> On 6/4/24 06:57, Amit Kapila wrote:
>
> > 2. Provide a command say Alter Subscription ...  Replicate Sequences
> > (or something like that) which users can perform before shutdown of
> > the publisher node during upgrade. This will allow copying all the
> > sequences from the publisher node to the subscriber node directly.
> > Similar to previous approach, this could also be inconvenient for
> > users.
>
> This is similar to option 1 except that it is a SQL command now.
>

Right, but I would still prefer a command as it provides clear steps
for the upgrade. Users need to perform (a) Replicate Sequences for a
particular subscription (b) Disable that subscription (c) Perform (a)
and (b) for all the subscriptions corresponding to the publisher we
want to shut down for upgrade.

I agree there are some manual steps involved here but it is advisable
for users to ensure that they have received the required data on the
subscriber before the upgrade of the publisher node, otherwise, they
may not be able to continue replication after the upgrade. For
example, see the "Prepare for publisher upgrades" step in pg_upgrade
docs [1].

>
> > 3. Replicate published sequences via walsender at the time of shutdown
> > or incrementally while decoding checkpoint record. The two ways to
> > achieve this are: (a) WAL log a special NOOP record just before
> > shutting down checkpointer. Then allow the WALsender to read the
> > sequence data and send it to the subscriber while decoding the new
> > NOOP record. (b) Similar to the previous idea but instead of WAL
> > logging a new record directly invokes a decoding callback after
> > walsender receives a request to shutdown which will allow pgoutput to
> > read and send required sequences. This approach has a drawback that we
> > are adding more work at the time of shutdown but note that we already
> > waits for all the WAL records to be decoded and sent before shutting
> > down the walsender during shutdown of the node.
>
> At the time of shutdown a) most logical upgrades don't necessarily call
> for shutdown
>

Won't the major version upgrade expect that the node is down? Refer to
step "Stop both servers" in [1].

>
 b) it will still add to total downtime with large set of
> sequences. Incremental option is better as it will not require a shutdown.
>
> I do see a scenario where sequence of events can lead to loss of sequence
> and generate duplicate sequence values, if subscriber starts consuming
> sequences while publisher is also consuming them. In such cases, subscriber
> shall not be allowed sequence consumption.
>

It would be fine to not allow subscribers to consume sequences that
are being logically replicated but what about the cases where we
haven't sent the latest values of sequences before the shutdown of the
publisher? In such a case, the publisher would have already consumed
some values that wouldn't have been sent to the subscriber and now
when the publisher is down then even if we re-allow the sequence
values to be consumed from the subscriber, it can lead to duplicate
values.

[1] - https://www.postgresql.org/docs/devel/pgupgrade.html

--
With Regards,
Amit Kapila.

On 04.06.24 12:57, Amit Kapila wrote:
> 2. Provide a command say Alter Subscription ...  Replicate Sequences
> (or something like that) which users can perform before shutdown of
> the publisher node during upgrade. This will allow copying all the
> sequences from the publisher node to the subscriber node directly.
> Similar to previous approach, this could also be inconvenient for
> users.

I would start with this.  In any case, you're going to need to write 
code to collect all the sequence values, send them over some protocol, 
apply them on the subscriber.  The easiest way to start is to trigger 
that manually.  Then later you can add other ways to trigger it, either 
by timer or around shutdown, or whatever other ideas there might be.

On Wed, Jun 5, 2024 at 9:13 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jun 4, 2024 at 8:56 PM Yogesh Sharma
> <yogesh.sharma@catprosystems.com> wrote:
> >
> > On 6/4/24 06:57, Amit Kapila wrote:
> >
> > > 2. Provide a command say Alter Subscription ...  Replicate Sequences
> > > (or something like that) which users can perform before shutdown of
> > > the publisher node during upgrade. This will allow copying all the
> > > sequences from the publisher node to the subscriber node directly.
> > > Similar to previous approach, this could also be inconvenient for
> > > users.
> >
> > This is similar to option 1 except that it is a SQL command now.
> >
>
> Right, but I would still prefer a command as it provides clear steps
> for the upgrade. Users need to perform (a) Replicate Sequences for a
> particular subscription (b) Disable that subscription (c) Perform (a)
> and (b) for all the subscriptions corresponding to the publisher we
> want to shut down for upgrade.
>

Another advantage of this approach over just a plain tool to copy all
sequences before upgrade is that here we can have the facility to copy
just the required sequences. I mean the set sequences that the user
has specified as part of the publication.

--
With Regards,
Amit Kapila.

Hi,

On Tue, Jun 4, 2024 at 5:40 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> Even if we decode it periodically (say each time we decode the
> checkpoint record) then also we need to send the entire set of
> sequences at shutdown. This is because the sequences may have changed
> from the last time we sent them.

Agree. How about decoding and sending only the sequences that are
changed from the last time when they were sent? I know it requires a
bit of tracking and more work, but all I'm looking for is to reduce
the amount of work that walsenders need to do during the shutdown.

Having said that, I like the idea of letting the user sync the
sequences via ALTER SUBSCRIPTION command and not weave the logic into
the shutdown checkpoint path. As Peter Eisentraut said here
https://www.postgresql.org/message-id/42e5cb35-4aeb-4f58-8091-90619c7c3ecc%40eisentraut.org,
this can be a good starting point to get going.

> > I can imagine a case where there are tens
> > of thousands of sequences in a production server, and surely decoding
> > and sending them just during the shutdown can take a lot of time
> > hampering the overall server uptime.
>
> It is possible but we will send only the sequences that belong to
> publications for which walsender is supposed to send the required
> data.

Right, but what if all the publication tables can have tens of
thousands of sequences.

> Now, we can also imagine providing option 2 (Alter Subscription
> ... Replicate Sequences) so that users can replicate sequences before
> shutdown and then disable the subscriptions so that there won't be a
> corresponding walsender.

As stated above, I like this idea to start with.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Wed, Jun 5, 2024 at 12:51 PM Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 04.06.24 12:57, Amit Kapila wrote:
> > 2. Provide a command say Alter Subscription ...  Replicate Sequences
> > (or something like that) which users can perform before shutdown of
> > the publisher node during upgrade. This will allow copying all the
> > sequences from the publisher node to the subscriber node directly.
> > Similar to previous approach, this could also be inconvenient for
> > users.
>
> I would start with this.  In any case, you're going to need to write
> code to collect all the sequence values, send them over some protocol,
> apply them on the subscriber.  The easiest way to start is to trigger
> that manually.  Then later you can add other ways to trigger it, either
> by timer or around shutdown, or whatever other ideas there might be.
>

Agreed. To achieve this, we can allow sequences to be copied during
the initial CREATE SUBSCRIPTION command similar to what we do for
tables. And then later by new/existing command, we re-copy the already
existing sequences on the subscriber.

The options for the new command could be:
Alter Subscription ... Refresh Sequences
Alter Subscription ... Replicate Sequences

In the second option, we need to introduce a new keyword Replicate.
Can you think of any better option?

In addition to the above, the command Alter Subscription .. Refresh
Publication will fetch any missing sequences similar to what it does
for tables.

Thoughts?

--
With Regards,
Amit Kapila.

On Wed, Jun 5, 2024 at 6:01 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Wed, Jun 5, 2024 at 8:45 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>>  How about periodically sending this information?
>> >
>>
>> Now, if we want to support some sort of failover then probably this
>> will help. Do you have that use case in mind?
>
>
> Regular failover was a goal for supporting logical replication of sequences. That might be more common than major
upgradescenario. 
>

We can't support regular failovers to subscribers unless we can
replicate/copy slots because the existing nodes connected to the
current publisher/primary would expect that. It should be primarily
useful for major version upgrades at this stage.

>>
>> If we want to send
>> periodically then we can do it when decoding checkpoint
>> (XLOG_CHECKPOINT_ONLINE) or some other periodic WAL record like
>> running_xacts (XLOG_RUNNING_XACTS).
>>
>
> Yeah. I am thinking along those lines.
>
> It must be noted, however, that none of those optional make sure that the replicated sequence's states are consistent
withthe replicated object state which use those sequences. 
>

Right, I feel as others are advocating, it seems better to support it
manually via command and then later we can extend it to do at shutdown
or at some regular intervals. If we do that then we should be able to
support major version upgrades and planned switchover.


--
With Regards,
Amit Kapila.

On Wed, Jun 5, 2024 at 12:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jun 4, 2024 at 8:56 PM Yogesh Sharma
> <yogesh.sharma@catprosystems.com> wrote:
> >
> > On 6/4/24 06:57, Amit Kapila wrote:
> >
> > > 2. Provide a command say Alter Subscription ...  Replicate Sequences
> > > (or something like that) which users can perform before shutdown of
> > > the publisher node during upgrade. This will allow copying all the
> > > sequences from the publisher node to the subscriber node directly.
> > > Similar to previous approach, this could also be inconvenient for
> > > users.
> >
> > This is similar to option 1 except that it is a SQL command now.
> >
>
> Right, but I would still prefer a command as it provides clear steps
> for the upgrade. Users need to perform (a) Replicate Sequences for a
> particular subscription (b) Disable that subscription (c) Perform (a)
> and (b) for all the subscriptions corresponding to the publisher we
> want to shut down for upgrade.
>
> I agree there are some manual steps involved here but it is advisable
> for users to ensure that they have received the required data on the
> subscriber before the upgrade of the publisher node, otherwise, they
> may not be able to continue replication after the upgrade. For
> example, see the "Prepare for publisher upgrades" step in pg_upgrade
> docs [1].
>
> >
> > > 3. Replicate published sequences via walsender at the time of shutdown
> > > or incrementally while decoding checkpoint record. The two ways to
> > > achieve this are: (a) WAL log a special NOOP record just before
> > > shutting down checkpointer. Then allow the WALsender to read the
> > > sequence data and send it to the subscriber while decoding the new
> > > NOOP record. (b) Similar to the previous idea but instead of WAL
> > > logging a new record directly invokes a decoding callback after
> > > walsender receives a request to shutdown which will allow pgoutput to
> > > read and send required sequences. This approach has a drawback that we
> > > are adding more work at the time of shutdown but note that we already
> > > waits for all the WAL records to be decoded and sent before shutting
> > > down the walsender during shutdown of the node.
> >
> > At the time of shutdown a) most logical upgrades don't necessarily call
> > for shutdown
> >
>
> Won't the major version upgrade expect that the node is down? Refer to
> step "Stop both servers" in [1].

I think the idea is that the publisher is the old version and the
subscriber is the new version, and changes generated on the publisher
are replicated to the subscriber via logical replication. And at some
point, we change the application (or a router) settings so that no
more transactions come to the publisher, do the last upgrade
preparation work (e.g. copying the latest sequence values if
requried), and then change the application so that new transactions
come to the subscriber.

I remember the blog post about Knock doing a similar process to
upgrade the clusters with minimal downtime[1].

Regards,

[1] https://knock.app/blog/zero-downtime-postgres-upgrades


--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Wed, Jun 5, 2024 at 3:17 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Tue, Jun 4, 2024 at 5:40 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Even if we decode it periodically (say each time we decode the
> > checkpoint record) then also we need to send the entire set of
> > sequences at shutdown. This is because the sequences may have changed
> > from the last time we sent them.
>
> Agree. How about decoding and sending only the sequences that are
> changed from the last time when they were sent? I know it requires a
> bit of tracking and more work, but all I'm looking for is to reduce
> the amount of work that walsenders need to do during the shutdown.
>

I see your point but going towards tracking the changed sequences
sounds like moving towards what we do for incremental backups unless
we can invent some other smart way.

> Having said that, I like the idea of letting the user sync the
> sequences via ALTER SUBSCRIPTION command and not weave the logic into
> the shutdown checkpoint path. As Peter Eisentraut said here
> https://www.postgresql.org/message-id/42e5cb35-4aeb-4f58-8091-90619c7c3ecc%40eisentraut.org,
> this can be a good starting point to get going.
>

Agreed.

> > > I can imagine a case where there are tens
> > > of thousands of sequences in a production server, and surely decoding
> > > and sending them just during the shutdown can take a lot of time
> > > hampering the overall server uptime.
> >
> > It is possible but we will send only the sequences that belong to
> > publications for which walsender is supposed to send the required
> > data.
>
> Right, but what if all the publication tables can have tens of
> thousands of sequences.
>

In such cases we have no option but to send all the sequences.

> > Now, we can also imagine providing option 2 (Alter Subscription
> > ... Replicate Sequences) so that users can replicate sequences before
> > shutdown and then disable the subscriptions so that there won't be a
> > corresponding walsender.
>
> As stated above, I like this idea to start with.
>

+1.


--
With Regards,
Amit Kapila.

On Wed, Jun 5, 2024 at 9:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jun 5, 2024 at 12:51 PM Peter Eisentraut <peter@eisentraut.org> wrote:
> >
> > On 04.06.24 12:57, Amit Kapila wrote:
> > > 2. Provide a command say Alter Subscription ...  Replicate Sequences
> > > (or something like that) which users can perform before shutdown of
> > > the publisher node during upgrade. This will allow copying all the
> > > sequences from the publisher node to the subscriber node directly.
> > > Similar to previous approach, this could also be inconvenient for
> > > users.
> >
> > I would start with this.  In any case, you're going to need to write
> > code to collect all the sequence values, send them over some protocol,
> > apply them on the subscriber.  The easiest way to start is to trigger
> > that manually.  Then later you can add other ways to trigger it, either
> > by timer or around shutdown, or whatever other ideas there might be.
> >
>
> Agreed.

+1

> To achieve this, we can allow sequences to be copied during
> the initial CREATE SUBSCRIPTION command similar to what we do for
> tables. And then later by new/existing command, we re-copy the already
> existing sequences on the subscriber.
>
> The options for the new command could be:
> Alter Subscription ... Refresh Sequences
> Alter Subscription ... Replicate Sequences
>
> In the second option, we need to introduce a new keyword Replicate.
> Can you think of any better option?

Another idea is doing that using options. For example,

For initial sequences synchronization:

CREATE SUBSCRIPTION ... WITH (copy_sequence = true);

For re-copy (or update) sequences:

ALTER SUBSCRIPTION ... REFRESH PUBLICATION WITH (copy_sequence = true);

>
> In addition to the above, the command Alter Subscription .. Refresh
> Publication will fetch any missing sequences similar to what it does
> for tables.

On the subscriber side, do we need to track which sequences are
created via CREATE/ALTER SUBSCRIPTION?

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Thu, Jun 6, 2024 at 9:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Jun 5, 2024 at 12:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Jun 4, 2024 at 8:56 PM Yogesh Sharma
> > <yogesh.sharma@catprosystems.com> wrote:
> > >
> > > On 6/4/24 06:57, Amit Kapila wrote:
> > >
> > > > 2. Provide a command say Alter Subscription ...  Replicate Sequences
> > > > (or something like that) which users can perform before shutdown of
> > > > the publisher node during upgrade. This will allow copying all the
> > > > sequences from the publisher node to the subscriber node directly.
> > > > Similar to previous approach, this could also be inconvenient for
> > > > users.
> > >
> > > This is similar to option 1 except that it is a SQL command now.
> > >
> >
> > Right, but I would still prefer a command as it provides clear steps
> > for the upgrade. Users need to perform (a) Replicate Sequences for a
> > particular subscription (b) Disable that subscription (c) Perform (a)
> > and (b) for all the subscriptions corresponding to the publisher we
> > want to shut down for upgrade.
> >
> > I agree there are some manual steps involved here but it is advisable
> > for users to ensure that they have received the required data on the
> > subscriber before the upgrade of the publisher node, otherwise, they
> > may not be able to continue replication after the upgrade. For
> > example, see the "Prepare for publisher upgrades" step in pg_upgrade
> > docs [1].
> >
> > >
> > > > 3. Replicate published sequences via walsender at the time of shutdown
> > > > or incrementally while decoding checkpoint record. The two ways to
> > > > achieve this are: (a) WAL log a special NOOP record just before
> > > > shutting down checkpointer. Then allow the WALsender to read the
> > > > sequence data and send it to the subscriber while decoding the new
> > > > NOOP record. (b) Similar to the previous idea but instead of WAL
> > > > logging a new record directly invokes a decoding callback after
> > > > walsender receives a request to shutdown which will allow pgoutput to
> > > > read and send required sequences. This approach has a drawback that we
> > > > are adding more work at the time of shutdown but note that we already
> > > > waits for all the WAL records to be decoded and sent before shutting
> > > > down the walsender during shutdown of the node.
> > >
> > > At the time of shutdown a) most logical upgrades don't necessarily call
> > > for shutdown
> > >
> >
> > Won't the major version upgrade expect that the node is down? Refer to
> > step "Stop both servers" in [1].
>
> I think the idea is that the publisher is the old version and the
> subscriber is the new version, and changes generated on the publisher
> are replicated to the subscriber via logical replication. And at some
> point, we change the application (or a router) settings so that no
> more transactions come to the publisher, do the last upgrade
> preparation work (e.g. copying the latest sequence values if
> requried), and then change the application so that new transactions
> come to the subscriber.
>

Okay, thanks for sharing the exact steps. If one has to follow that
path then sending incrementally (at checkpoint WAL or other times)
won't work because we want to ensure that the sequences are up-to-date
before the application starts using the new database. To do that in a
bullet-proof way, one has to copy/replicate sequences during the
requests to the new database are paused (Reference from the blog you
shared: For the first second after flipping the flag, our application
artificially paused any new database requests for one second.).
Currently, they are using some guesswork to replicate sequences that
require manual verification and more manual work for each sequence.
The new command (Alter Subscription ... Replicate Sequence) should
ease their procedure and can do things where they would require no or
very less verification.

> I remember the blog post about Knock doing a similar process to
> upgrade the clusters with minimal downtime[1].
>

Thanks for sharing the blog post.

--
With Regards,
Amit Kapila.

On Thu, Jun 6, 2024 at 11:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Jun 5, 2024 at 9:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
>
> > To achieve this, we can allow sequences to be copied during
> > the initial CREATE SUBSCRIPTION command similar to what we do for
> > tables. And then later by new/existing command, we re-copy the already
> > existing sequences on the subscriber.
> >
> > The options for the new command could be:
> > Alter Subscription ... Refresh Sequences
> > Alter Subscription ... Replicate Sequences
> >
> > In the second option, we need to introduce a new keyword Replicate.
> > Can you think of any better option?
>
> Another idea is doing that using options. For example,
>
> For initial sequences synchronization:
>
> CREATE SUBSCRIPTION ... WITH (copy_sequence = true);
>

How will it interact with the existing copy_data option? So copy_data
will become equivalent to copy_table_data, right?

> For re-copy (or update) sequences:
>
> ALTER SUBSCRIPTION ... REFRESH PUBLICATION WITH (copy_sequence = true);
>

Similar to the previous point it can be slightly confusing w.r.t
copy_data. And would copy_sequence here mean that it would copy
sequence values of both pre-existing and newly added sequences, if so,
that would make it behave differently than copy_data? The other
possibility in this direction would be to introduce an option like
replicate_all_sequences/copy_all_sequences which indicates a copy of
both pre-existing and new sequences, if any.

If we want to go in the direction of having an option such as
copy_(all)_sequences then do you think specifying that copy_data is
just for tables in the docs would be sufficient? I am afraid that it
would be confusing for users.

> >
> > In addition to the above, the command Alter Subscription .. Refresh
> > Publication will fetch any missing sequences similar to what it does
> > for tables.
>
> On the subscriber side, do we need to track which sequences are
> created via CREATE/ALTER SUBSCRIPTION?
>

I think so unless we find some other way to know at refresh
publication time which all new sequences need to be part of the
subscription. What should be the behavior w.r.t sequences when the
user performs ALTER SUBSCRIPTION ... REFRESH PUBLICATION? I was
thinking similar to tables, it should fetch any missing sequence
information from the publisher.

--
With Regards,
Amit Kapila.

On Thu, Jun 6, 2024 at 3:44 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> On Thu, Jun 6, 2024 at 9:22 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Wed, Jun 5, 2024 at 6:01 PM Ashutosh Bapat
>> <ashutosh.bapat.oss@gmail.com> wrote:
>> >
>> > On Wed, Jun 5, 2024 at 8:45 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>> >>
>> >>  How about periodically sending this information?
>> >> >
>> >>
>> >> Now, if we want to support some sort of failover then probably this
>> >> will help. Do you have that use case in mind?
>> >
>> >
>> > Regular failover was a goal for supporting logical replication of sequences. That might be more common than major
upgradescenario. 
>> >
>>
>> We can't support regular failovers to subscribers unless we can
>> replicate/copy slots because the existing nodes connected to the
>> current publisher/primary would expect that. It should be primarily
>> useful for major version upgrades at this stage.
>
>
> We don't want to design it in a way that requires major rework when we are able to copy slots and then support
regularfailover. 
>

I don't think we can just copy slots like we do for standbys. The
slots would require WAL locations to continue, so not sure if we can
make it work for failover for subscribers.

>
 That's when the consistency between a sequence and the table using it
would be a must. So it's better that we take that into consideration
now.
>

With the ideas being discussed here, I could only see the use case of
a major version upgrade or planned switchover to work. If we come up
with any other agreeable way that is better than this then we can
consider the same.

--
With Regards,
Amit Kapila.

On Thu, Jun 6, 2024 at 9:34 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jun 5, 2024 at 3:17 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > On Tue, Jun 4, 2024 at 5:40 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > Even if we decode it periodically (say each time we decode the
> > > checkpoint record) then also we need to send the entire set of
> > > sequences at shutdown. This is because the sequences may have changed
> > > from the last time we sent them.
> >
> > Agree. How about decoding and sending only the sequences that are
> > changed from the last time when they were sent? I know it requires a
> > bit of tracking and more work, but all I'm looking for is to reduce
> > the amount of work that walsenders need to do during the shutdown.
> >
>
> I see your point but going towards tracking the changed sequences
> sounds like moving towards what we do for incremental backups unless
> we can invent some other smart way.

Yes, we would need an entirely new infrastructure to track the
sequence change since the last sync. We can only determine this from
WAL, and relying on it would somehow bring us back to the approach we
were trying to achieve with logical decoding of sequences patch.

> > Having said that, I like the idea of letting the user sync the
> > sequences via ALTER SUBSCRIPTION command and not weave the logic into
> > the shutdown checkpoint path. As Peter Eisentraut said here
> > https://www.postgresql.org/message-id/42e5cb35-4aeb-4f58-8091-90619c7c3ecc%40eisentraut.org,
> > this can be a good starting point to get going.
> >
>
> Agreed.

+1

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

On Thu, Jun 6, 2024 at 6:40 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jun 6, 2024 at 11:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Wed, Jun 5, 2024 at 9:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> >
> > > To achieve this, we can allow sequences to be copied during
> > > the initial CREATE SUBSCRIPTION command similar to what we do for
> > > tables. And then later by new/existing command, we re-copy the already
> > > existing sequences on the subscriber.
> > >
> > > The options for the new command could be:
> > > Alter Subscription ... Refresh Sequences
> > > Alter Subscription ... Replicate Sequences
> > >
> > > In the second option, we need to introduce a new keyword Replicate.
> > > Can you think of any better option?
> >
> > Another idea is doing that using options. For example,
> >
> > For initial sequences synchronization:
> >
> > CREATE SUBSCRIPTION ... WITH (copy_sequence = true);
> >
>
> How will it interact with the existing copy_data option? So copy_data
> will become equivalent to copy_table_data, right?

Right.

>
> > For re-copy (or update) sequences:
> >
> > ALTER SUBSCRIPTION ... REFRESH PUBLICATION WITH (copy_sequence = true);
> >
>
> Similar to the previous point it can be slightly confusing w.r.t
> copy_data. And would copy_sequence here mean that it would copy
> sequence values of both pre-existing and newly added sequences, if so,
> that would make it behave differently than copy_data?  The other
> possibility in this direction would be to introduce an option like
> replicate_all_sequences/copy_all_sequences which indicates a copy of
> both pre-existing and new sequences, if any.

Copying sequence data works differently than replicating table data
(initial data copy and logical replication). So I thought the
copy_sequence option (or whatever better name) always does both
updating pre-existing sequences and adding new sequences. REFRESH
PUBLICATION updates the tables to be subscribed, so we also update or
add sequences associated to these tables.

>
> If we want to go in the direction of having an option such as
> copy_(all)_sequences then do you think specifying that copy_data is
> just for tables in the docs would be sufficient? I am afraid that it
> would be confusing for users.

I see your point. But I guess it would not be very problematic as it
doesn't break the current behavior and copy_(all)_sequences is
primarily for upgrade use cases.

>
> > >
> > > In addition to the above, the command Alter Subscription .. Refresh
> > > Publication will fetch any missing sequences similar to what it does
> > > for tables.
> >
> > On the subscriber side, do we need to track which sequences are
> > created via CREATE/ALTER SUBSCRIPTION?
> >
>
> I think so unless we find some other way to know at refresh
> publication time which all new sequences need to be part of the
> subscription. What should be the behavior w.r.t sequences when the
> user performs ALTER SUBSCRIPTION ... REFRESH PUBLICATION? I was
> thinking similar to tables, it should fetch any missing sequence
> information from the publisher.

It seems to make sense to me. But I have one question: do we want to
support replicating sequences that are not associated with any tables?
if yes, what if we refresh two different subscriptions that subscribe
to different tables on the same database? On the other hand, if no
(i.e. replicating only sequences owned by tables), can we know which
sequences to replicate by checking the subscribed tables?

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Fri, Jun 7, 2024 at 7:55 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Thu, Jun 6, 2024 at 6:40 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Jun 6, 2024 at 11:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Wed, Jun 5, 2024 at 9:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > >
> > > > To achieve this, we can allow sequences to be copied during
> > > > the initial CREATE SUBSCRIPTION command similar to what we do for
> > > > tables. And then later by new/existing command, we re-copy the already
> > > > existing sequences on the subscriber.
> > > >
> > > > The options for the new command could be:
> > > > Alter Subscription ... Refresh Sequences
> > > > Alter Subscription ... Replicate Sequences
> > > >
> > > > In the second option, we need to introduce a new keyword Replicate.
> > > > Can you think of any better option?
> > >
> > > Another idea is doing that using options. For example,
> > >
> > > For initial sequences synchronization:
> > >
> > > CREATE SUBSCRIPTION ... WITH (copy_sequence = true);
> > >
> >
> > How will it interact with the existing copy_data option? So copy_data
> > will become equivalent to copy_table_data, right?
>
> Right.
>
> >
> > > For re-copy (or update) sequences:
> > >
> > > ALTER SUBSCRIPTION ... REFRESH PUBLICATION WITH (copy_sequence = true);
> > >
> >
> > Similar to the previous point it can be slightly confusing w.r.t
> > copy_data. And would copy_sequence here mean that it would copy
> > sequence values of both pre-existing and newly added sequences, if so,
> > that would make it behave differently than copy_data?  The other
> > possibility in this direction would be to introduce an option like
> > replicate_all_sequences/copy_all_sequences which indicates a copy of
> > both pre-existing and new sequences, if any.
>
> Copying sequence data works differently than replicating table data
> (initial data copy and logical replication). So I thought the
> copy_sequence option (or whatever better name) always does both
> updating pre-existing sequences and adding new sequences. REFRESH
> PUBLICATION updates the tables to be subscribed, so we also update or
> add sequences associated to these tables.
>

Are you imagining the behavior for sequences associated with tables
differently than the ones defined by the CREATE SEQUENCE .. command? I
was thinking that users would associate sequences with publications
similar to what we do for tables for both cases. For example, they
need to explicitly mention the sequences they want to replicate by
commands like CREATE PUBLICATION ... FOR SEQUENCE s1, s2, ...; CREATE
PUBLICATION ... FOR ALL SEQUENCES, or CREATE PUBLICATION ... FOR
SEQUENCES IN SCHEMA sch1;

In this, variants FOR ALL SEQUENCES and SEQUENCES IN SCHEMA sch1
should copy both the explicitly defined sequences and sequences
defined with the tables. Do you think a different variant for just
copying sequences implicitly associated with tables (say for identity
columns)?

>
> >
> > > >
> > > > In addition to the above, the command Alter Subscription .. Refresh
> > > > Publication will fetch any missing sequences similar to what it does
> > > > for tables.
> > >
> > > On the subscriber side, do we need to track which sequences are
> > > created via CREATE/ALTER SUBSCRIPTION?
> > >
> >
> > I think so unless we find some other way to know at refresh
> > publication time which all new sequences need to be part of the
> > subscription. What should be the behavior w.r.t sequences when the
> > user performs ALTER SUBSCRIPTION ... REFRESH PUBLICATION? I was
> > thinking similar to tables, it should fetch any missing sequence
> > information from the publisher.
>
> It seems to make sense to me. But I have one question: do we want to
> support replicating sequences that are not associated with any tables?
>

Yes, unless we see a problem with it.

> if yes, what if we refresh two different subscriptions that subscribe
> to different tables on the same database?

What problem do you see with it?

>
 On the other hand, if no
> (i.e. replicating only sequences owned by tables), can we know which
> sequences to replicate by checking the subscribed tables?
>

Sorry, I didn't understand your question. Can you please try to
explain in more words or use some examples?

--
With Regards,
Amit Kapila.

On Wed, 5 Jun 2024 at 14:11, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jun 5, 2024 at 9:13 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Jun 4, 2024 at 8:56 PM Yogesh Sharma
> > <yogesh.sharma@catprosystems.com> wrote:
> > >
> > > On 6/4/24 06:57, Amit Kapila wrote:
> > >
> > > > 2. Provide a command say Alter Subscription ...  Replicate Sequences
> > > > (or something like that) which users can perform before shutdown of
> > > > the publisher node during upgrade. This will allow copying all the
> > > > sequences from the publisher node to the subscriber node directly.
> > > > Similar to previous approach, this could also be inconvenient for
> > > > users.
> > >
> > > This is similar to option 1 except that it is a SQL command now.
> > >
> >
> > Right, but I would still prefer a command as it provides clear steps
> > for the upgrade. Users need to perform (a) Replicate Sequences for a
> > particular subscription (b) Disable that subscription (c) Perform (a)
> > and (b) for all the subscriptions corresponding to the publisher we
> > want to shut down for upgrade.
> >
>
> Another advantage of this approach over just a plain tool to copy all
> sequences before upgrade is that here we can have the facility to copy
> just the required sequences. I mean the set sequences that the user
> has specified as part of the publication.

Here is a WIP patch to handle synchronizing the sequence during
create/alter subscription. The following changes were made for it:
Subscriber modifications:
Enable sequence synchronization during subscription creation or
alteration using the following syntax:
CREATE SUBSCRIPTION ... WITH (sequences=true);
When a subscription is created with the sequence option enabled, the
sequence list from the specified publications in the subscription will
be retrieved from the publisher. Each sequence's data will then be
copied from the remote publisher sequence to the local subscriber
sequence by using a wal receiver connection. Since all of the sequence
updating is done within a single transaction, if any errors occur
during the copying process, the entire transaction will be rolled
back.

To refresh sequences, use the syntax:
ALTER SUBSCRIPTION REFRESH SEQUENCES;
During sequence refresh, the sequence list is updated by removing
stale sequences and adding any missing sequences. The updated sequence
list is then re-synchronized.

A new catalog table, pg_subscription_seq, has been introduced for
mapping subscriptions to sequences. Additionally, the sequence LSN
(Log Sequence Number) is stored, facilitating determination of
sequence changes occurring before or after the returned sequence
state.

I have taken some code changes from Tomas's patch at [1].
I'll adjust the syntax as needed based on the ongoing discussion at  [2].

[1] - https://www.postgresql.org/message-id/09613730-5ee9-4cc3-82d8-f089be90aa64%40enterprisedb.com
[2] - https://www.postgresql.org/message-id/CAA4eK1K2X%2BPaErtGVQPD0k_5XqxjV_Cwg37%2B-pWsmKFncwc7Wg%40mail.gmail.com

Regards,
Vignesh

On Fri, Jun 7, 2024 at 7:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jun 7, 2024 at 7:55 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Thu, Jun 6, 2024 at 6:40 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, Jun 6, 2024 at 11:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Wed, Jun 5, 2024 at 9:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > >
> > > > > To achieve this, we can allow sequences to be copied during
> > > > > the initial CREATE SUBSCRIPTION command similar to what we do for
> > > > > tables. And then later by new/existing command, we re-copy the already
> > > > > existing sequences on the subscriber.
> > > > >
> > > > > The options for the new command could be:
> > > > > Alter Subscription ... Refresh Sequences
> > > > > Alter Subscription ... Replicate Sequences
> > > > >
> > > > > In the second option, we need to introduce a new keyword Replicate.
> > > > > Can you think of any better option?
> > > >
> > > > Another idea is doing that using options. For example,
> > > >
> > > > For initial sequences synchronization:
> > > >
> > > > CREATE SUBSCRIPTION ... WITH (copy_sequence = true);
> > > >
> > >
> > > How will it interact with the existing copy_data option? So copy_data
> > > will become equivalent to copy_table_data, right?
> >
> > Right.
> >
> > >
> > > > For re-copy (or update) sequences:
> > > >
> > > > ALTER SUBSCRIPTION ... REFRESH PUBLICATION WITH (copy_sequence = true);
> > > >
> > >
> > > Similar to the previous point it can be slightly confusing w.r.t
> > > copy_data. And would copy_sequence here mean that it would copy
> > > sequence values of both pre-existing and newly added sequences, if so,
> > > that would make it behave differently than copy_data?  The other
> > > possibility in this direction would be to introduce an option like
> > > replicate_all_sequences/copy_all_sequences which indicates a copy of
> > > both pre-existing and new sequences, if any.
> >
> > Copying sequence data works differently than replicating table data
> > (initial data copy and logical replication). So I thought the
> > copy_sequence option (or whatever better name) always does both
> > updating pre-existing sequences and adding new sequences. REFRESH
> > PUBLICATION updates the tables to be subscribed, so we also update or
> > add sequences associated to these tables.
> >
>
> Are you imagining the behavior for sequences associated with tables
> differently than the ones defined by the CREATE SEQUENCE .. command? I
> was thinking that users would associate sequences with publications
> similar to what we do for tables for both cases. For example, they
> need to explicitly mention the sequences they want to replicate by
> commands like CREATE PUBLICATION ... FOR SEQUENCE s1, s2, ...; CREATE
> PUBLICATION ... FOR ALL SEQUENCES, or CREATE PUBLICATION ... FOR
> SEQUENCES IN SCHEMA sch1;
>
> In this, variants FOR ALL SEQUENCES and SEQUENCES IN SCHEMA sch1
> should copy both the explicitly defined sequences and sequences
> defined with the tables. Do you think a different variant for just
> copying sequences implicitly associated with tables (say for identity
> columns)?

Oh, I was thinking that your proposal was to copy literally all
sequences by REPLICA/REFRESH SEQUENCE command. But it seems to make
sense to explicitly specify the sequences they want to replicate. It
also means that they can create a publication that has only sequences.
In this case, even if they create a subscription for that publication,
we don't launch any apply workers for that subscription. Right?

Also, given that the main use case (at least as the first step) is
version upgrade, do we really need to support SEQUENCES IN SCHEMA and
even FOR SEQUENCE? The WIP patch Vignesh recently submitted is more
than 6k lines. I think we can cut the scope for the first
implementation so as to make the review easy.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Mon, Jun 10, 2024 at 3:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Fri, Jun 7, 2024 at 7:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, Jun 7, 2024 at 7:55 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Thu, Jun 6, 2024 at 6:40 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Thu, Jun 6, 2024 at 11:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > On Wed, Jun 5, 2024 at 9:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > >
> > > > >
> > > > > > To achieve this, we can allow sequences to be copied during
> > > > > > the initial CREATE SUBSCRIPTION command similar to what we do for
> > > > > > tables. And then later by new/existing command, we re-copy the already
> > > > > > existing sequences on the subscriber.
> > > > > >
> > > > > > The options for the new command could be:
> > > > > > Alter Subscription ... Refresh Sequences
> > > > > > Alter Subscription ... Replicate Sequences
> > > > > >
> > > > > > In the second option, we need to introduce a new keyword Replicate.
> > > > > > Can you think of any better option?
> > > > >
> > > > > Another idea is doing that using options. For example,
> > > > >
> > > > > For initial sequences synchronization:
> > > > >
> > > > > CREATE SUBSCRIPTION ... WITH (copy_sequence = true);
> > > > >
> > > >
> > > > How will it interact with the existing copy_data option? So copy_data
> > > > will become equivalent to copy_table_data, right?
> > >
> > > Right.
> > >
> > > >
> > > > > For re-copy (or update) sequences:
> > > > >
> > > > > ALTER SUBSCRIPTION ... REFRESH PUBLICATION WITH (copy_sequence = true);
> > > > >
> > > >
> > > > Similar to the previous point it can be slightly confusing w.r.t
> > > > copy_data. And would copy_sequence here mean that it would copy
> > > > sequence values of both pre-existing and newly added sequences, if so,
> > > > that would make it behave differently than copy_data?  The other
> > > > possibility in this direction would be to introduce an option like
> > > > replicate_all_sequences/copy_all_sequences which indicates a copy of
> > > > both pre-existing and new sequences, if any.
> > >
> > > Copying sequence data works differently than replicating table data
> > > (initial data copy and logical replication). So I thought the
> > > copy_sequence option (or whatever better name) always does both
> > > updating pre-existing sequences and adding new sequences. REFRESH
> > > PUBLICATION updates the tables to be subscribed, so we also update or
> > > add sequences associated to these tables.
> > >
> >
> > Are you imagining the behavior for sequences associated with tables
> > differently than the ones defined by the CREATE SEQUENCE .. command? I
> > was thinking that users would associate sequences with publications
> > similar to what we do for tables for both cases. For example, they
> > need to explicitly mention the sequences they want to replicate by
> > commands like CREATE PUBLICATION ... FOR SEQUENCE s1, s2, ...; CREATE
> > PUBLICATION ... FOR ALL SEQUENCES, or CREATE PUBLICATION ... FOR
> > SEQUENCES IN SCHEMA sch1;
> >
> > In this, variants FOR ALL SEQUENCES and SEQUENCES IN SCHEMA sch1
> > should copy both the explicitly defined sequences and sequences
> > defined with the tables. Do you think a different variant for just
> > copying sequences implicitly associated with tables (say for identity
> > columns)?
>
> Oh, I was thinking that your proposal was to copy literally all
> sequences by REPLICA/REFRESH SEQUENCE command. But it seems to make
> sense to explicitly specify the sequences they want to replicate. It
> also means that they can create a publication that has only sequences.
> In this case, even if they create a subscription for that publication,
> we don't launch any apply workers for that subscription. Right?
>
> Also, given that the main use case (at least as the first step) is
> version upgrade, do we really need to support SEQUENCES IN SCHEMA and
> even FOR SEQUENCE?

Also, I guess that specifying individual sequences might not be easy
to use for users in some cases. For sequences owned by a column of a
table, users might want to specify them altogether, rather than
separately. For example, CREATE PUBLICATION ... FOR TABLE tab1 WITH
SEQUENCES means to add the table tab1 and its sequences to the
publication. For other sequences (i.e., not owned by any tables),
users might want to specify them individually.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Mon, Jun 10, 2024 at 12:43 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Jun 10, 2024 at 3:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Fri, Jun 7, 2024 at 7:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Are you imagining the behavior for sequences associated with tables
> > > differently than the ones defined by the CREATE SEQUENCE .. command? I
> > > was thinking that users would associate sequences with publications
> > > similar to what we do for tables for both cases. For example, they
> > > need to explicitly mention the sequences they want to replicate by
> > > commands like CREATE PUBLICATION ... FOR SEQUENCE s1, s2, ...; CREATE
> > > PUBLICATION ... FOR ALL SEQUENCES, or CREATE PUBLICATION ... FOR
> > > SEQUENCES IN SCHEMA sch1;
> > >
> > > In this, variants FOR ALL SEQUENCES and SEQUENCES IN SCHEMA sch1
> > > should copy both the explicitly defined sequences and sequences
> > > defined with the tables. Do you think a different variant for just
> > > copying sequences implicitly associated with tables (say for identity
> > > columns)?
> >
> > Oh, I was thinking that your proposal was to copy literally all
> > sequences by REPLICA/REFRESH SEQUENCE command.
> >

I am trying to keep the behavior as close to tables as possible.

> > But it seems to make
> > sense to explicitly specify the sequences they want to replicate. It
> > also means that they can create a publication that has only sequences.
> > In this case, even if they create a subscription for that publication,
> > we don't launch any apply workers for that subscription. Right?
> >

Right, good point. I had not thought about this.

> > Also, given that the main use case (at least as the first step) is
> > version upgrade, do we really need to support SEQUENCES IN SCHEMA and
> > even FOR SEQUENCE?
>

At the very least, we can split the patch to move these variants to a
separate patch. Once the main patch is finalized, we can try to
evaluate the remaining separately.

> Also, I guess that specifying individual sequences might not be easy
> to use for users in some cases. For sequences owned by a column of a
> table, users might want to specify them altogether, rather than
> separately. For example, CREATE PUBLICATION ... FOR TABLE tab1 WITH
> SEQUENCES means to add the table tab1 and its sequences to the
> publication. For other sequences (i.e., not owned by any tables),
> users might want to specify them individually.
>

Yeah, or we can have a syntax like CREATE PUBLICATION ... FOR TABLE
tab1 INCLUDE SEQUENCES.  Normally, we use the WITH clause for options
(For example, CREATE SUBSCRIPTION ... WITH (streaming=...)).

--
With Regards,
Amit Kapila.

On Mon, 10 Jun 2024 at 12:24, Amul Sul <sulamul@gmail.com> wrote:
>
>
>
> On Sat, Jun 8, 2024 at 6:43 PM vignesh C <vignesh21@gmail.com> wrote:
>>
>> On Wed, 5 Jun 2024 at 14:11, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> [...]
>> A new catalog table, pg_subscription_seq, has been introduced for
>> mapping subscriptions to sequences. Additionally, the sequence LSN
>> (Log Sequence Number) is stored, facilitating determination of
>> sequence changes occurring before or after the returned sequence
>> state.
>
>
> Can't it be done using pg_depend? It seems a bit excessive unless I'm missing
> something.

We'll require the lsn because the sequence LSN informs the user that
it has been synchronized up to the LSN in pg_subscription_seq. Since
we are not supporting incremental sync, the user will be able to
identify if he should run refresh sequences or not by checking the lsn
of the pg_subscription_seq and the lsn of the sequence(using
pg_sequence_state added) in the publisher.  Also, this parallels our
implementation for pg_subscription_seq and will aid in expanding for
a) incremental synchronization and b) utilizing workers for
synchronization using sequence states if necessary.

How do you track sequence mapping with the publication?

In the publisher we use pg_publication_rel and
pg_publication_namespace for mapping the sequences with the
publication.

Regards,
Vignesh
Vignesh

On Mon, 10 Jun 2024 at 14:48, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jun 10, 2024 at 12:43 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Jun 10, 2024 at 3:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Fri, Jun 7, 2024 at 7:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > >
> > > > Are you imagining the behavior for sequences associated with tables
> > > > differently than the ones defined by the CREATE SEQUENCE .. command? I
> > > > was thinking that users would associate sequences with publications
> > > > similar to what we do for tables for both cases. For example, they
> > > > need to explicitly mention the sequences they want to replicate by
> > > > commands like CREATE PUBLICATION ... FOR SEQUENCE s1, s2, ...; CREATE
> > > > PUBLICATION ... FOR ALL SEQUENCES, or CREATE PUBLICATION ... FOR
> > > > SEQUENCES IN SCHEMA sch1;
> > > >
> > > > In this, variants FOR ALL SEQUENCES and SEQUENCES IN SCHEMA sch1
> > > > should copy both the explicitly defined sequences and sequences
> > > > defined with the tables. Do you think a different variant for just
> > > > copying sequences implicitly associated with tables (say for identity
> > > > columns)?
> > >
> > > Oh, I was thinking that your proposal was to copy literally all
> > > sequences by REPLICA/REFRESH SEQUENCE command.
> > >
>
> I am trying to keep the behavior as close to tables as possible.
>
> > > But it seems to make
> > > sense to explicitly specify the sequences they want to replicate. It
> > > also means that they can create a publication that has only sequences.
> > > In this case, even if they create a subscription for that publication,
> > > we don't launch any apply workers for that subscription. Right?
> > >
>
> Right, good point. I had not thought about this.
>
> > > Also, given that the main use case (at least as the first step) is
> > > version upgrade, do we really need to support SEQUENCES IN SCHEMA and
> > > even FOR SEQUENCE?
> >
>
> At the very least, we can split the patch to move these variants to a
> separate patch. Once the main patch is finalized, we can try to
> evaluate the remaining separately.

I engaged in an offline discussion with Amit about strategizing the
division of patches to facilitate the review process. We agreed on the
following split: The first patch will encompass the setting and
getting of sequence values (core sequence changes). The second patch
will cover all changes on the publisher side related to "FOR ALL
SEQUENCES." The third patch will address subscriber side changes aimed
at synchronizing "FOR ALL SEQUENCES" publications. The fourth patch
will focus on supporting "FOR SEQUENCE" publication. Lastly, the fifth
patch will introduce support for  "FOR ALL SEQUENCES IN SCHEMA"
publication.

I will work on this and share an updated patch for the same soon.

Regards,
Vignesh

On Tue, 11 Jun 2024 at 09:41, Amul Sul <sulamul@gmail.com> wrote:
>
> On Mon, Jun 10, 2024 at 5:00 PM vignesh C <vignesh21@gmail.com> wrote:
>>
>> On Mon, 10 Jun 2024 at 12:24, Amul Sul <sulamul@gmail.com> wrote:
>> >
>> >
>> >
>> > On Sat, Jun 8, 2024 at 6:43 PM vignesh C <vignesh21@gmail.com> wrote:
>> >>
>> >> On Wed, 5 Jun 2024 at 14:11, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> >> [...]
>> >> A new catalog table, pg_subscription_seq, has been introduced for
>> >> mapping subscriptions to sequences. Additionally, the sequence LSN
>> >> (Log Sequence Number) is stored, facilitating determination of
>> >> sequence changes occurring before or after the returned sequence
>> >> state.
>> >
>> >
>> > Can't it be done using pg_depend? It seems a bit excessive unless I'm missing
>> > something.
>>
>> We'll require the lsn because the sequence LSN informs the user that
>> it has been synchronized up to the LSN in pg_subscription_seq. Since
>> we are not supporting incremental sync, the user will be able to
>> identify if he should run refresh sequences or not by checking the lsn
>> of the pg_subscription_seq and the lsn of the sequence(using
>> pg_sequence_state added) in the publisher.  Also, this parallels our
>> implementation for pg_subscription_seq and will aid in expanding for
>> a) incremental synchronization and b) utilizing workers for
>> synchronization using sequence states if necessary.
>>
>> How do you track sequence mapping with the publication?
>>
>> In the publisher we use pg_publication_rel and
>> pg_publication_namespace for mapping the sequences with the
>> publication.
>
>
> Thanks for the explanation. I'm wondering what the complexity would be, if we
> wanted to do something similar on the subscriber side, i.e., tracking via
> pg_subscription_rel.

Because we won't utilize sync workers to synchronize the sequence, and
the sequence won't necessitate sync states like init, sync,
finishedcopy, syncdone, ready, etc., initially, I considered keeping
the sequences separate. However, I'm ok with using pg_subscription_rel
as it could potentially help in enhancing incremental synchronization
and parallelizing later on.

Regards,
Vignesh

On Tue, Jun 11, 2024 at 12:25 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, 10 Jun 2024 at 14:48, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Jun 10, 2024 at 12:43 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Mon, Jun 10, 2024 at 3:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Fri, Jun 7, 2024 at 7:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > >
> > > > > Are you imagining the behavior for sequences associated with tables
> > > > > differently than the ones defined by the CREATE SEQUENCE .. command? I
> > > > > was thinking that users would associate sequences with publications
> > > > > similar to what we do for tables for both cases. For example, they
> > > > > need to explicitly mention the sequences they want to replicate by
> > > > > commands like CREATE PUBLICATION ... FOR SEQUENCE s1, s2, ...; CREATE
> > > > > PUBLICATION ... FOR ALL SEQUENCES, or CREATE PUBLICATION ... FOR
> > > > > SEQUENCES IN SCHEMA sch1;
> > > > >
> > > > > In this, variants FOR ALL SEQUENCES and SEQUENCES IN SCHEMA sch1
> > > > > should copy both the explicitly defined sequences and sequences
> > > > > defined with the tables. Do you think a different variant for just
> > > > > copying sequences implicitly associated with tables (say for identity
> > > > > columns)?
> > > >
> > > > Oh, I was thinking that your proposal was to copy literally all
> > > > sequences by REPLICA/REFRESH SEQUENCE command.
> > > >
> >
> > I am trying to keep the behavior as close to tables as possible.
> >
> > > > But it seems to make
> > > > sense to explicitly specify the sequences they want to replicate. It
> > > > also means that they can create a publication that has only sequences.
> > > > In this case, even if they create a subscription for that publication,
> > > > we don't launch any apply workers for that subscription. Right?
> > > >
> >
> > Right, good point. I had not thought about this.
> >
> > > > Also, given that the main use case (at least as the first step) is
> > > > version upgrade, do we really need to support SEQUENCES IN SCHEMA and
> > > > even FOR SEQUENCE?
> > >
> >
> > At the very least, we can split the patch to move these variants to a
> > separate patch. Once the main patch is finalized, we can try to
> > evaluate the remaining separately.
>
> I engaged in an offline discussion with Amit about strategizing the
> division of patches to facilitate the review process. We agreed on the
> following split: The first patch will encompass the setting and
> getting of sequence values (core sequence changes). The second patch
> will cover all changes on the publisher side related to "FOR ALL
> SEQUENCES." The third patch will address subscriber side changes aimed
> at synchronizing "FOR ALL SEQUENCES" publications. The fourth patch
> will focus on supporting "FOR SEQUENCE" publication. Lastly, the fifth
> patch will introduce support for  "FOR ALL SEQUENCES IN SCHEMA"
> publication.
>
> I will work on this and share an updated patch for the same soon.

+1. Sounds like a good plan.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Tue, 11 Jun 2024 at 12:38, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Jun 11, 2024 at 12:25 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Mon, 10 Jun 2024 at 14:48, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Mon, Jun 10, 2024 at 12:43 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Mon, Jun 10, 2024 at 3:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > On Fri, Jun 7, 2024 at 7:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > >
> > > > > >
> > > > > > Are you imagining the behavior for sequences associated with tables
> > > > > > differently than the ones defined by the CREATE SEQUENCE .. command? I
> > > > > > was thinking that users would associate sequences with publications
> > > > > > similar to what we do for tables for both cases. For example, they
> > > > > > need to explicitly mention the sequences they want to replicate by
> > > > > > commands like CREATE PUBLICATION ... FOR SEQUENCE s1, s2, ...; CREATE
> > > > > > PUBLICATION ... FOR ALL SEQUENCES, or CREATE PUBLICATION ... FOR
> > > > > > SEQUENCES IN SCHEMA sch1;
> > > > > >
> > > > > > In this, variants FOR ALL SEQUENCES and SEQUENCES IN SCHEMA sch1
> > > > > > should copy both the explicitly defined sequences and sequences
> > > > > > defined with the tables. Do you think a different variant for just
> > > > > > copying sequences implicitly associated with tables (say for identity
> > > > > > columns)?
> > > > >
> > > > > Oh, I was thinking that your proposal was to copy literally all
> > > > > sequences by REPLICA/REFRESH SEQUENCE command.
> > > > >
> > >
> > > I am trying to keep the behavior as close to tables as possible.
> > >
> > > > > But it seems to make
> > > > > sense to explicitly specify the sequences they want to replicate. It
> > > > > also means that they can create a publication that has only sequences.
> > > > > In this case, even if they create a subscription for that publication,
> > > > > we don't launch any apply workers for that subscription. Right?
> > > > >
> > >
> > > Right, good point. I had not thought about this.
> > >
> > > > > Also, given that the main use case (at least as the first step) is
> > > > > version upgrade, do we really need to support SEQUENCES IN SCHEMA and
> > > > > even FOR SEQUENCE?
> > > >
> > >
> > > At the very least, we can split the patch to move these variants to a
> > > separate patch. Once the main patch is finalized, we can try to
> > > evaluate the remaining separately.
> >
> > I engaged in an offline discussion with Amit about strategizing the
> > division of patches to facilitate the review process. We agreed on the
> > following split: The first patch will encompass the setting and
> > getting of sequence values (core sequence changes). The second patch
> > will cover all changes on the publisher side related to "FOR ALL
> > SEQUENCES." The third patch will address subscriber side changes aimed
> > at synchronizing "FOR ALL SEQUENCES" publications. The fourth patch
> > will focus on supporting "FOR SEQUENCE" publication. Lastly, the fifth
> > patch will introduce support for  "FOR ALL SEQUENCES IN SCHEMA"
> > publication.
> >
> > I will work on this and share an updated patch for the same soon.
>
> +1. Sounds like a good plan.

Amit and I engaged in an offline discussion regarding the design and
contemplated that it could be like below:
1) CREATE PUBLICATION syntax enhancement:
CREATE PUBLICATION ... FOR ALL SEQUENCES;
The addition of a new column titled "all sequences" in the
pg_publication system table will signify whether the publication is
designated as all sequences publication or not.

2)  CREATE SUBSCRIPTION -- no syntax change.
Upon creation of a subscription, the following additional steps will
be managed by the subscriber:
i) The subscriber will retrieve the list of sequences associated with
the subscription's publications.
ii) For each sequence: a) Retrieve the sequence value from the
publisher by invoking the pg_sequence_state function. b) Set the
sequence with the value obtained from the publisher. iv) Once the
subscription creation is completed, all sequence values will become
visible at the subscriber's end.

An alternative design approach could involve retrieving the sequence
list from the publisher during subscription creation and inserting the
sequences with an "init" state into the pg_subscription_rel system
table. These tasks could be executed by a single sequence sync worker,
which would:
i) Retrieve the list of sequences in the "init" state from the
pg_subscription_rel system table.
ii) Initiate a transaction.
iii) For each sequence: a) Obtain the sequence value from the
publisher by utilizing the pg_sequence_state function. b) Update the
sequence with the value obtained from the publisher.
iv) Commit the transaction.

The benefit with the second approach is that if there are large number
of sequences, the sequence sync can be enhanced to happen in parallel
and also if there are any locks held on the sequences in the
publisher, the sequence worker can wait to acquire the lock instead of
blocking the whole create subscription command which will delay the
initial copy of the tables too.

3) Refreshing the sequence can be achieved through the existing
command: ALTER SUBSCRIPTION ... REFRESH PUBLICATION(no syntax change
here).
The subscriber identifies stale sequences, meaning sequences present
in pg_subscription_rel but absent from the publication, and removes
them from the pg_subscription_rel system table. The subscriber also
checks for newly added sequences in the publisher and synchronizes
their values from the publisher using the steps outlined in the
subscription creation process. It's worth noting that previously
synchronized sequences won't be synchronized again; the sequence sync
will occur solely for the newly added sequences.

4) Introducing a new command for refreshing all sequences: ALTER
SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES.
The subscriber will remove stale sequences and add newly added
sequences from the publisher. Following this, it will re-synchronize
the sequence values for all sequences in the updated list from the
publisher, following the steps outlined in the subscription creation
process.

5) Incorporate the pg_sequence_state function to fetch the sequence
value from the publisher, along with the page LSN. Incorporate
SetSequence function, which will procure a new relfilenode for the
sequence and set the new relfilenode with the specified value. This
will facilitate rollback in case of any failures.

Thoughts?

Regards,
Vignesh

On Tue, Jun 11, 2024 at 7:36 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 11 Jun 2024 at 12:38, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Tue, Jun 11, 2024 at 12:25 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Mon, 10 Jun 2024 at 14:48, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Mon, Jun 10, 2024 at 12:43 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > On Mon, Jun 10, 2024 at 3:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > > >
> > > > > > On Fri, Jun 7, 2024 at 7:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > > >
> > > > > > >
> > > > > > > Are you imagining the behavior for sequences associated with tables
> > > > > > > differently than the ones defined by the CREATE SEQUENCE .. command? I
> > > > > > > was thinking that users would associate sequences with publications
> > > > > > > similar to what we do for tables for both cases. For example, they
> > > > > > > need to explicitly mention the sequences they want to replicate by
> > > > > > > commands like CREATE PUBLICATION ... FOR SEQUENCE s1, s2, ...; CREATE
> > > > > > > PUBLICATION ... FOR ALL SEQUENCES, or CREATE PUBLICATION ... FOR
> > > > > > > SEQUENCES IN SCHEMA sch1;
> > > > > > >
> > > > > > > In this, variants FOR ALL SEQUENCES and SEQUENCES IN SCHEMA sch1
> > > > > > > should copy both the explicitly defined sequences and sequences
> > > > > > > defined with the tables. Do you think a different variant for just
> > > > > > > copying sequences implicitly associated with tables (say for identity
> > > > > > > columns)?
> > > > > >
> > > > > > Oh, I was thinking that your proposal was to copy literally all
> > > > > > sequences by REPLICA/REFRESH SEQUENCE command.
> > > > > >
> > > >
> > > > I am trying to keep the behavior as close to tables as possible.
> > > >
> > > > > > But it seems to make
> > > > > > sense to explicitly specify the sequences they want to replicate. It
> > > > > > also means that they can create a publication that has only sequences.
> > > > > > In this case, even if they create a subscription for that publication,
> > > > > > we don't launch any apply workers for that subscription. Right?
> > > > > >
> > > >
> > > > Right, good point. I had not thought about this.
> > > >
> > > > > > Also, given that the main use case (at least as the first step) is
> > > > > > version upgrade, do we really need to support SEQUENCES IN SCHEMA and
> > > > > > even FOR SEQUENCE?
> > > > >
> > > >
> > > > At the very least, we can split the patch to move these variants to a
> > > > separate patch. Once the main patch is finalized, we can try to
> > > > evaluate the remaining separately.
> > >
> > > I engaged in an offline discussion with Amit about strategizing the
> > > division of patches to facilitate the review process. We agreed on the
> > > following split: The first patch will encompass the setting and
> > > getting of sequence values (core sequence changes). The second patch
> > > will cover all changes on the publisher side related to "FOR ALL
> > > SEQUENCES." The third patch will address subscriber side changes aimed
> > > at synchronizing "FOR ALL SEQUENCES" publications. The fourth patch
> > > will focus on supporting "FOR SEQUENCE" publication. Lastly, the fifth
> > > patch will introduce support for  "FOR ALL SEQUENCES IN SCHEMA"
> > > publication.
> > >
> > > I will work on this and share an updated patch for the same soon.
> >
> > +1. Sounds like a good plan.
>
> Amit and I engaged in an offline discussion regarding the design and
> contemplated that it could be like below:
> 1) CREATE PUBLICATION syntax enhancement:
> CREATE PUBLICATION ... FOR ALL SEQUENCES;
> The addition of a new column titled "all sequences" in the
> pg_publication system table will signify whether the publication is
> designated as all sequences publication or not.
>

The first approach sounds like we don't create entries for sequences
in pg_subscription_rel. In this case, how do we know all sequences
that we need to refresh when executing the REFRESH PUBLICATION
SEQUENCES command you mentioned below?

> 2)  CREATE SUBSCRIPTION -- no syntax change.
> Upon creation of a subscription, the following additional steps will
> be managed by the subscriber:
> i) The subscriber will retrieve the list of sequences associated with
> the subscription's publications.
> ii) For each sequence: a) Retrieve the sequence value from the
> publisher by invoking the pg_sequence_state function. b) Set the
> sequence with the value obtained from the publisher. iv) Once the
> subscription creation is completed, all sequence values will become
> visible at the subscriber's end.

Sequence values are always copied from the publisher? or does it
happen only when copy_data = true?

>
> An alternative design approach could involve retrieving the sequence
> list from the publisher during subscription creation and inserting the
> sequences with an "init" state into the pg_subscription_rel system
> table. These tasks could be executed by a single sequence sync worker,
> which would:
> i) Retrieve the list of sequences in the "init" state from the
> pg_subscription_rel system table.
> ii) Initiate a transaction.
> iii) For each sequence: a) Obtain the sequence value from the
> publisher by utilizing the pg_sequence_state function. b) Update the
> sequence with the value obtained from the publisher.
> iv) Commit the transaction.
>
> The benefit with the second approach is that if there are large number
> of sequences, the sequence sync can be enhanced to happen in parallel
> and also if there are any locks held on the sequences in the
> publisher, the sequence worker can wait to acquire the lock instead of
> blocking the whole create subscription command which will delay the
> initial copy of the tables too.

I prefer to have separate workers to sync sequences. Probably we can
start with a single worker and extend it to have multiple workers. BTW
the sequence-sync worker will be taken from
max_sync_workers_per_subscription pool?

Or yet another idea I came up with is that a tablesync worker will
synchronize both the table and sequences owned by the table. That is,
after the tablesync worker caught up with the apply worker, the
tablesync worker synchronizes sequences associated with the target
table as well. One benefit would be that at the time of initial table
sync being completed, the table and its sequence data are consistent.
As soon as new changes come to the table, it would become inconsistent
so it might not be helpful much, though. Also, sequences that are not
owned by any table will still need to be synchronized by someone.

>
> 3) Refreshing the sequence can be achieved through the existing
> command: ALTER SUBSCRIPTION ... REFRESH PUBLICATION(no syntax change
> here).
> The subscriber identifies stale sequences, meaning sequences present
> in pg_subscription_rel but absent from the publication, and removes
> them from the pg_subscription_rel system table. The subscriber also
> checks for newly added sequences in the publisher and synchronizes
> their values from the publisher using the steps outlined in the
> subscription creation process. It's worth noting that previously
> synchronized sequences won't be synchronized again; the sequence sync
> will occur solely for the newly added sequences.
>
> 4) Introducing a new command for refreshing all sequences: ALTER
> SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES.
> The subscriber will remove stale sequences and add newly added
> sequences from the publisher. Following this, it will re-synchronize
> the sequence values for all sequences in the updated list from the
> publisher, following the steps outlined in the subscription creation
> process.

The difference between 3) and 4) is whether or not to re-synchronize
the previously synchronized sequences. Do we really want to introduce
a new command for 4)? I felt that we can invent an option say
copy_all_sequence for the REFRESH PUBLICATION command to cover the 4)
case.

>
> 5) Incorporate the pg_sequence_state function to fetch the sequence
> value from the publisher, along with the page LSN. Incorporate
> SetSequence function, which will procure a new relfilenode for the
> sequence and set the new relfilenode with the specified value. This
> will facilitate rollback in case of any failures.

Does it mean that we create a new relfilenode for every update of the value?

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Tue, Jun 11, 2024 at 4:06 PM vignesh C <vignesh21@gmail.com> wrote:
>
> Amit and I engaged in an offline discussion regarding the design and
> contemplated that it could be like below:

If I understand correctly, does this require the sequences to already
exist on the subscribing node before creating the subscription, or
will it also copy any non-existing sequences?

> 1) CREATE PUBLICATION syntax enhancement:
> CREATE PUBLICATION ... FOR ALL SEQUENCES;
> The addition of a new column titled "all sequences" in the
> pg_publication system table will signify whether the publication is
> designated as all sequences publication or not.
>
> 2)  CREATE SUBSCRIPTION -- no syntax change.
> Upon creation of a subscription, the following additional steps will
> be managed by the subscriber:
> i) The subscriber will retrieve the list of sequences associated with
> the subscription's publications.
> ii) For each sequence: a) Retrieve the sequence value from the
> publisher by invoking the pg_sequence_state function. b) Set the
> sequence with the value obtained from the publisher. iv) Once the
> subscription creation is completed, all sequence values will become
> visible at the subscriber's end.
>
> An alternative design approach could involve retrieving the sequence
> list from the publisher during subscription creation and inserting the
> sequences with an "init" state into the pg_subscription_rel system
> table. These tasks could be executed by a single sequence sync worker,
> which would:
> i) Retrieve the list of sequences in the "init" state from the
> pg_subscription_rel system table.
> ii) Initiate a transaction.
> iii) For each sequence: a) Obtain the sequence value from the
> publisher by utilizing the pg_sequence_state function. b) Update the
> sequence with the value obtained from the publisher.
> iv) Commit the transaction.
>
> The benefit with the second approach is that if there are large number
> of sequences, the sequence sync can be enhanced to happen in parallel
> and also if there are any locks held on the sequences in the
> publisher, the sequence worker can wait to acquire the lock instead of
> blocking the whole create subscription command which will delay the
> initial copy of the tables too.

Yeah w.r.t. this point second approach seems better.

> 3) Refreshing the sequence can be achieved through the existing
> command: ALTER SUBSCRIPTION ... REFRESH PUBLICATION(no syntax change
> here).
> The subscriber identifies stale sequences, meaning sequences present
> in pg_subscription_rel but absent from the publication, and removes
> them from the pg_subscription_rel system table. The subscriber also
> checks for newly added sequences in the publisher and synchronizes
> their values from the publisher using the steps outlined in the
> subscription creation process. It's worth noting that previously
> synchronized sequences won't be synchronized again; the sequence sync
> will occur solely for the newly added sequences.

> 4) Introducing a new command for refreshing all sequences: ALTER
> SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES.
> The subscriber will remove stale sequences and add newly added
> sequences from the publisher. Following this, it will re-synchronize
> the sequence values for all sequences in the updated list from the
> publisher, following the steps outlined in the subscription creation
> process.

Okay, this answers my first question: we will remove the sequences
that are removed from the publisher and add the new sequences. I don't
see any problem with this, but doesn't it seem like we are effectively
doing DDL replication only for sequences without having a
comprehensive plan for overall DDL replication?

> 5) Incorporate the pg_sequence_state function to fetch the sequence
> value from the publisher, along with the page LSN. Incorporate
> SetSequence function, which will procure a new relfilenode for the
> sequence and set the new relfilenode with the specified value. This
> will facilitate rollback in case of any failures.

I do not understand this point, you mean whenever we are fetching the
sequence value from the publisher we need to create a new relfilenode
on the subscriber?  Why not just update the catalog tuple is
sufficient?  Or this is for handling the ALTER SEQUENCE case?

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

On Wed, Jun 12, 2024 at 10:44 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Jun 11, 2024 at 7:36 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > 1) CREATE PUBLICATION syntax enhancement:
> > CREATE PUBLICATION ... FOR ALL SEQUENCES;
> > The addition of a new column titled "all sequences" in the
> > pg_publication system table will signify whether the publication is
> > designated as all sequences publication or not.
> >
>
> The first approach sounds like we don't create entries for sequences
> in pg_subscription_rel. In this case, how do we know all sequences
> that we need to refresh when executing the REFRESH PUBLICATION
> SEQUENCES command you mentioned below?
>

As per my understanding, we should be creating entries for sequences
in pg_subscription_rel similar to tables. The difference would be that
we won't need all the sync_states (i = initialize, d = data is being
copied, f = finished table copy, s = synchronized, r = ready) as we
don't need any synchronization with apply workers.

> > 2)  CREATE SUBSCRIPTION -- no syntax change.
> > Upon creation of a subscription, the following additional steps will
> > be managed by the subscriber:
> > i) The subscriber will retrieve the list of sequences associated with
> > the subscription's publications.
> > ii) For each sequence: a) Retrieve the sequence value from the
> > publisher by invoking the pg_sequence_state function. b) Set the
> > sequence with the value obtained from the publisher. iv) Once the
> > subscription creation is completed, all sequence values will become
> > visible at the subscriber's end.
>
> Sequence values are always copied from the publisher? or does it
> happen only when copy_data = true?
>

It is better to do it when "copy_data = true" to keep it compatible
with the table's behavior.

> >
> > An alternative design approach could involve retrieving the sequence
> > list from the publisher during subscription creation and inserting the
> > sequences with an "init" state into the pg_subscription_rel system
> > table. These tasks could be executed by a single sequence sync worker,
> > which would:
> > i) Retrieve the list of sequences in the "init" state from the
> > pg_subscription_rel system table.
> > ii) Initiate a transaction.
> > iii) For each sequence: a) Obtain the sequence value from the
> > publisher by utilizing the pg_sequence_state function. b) Update the
> > sequence with the value obtained from the publisher.
> > iv) Commit the transaction.
> >
> > The benefit with the second approach is that if there are large number
> > of sequences, the sequence sync can be enhanced to happen in parallel
> > and also if there are any locks held on the sequences in the
> > publisher, the sequence worker can wait to acquire the lock instead of
> > blocking the whole create subscription command which will delay the
> > initial copy of the tables too.
>
> I prefer to have separate workers to sync sequences.
>

+1.

> Probably we can
> start with a single worker and extend it to have multiple workers.

Yeah, starting with a single worker sounds good for now. Do you think
we should sync all the sequences in a single transaction or have some
threshold value above which a different transaction would be required
or maybe a different sequence sync worker altogether? Now, having
multiple sequence-sync workers requires some synchronization so that
only a single worker is allocated for one sequence.

The simplest thing is to use a single sequence sync worker that syncs
all sequences in one transaction but with a large number of sequences,
it could be inefficient. OTOH, I am not sure if it would be a problem
in reality.

>
> BTW
> the sequence-sync worker will be taken from
> max_sync_workers_per_subscription pool?
>

I think so.

> Or yet another idea I came up with is that a tablesync worker will
> synchronize both the table and sequences owned by the table. That is,
> after the tablesync worker caught up with the apply worker, the
> tablesync worker synchronizes sequences associated with the target
> table as well. One benefit would be that at the time of initial table
> sync being completed, the table and its sequence data are consistent.
> As soon as new changes come to the table, it would become inconsistent
> so it might not be helpful much, though. Also, sequences that are not
> owned by any table will still need to be synchronized by someone.
>

The other thing to consider in this idea is that we somehow need to
distinguish the sequences owned by the table.

> >
> > 3) Refreshing the sequence can be achieved through the existing
> > command: ALTER SUBSCRIPTION ... REFRESH PUBLICATION(no syntax change
> > here).
> > The subscriber identifies stale sequences, meaning sequences present
> > in pg_subscription_rel but absent from the publication, and removes
> > them from the pg_subscription_rel system table. The subscriber also
> > checks for newly added sequences in the publisher and synchronizes
> > their values from the publisher using the steps outlined in the
> > subscription creation process. It's worth noting that previously
> > synchronized sequences won't be synchronized again; the sequence sync
> > will occur solely for the newly added sequences.
> >
> > 4) Introducing a new command for refreshing all sequences: ALTER
> > SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES.
> > The subscriber will remove stale sequences and add newly added
> > sequences from the publisher. Following this, it will re-synchronize
> > the sequence values for all sequences in the updated list from the
> > publisher, following the steps outlined in the subscription creation
> > process.
>
> The difference between 3) and 4) is whether or not to re-synchronize
> the previously synchronized sequences. Do we really want to introduce
> a new command for 4)? I felt that we can invent an option say
> copy_all_sequence for the REFRESH PUBLICATION command to cover the 4)
> case.
>

Yeah, that is also an option but it could confuse along with copy_data
option. Say the user has selected copy_data = false but
copy_all_sequences = true then the first option indicates to *not*
copy the data of table and sequences and the second option indicates
to copy the sequences data which sounds contradictory. The other idea
is to have an option copy_existing_sequences (which indicates to copy
existing sequence values) but that also has somewhat the same drawback
as copy_all_sequences but to a lesser degree.

> >
> > 5) Incorporate the pg_sequence_state function to fetch the sequence
> > value from the publisher, along with the page LSN. Incorporate
> > SetSequence function, which will procure a new relfilenode for the
> > sequence and set the new relfilenode with the specified value. This
> > will facilitate rollback in case of any failures.
>
> Does it mean that we create a new relfilenode for every update of the value?
>

We need it for initial sync so that if there is an error both the
sequence state in pg_subscription_rel and sequence values can be
rolled back together. However, it is unclear whether we need to create
a new relfilenode while copying existing sequences (say during ALTER
SUBSCRIPTION .. REFRESH PUBLICATION SEQUENCES, or whatever command we
decide)? Probably the answer lies in how we want to implement this
command. If we want to copy all sequence values during the command
itself then it is probably okay but if we want to handover this task
to the sequence-sync worker then we need some state management and a
new relfilenode so that on error both state and sequence values are
rolled back.

--
With Regards,
Amit Kapila.

On Wed, 12 Jun 2024 at 10:51, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jun 11, 2024 at 4:06 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Amit and I engaged in an offline discussion regarding the design and
> > contemplated that it could be like below:
>
> If I understand correctly, does this require the sequences to already
> exist on the subscribing node before creating the subscription, or
> will it also copy any non-existing sequences?

Sequences must exist in the subscriber; we'll synchronize only their
values. Any sequences that are not present in the subscriber will
trigger an error.

> > 1) CREATE PUBLICATION syntax enhancement:
> > CREATE PUBLICATION ... FOR ALL SEQUENCES;
> > The addition of a new column titled "all sequences" in the
> > pg_publication system table will signify whether the publication is
> > designated as all sequences publication or not.
> >
> > 2)  CREATE SUBSCRIPTION -- no syntax change.
> > Upon creation of a subscription, the following additional steps will
> > be managed by the subscriber:
> > i) The subscriber will retrieve the list of sequences associated with
> > the subscription's publications.
> > ii) For each sequence: a) Retrieve the sequence value from the
> > publisher by invoking the pg_sequence_state function. b) Set the
> > sequence with the value obtained from the publisher. iv) Once the
> > subscription creation is completed, all sequence values will become
> > visible at the subscriber's end.
> >
> > An alternative design approach could involve retrieving the sequence
> > list from the publisher during subscription creation and inserting the
> > sequences with an "init" state into the pg_subscription_rel system
> > table. These tasks could be executed by a single sequence sync worker,
> > which would:
> > i) Retrieve the list of sequences in the "init" state from the
> > pg_subscription_rel system table.
> > ii) Initiate a transaction.
> > iii) For each sequence: a) Obtain the sequence value from the
> > publisher by utilizing the pg_sequence_state function. b) Update the
> > sequence with the value obtained from the publisher.
> > iv) Commit the transaction.
> >
> > The benefit with the second approach is that if there are large number
> > of sequences, the sequence sync can be enhanced to happen in parallel
> > and also if there are any locks held on the sequences in the
> > publisher, the sequence worker can wait to acquire the lock instead of
> > blocking the whole create subscription command which will delay the
> > initial copy of the tables too.
>
> Yeah w.r.t. this point second approach seems better.

ok

> > 3) Refreshing the sequence can be achieved through the existing
> > command: ALTER SUBSCRIPTION ... REFRESH PUBLICATION(no syntax change
> > here).
> > The subscriber identifies stale sequences, meaning sequences present
> > in pg_subscription_rel but absent from the publication, and removes
> > them from the pg_subscription_rel system table. The subscriber also
> > checks for newly added sequences in the publisher and synchronizes
> > their values from the publisher using the steps outlined in the
> > subscription creation process. It's worth noting that previously
> > synchronized sequences won't be synchronized again; the sequence sync
> > will occur solely for the newly added sequences.
>
> > 4) Introducing a new command for refreshing all sequences: ALTER
> > SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES.
> > The subscriber will remove stale sequences and add newly added
> > sequences from the publisher. Following this, it will re-synchronize
> > the sequence values for all sequences in the updated list from the
> > publisher, following the steps outlined in the subscription creation
> > process.
>
> Okay, this answers my first question: we will remove the sequences
> that are removed from the publisher and add the new sequences. I don't
> see any problem with this, but doesn't it seem like we are effectively
> doing DDL replication only for sequences without having a
> comprehensive plan for overall DDL replication?

What I intended to convey is that we'll eliminate the sequences from
pg_subscription_rel. We won't facilitate the DDL replication of
sequences; instead, we anticipate users to create the sequences
themselves.

> > 5) Incorporate the pg_sequence_state function to fetch the sequence
> > value from the publisher, along with the page LSN. Incorporate
> > SetSequence function, which will procure a new relfilenode for the
> > sequence and set the new relfilenode with the specified value. This
> > will facilitate rollback in case of any failures.
>
> I do not understand this point, you mean whenever we are fetching the
> sequence value from the publisher we need to create a new relfilenode
> on the subscriber?  Why not just update the catalog tuple is
> sufficient?  Or this is for handling the ALTER SEQUENCE case?

Sequences operate distinctively from tables. Alterations to sequences
reflect instantly in another session, even before committing the
transaction. To ensure the synchronization of sequence value and state
updates in pg_subscription_rel, we assign it a new relfilenode. This
strategy ensures that any potential errors allow for the rollback of
both the sequence state in pg_subscription_rel and the sequence values
simultaneously.

Regards,
Vignesh

On Wed, Jun 12, 2024 at 4:08 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 12 Jun 2024 at 10:51, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jun 11, 2024 at 4:06 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > Amit and I engaged in an offline discussion regarding the design and
> > > contemplated that it could be like below:
> >
> > If I understand correctly, does this require the sequences to already
> > exist on the subscribing node before creating the subscription, or
> > will it also copy any non-existing sequences?
>
> Sequences must exist in the subscriber; we'll synchronize only their
> values. Any sequences that are not present in the subscriber will
> trigger an error.

Okay, that makes sense.

>
> > > 3) Refreshing the sequence can be achieved through the existing
> > > command: ALTER SUBSCRIPTION ... REFRESH PUBLICATION(no syntax change
> > > here).
> > > The subscriber identifies stale sequences, meaning sequences present
> > > in pg_subscription_rel but absent from the publication, and removes
> > > them from the pg_subscription_rel system table. The subscriber also
> > > checks for newly added sequences in the publisher and synchronizes
> > > their values from the publisher using the steps outlined in the
> > > subscription creation process. It's worth noting that previously
> > > synchronized sequences won't be synchronized again; the sequence sync
> > > will occur solely for the newly added sequences.
> >
> > > 4) Introducing a new command for refreshing all sequences: ALTER
> > > SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES.
> > > The subscriber will remove stale sequences and add newly added
> > > sequences from the publisher. Following this, it will re-synchronize
> > > the sequence values for all sequences in the updated list from the
> > > publisher, following the steps outlined in the subscription creation
> > > process.
> >
> > Okay, this answers my first question: we will remove the sequences
> > that are removed from the publisher and add the new sequences. I don't
> > see any problem with this, but doesn't it seem like we are effectively
> > doing DDL replication only for sequences without having a
> > comprehensive plan for overall DDL replication?
>
> What I intended to convey is that we'll eliminate the sequences from
> pg_subscription_rel. We won't facilitate the DDL replication of
> sequences; instead, we anticipate users to create the sequences
> themselves.

hmm okay.

> > > 5) Incorporate the pg_sequence_state function to fetch the sequence
> > > value from the publisher, along with the page LSN. Incorporate
> > > SetSequence function, which will procure a new relfilenode for the
> > > sequence and set the new relfilenode with the specified value. This
> > > will facilitate rollback in case of any failures.
> >
> > I do not understand this point, you mean whenever we are fetching the
> > sequence value from the publisher we need to create a new relfilenode
> > on the subscriber?  Why not just update the catalog tuple is
> > sufficient?  Or this is for handling the ALTER SEQUENCE case?
>
> Sequences operate distinctively from tables. Alterations to sequences
> reflect instantly in another session, even before committing the
> transaction. To ensure the synchronization of sequence value and state
> updates in pg_subscription_rel, we assign it a new relfilenode. This
> strategy ensures that any potential errors allow for the rollback of
> both the sequence state in pg_subscription_rel and the sequence values
> simultaneously.

So, you're saying that when we synchronize the sequence values on the
subscriber side, we will create a new relfilenode to allow reverting
to the old state of the sequence in case of an error or transaction
rollback? But why would we want to do that? Generally, even if you
call nextval() on a sequence and then roll back the transaction, the
sequence value doesn't revert to the old value. So, what specific
problem on the subscriber side are we trying to avoid by operating on
a new relfilenode?

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

On Wed, 12 Jun 2024 at 17:09, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Jun 12, 2024 at 4:08 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, 12 Jun 2024 at 10:51, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Tue, Jun 11, 2024 at 4:06 PM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > Amit and I engaged in an offline discussion regarding the design and
> > > > contemplated that it could be like below:
> > >
> > > If I understand correctly, does this require the sequences to already
> > > exist on the subscribing node before creating the subscription, or
> > > will it also copy any non-existing sequences?
> >
> > Sequences must exist in the subscriber; we'll synchronize only their
> > values. Any sequences that are not present in the subscriber will
> > trigger an error.
>
> Okay, that makes sense.
>
> >
> > > > 3) Refreshing the sequence can be achieved through the existing
> > > > command: ALTER SUBSCRIPTION ... REFRESH PUBLICATION(no syntax change
> > > > here).
> > > > The subscriber identifies stale sequences, meaning sequences present
> > > > in pg_subscription_rel but absent from the publication, and removes
> > > > them from the pg_subscription_rel system table. The subscriber also
> > > > checks for newly added sequences in the publisher and synchronizes
> > > > their values from the publisher using the steps outlined in the
> > > > subscription creation process. It's worth noting that previously
> > > > synchronized sequences won't be synchronized again; the sequence sync
> > > > will occur solely for the newly added sequences.
> > >
> > > > 4) Introducing a new command for refreshing all sequences: ALTER
> > > > SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES.
> > > > The subscriber will remove stale sequences and add newly added
> > > > sequences from the publisher. Following this, it will re-synchronize
> > > > the sequence values for all sequences in the updated list from the
> > > > publisher, following the steps outlined in the subscription creation
> > > > process.
> > >
> > > Okay, this answers my first question: we will remove the sequences
> > > that are removed from the publisher and add the new sequences. I don't
> > > see any problem with this, but doesn't it seem like we are effectively
> > > doing DDL replication only for sequences without having a
> > > comprehensive plan for overall DDL replication?
> >
> > What I intended to convey is that we'll eliminate the sequences from
> > pg_subscription_rel. We won't facilitate the DDL replication of
> > sequences; instead, we anticipate users to create the sequences
> > themselves.
>
> hmm okay.
>
> > > > 5) Incorporate the pg_sequence_state function to fetch the sequence
> > > > value from the publisher, along with the page LSN. Incorporate
> > > > SetSequence function, which will procure a new relfilenode for the
> > > > sequence and set the new relfilenode with the specified value. This
> > > > will facilitate rollback in case of any failures.
> > >
> > > I do not understand this point, you mean whenever we are fetching the
> > > sequence value from the publisher we need to create a new relfilenode
> > > on the subscriber?  Why not just update the catalog tuple is
> > > sufficient?  Or this is for handling the ALTER SEQUENCE case?
> >
> > Sequences operate distinctively from tables. Alterations to sequences
> > reflect instantly in another session, even before committing the
> > transaction. To ensure the synchronization of sequence value and state
> > updates in pg_subscription_rel, we assign it a new relfilenode. This
> > strategy ensures that any potential errors allow for the rollback of
> > both the sequence state in pg_subscription_rel and the sequence values
> > simultaneously.
>
> So, you're saying that when we synchronize the sequence values on the
> subscriber side, we will create a new relfilenode to allow reverting
> to the old state of the sequence in case of an error or transaction
> rollback? But why would we want to do that? Generally, even if you
> call nextval() on a sequence and then roll back the transaction, the
> sequence value doesn't revert to the old value. So, what specific
> problem on the subscriber side are we trying to avoid by operating on
> a new relfilenode?

Let's consider a situation where we have two sequences: seq1 with a
value of 100 and seq2 with a value of 200. Now, let's say seq1 is
synced and updated to 100, then we attempt to synchronize seq2,
there's a failure due to the sequence not existing or encountering
some other issue. In this scenario, we don't want to halt operations
where seq1 is synchronized, but the sequence state for sequence isn't
changed to "ready" in pg_subscription_rel.
Updating the sequence data directly reflects the sequence change
immediately. However, if we assign a new relfile node for the sequence
and update the sequence value for the new relfile node, until the
transaction is committed, other concurrent users will still be
utilizing the old relfile node for the sequence, and only the old data
will be visible. Once all sequences are synchronized, and the sequence
state is updated in pg_subscription_rel, the transaction will either
be committed or aborted. If committed, users will be able to observe
the new sequence values because the sequences will be updated with the
new relfile node containing the updated sequence value.

Regards,
Vignesh

On Thu, Jun 13, 2024 at 10:10 AM vignesh C <vignesh21@gmail.com> wrote:
>
> > So, you're saying that when we synchronize the sequence values on the
> > subscriber side, we will create a new relfilenode to allow reverting
> > to the old state of the sequence in case of an error or transaction
> > rollback? But why would we want to do that? Generally, even if you
> > call nextval() on a sequence and then roll back the transaction, the
> > sequence value doesn't revert to the old value. So, what specific
> > problem on the subscriber side are we trying to avoid by operating on
> > a new relfilenode?
>
> Let's consider a situation where we have two sequences: seq1 with a
> value of 100 and seq2 with a value of 200. Now, let's say seq1 is
> synced and updated to 100, then we attempt to synchronize seq2,
> there's a failure due to the sequence not existing or encountering
> some other issue. In this scenario, we don't want to halt operations
> where seq1 is synchronized, but the sequence state for sequence isn't
> changed to "ready" in pg_subscription_rel.

Thanks for the explanation, but I am still not getting it completely,
do you mean to say unless all the sequences are not synced any of the
sequences would not be marked "ready" in pg_subscription_rel? Is that
necessary? I mean why we can not sync the sequences one by one and
mark them ready?  Why it is necessary to either have all the sequences
synced or none of them?

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

On Thu, 13 Jun 2024 at 10:27, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Jun 13, 2024 at 10:10 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > > So, you're saying that when we synchronize the sequence values on the
> > > subscriber side, we will create a new relfilenode to allow reverting
> > > to the old state of the sequence in case of an error or transaction
> > > rollback? But why would we want to do that? Generally, even if you
> > > call nextval() on a sequence and then roll back the transaction, the
> > > sequence value doesn't revert to the old value. So, what specific
> > > problem on the subscriber side are we trying to avoid by operating on
> > > a new relfilenode?
> >
> > Let's consider a situation where we have two sequences: seq1 with a
> > value of 100 and seq2 with a value of 200. Now, let's say seq1 is
> > synced and updated to 100, then we attempt to synchronize seq2,
> > there's a failure due to the sequence not existing or encountering
> > some other issue. In this scenario, we don't want to halt operations
> > where seq1 is synchronized, but the sequence state for sequence isn't
> > changed to "ready" in pg_subscription_rel.
>
> Thanks for the explanation, but I am still not getting it completely,
> do you mean to say unless all the sequences are not synced any of the
> sequences would not be marked "ready" in pg_subscription_rel? Is that
> necessary? I mean why we can not sync the sequences one by one and
> mark them ready?  Why it is necessary to either have all the sequences
> synced or none of them?

Since updating the sequence is one operation and setting
pg_subscription_rel is another, I was trying to avoid a situation
where the sequence is updated but its state is not reflected in
pg_subscription_rel. It seems you are suggesting that it's acceptable
for the sequence to be updated even if its state isn't updated in
pg_subscription_rel, and in such cases, the sequence value does not
need to be reverted.

Regards,
Vignesh

On Thu, Jun 13, 2024 at 11:53 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 13 Jun 2024 at 10:27, Dilip Kumar <dilipbalaut@gmail.com> wrote:

> > Thanks for the explanation, but I am still not getting it completely,
> > do you mean to say unless all the sequences are not synced any of the
> > sequences would not be marked "ready" in pg_subscription_rel? Is that
> > necessary? I mean why we can not sync the sequences one by one and
> > mark them ready?  Why it is necessary to either have all the sequences
> > synced or none of them?
>
> Since updating the sequence is one operation and setting
> pg_subscription_rel is another, I was trying to avoid a situation
> where the sequence is updated but its state is not reflected in
> pg_subscription_rel. It seems you are suggesting that it's acceptable
> for the sequence to be updated even if its state isn't updated in
> pg_subscription_rel, and in such cases, the sequence value does not
> need to be reverted.

Right, the complexity we're adding to achieve a behavior that may not
be truly desirable is a concern. For instance, if we mark the status
as ready but do not sync the sequences, it could lead to issues.
However, if we have synced some sequences but encounter a failure
without marking the status as ready, I don't consider it inconsistent
in any way.  But anyway, now I understand your thinking behind that so
it's a good idea to leave this design behavior for a later decision.
Gathering more opinions and insights during later stages will provide
a clearer perspective on how to proceed with this aspect.  Thanks.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

On Wed, Jun 12, 2024 at 6:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jun 12, 2024 at 10:44 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Tue, Jun 11, 2024 at 7:36 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > 1) CREATE PUBLICATION syntax enhancement:
> > > CREATE PUBLICATION ... FOR ALL SEQUENCES;
> > > The addition of a new column titled "all sequences" in the
> > > pg_publication system table will signify whether the publication is
> > > designated as all sequences publication or not.
> > >
> >
> > The first approach sounds like we don't create entries for sequences
> > in pg_subscription_rel. In this case, how do we know all sequences
> > that we need to refresh when executing the REFRESH PUBLICATION
> > SEQUENCES command you mentioned below?
> >
>
> As per my understanding, we should be creating entries for sequences
> in pg_subscription_rel similar to tables. The difference would be that
> we won't need all the sync_states (i = initialize, d = data is being
> copied, f = finished table copy, s = synchronized, r = ready) as we
> don't need any synchronization with apply workers.

Agreed.

>
> > > 2)  CREATE SUBSCRIPTION -- no syntax change.
> > > Upon creation of a subscription, the following additional steps will
> > > be managed by the subscriber:
> > > i) The subscriber will retrieve the list of sequences associated with
> > > the subscription's publications.
> > > ii) For each sequence: a) Retrieve the sequence value from the
> > > publisher by invoking the pg_sequence_state function. b) Set the
> > > sequence with the value obtained from the publisher. iv) Once the
> > > subscription creation is completed, all sequence values will become
> > > visible at the subscriber's end.
> >
> > Sequence values are always copied from the publisher? or does it
> > happen only when copy_data = true?
> >
>
> It is better to do it when "copy_data = true" to keep it compatible
> with the table's behavior.

+1

>
> > Probably we can
> > start with a single worker and extend it to have multiple workers.
>
> Yeah, starting with a single worker sounds good for now. Do you think
> we should sync all the sequences in a single transaction or have some
> threshold value above which a different transaction would be required
> or maybe a different sequence sync worker altogether? Now, having
> multiple sequence-sync workers requires some synchronization so that
> only a single worker is allocated for one sequence.
>
> The simplest thing is to use a single sequence sync worker that syncs
> all sequences in one transaction but with a large number of sequences,
> it could be inefficient. OTOH, I am not sure if it would be a problem
> in reality.

I think that we can start with using a single worker and one
transaction, and measure the performance with a large number of
sequences.

> > Or yet another idea I came up with is that a tablesync worker will
> > synchronize both the table and sequences owned by the table. That is,
> > after the tablesync worker caught up with the apply worker, the
> > tablesync worker synchronizes sequences associated with the target
> > table as well. One benefit would be that at the time of initial table
> > sync being completed, the table and its sequence data are consistent.

Correction; it's not guaranteed that the sequence data and table data
are consistent even in this case since the tablesync worker could get
on-disk sequence data that might have already been updated.

> > As soon as new changes come to the table, it would become inconsistent
> > so it might not be helpful much, though. Also, sequences that are not
> > owned by any table will still need to be synchronized by someone.
> >
>
> The other thing to consider in this idea is that we somehow need to
> distinguish the sequences owned by the table.

I think we can check pg_depend. The owned sequences reference to the table.

>
> > >
> > > 3) Refreshing the sequence can be achieved through the existing
> > > command: ALTER SUBSCRIPTION ... REFRESH PUBLICATION(no syntax change
> > > here).
> > > The subscriber identifies stale sequences, meaning sequences present
> > > in pg_subscription_rel but absent from the publication, and removes
> > > them from the pg_subscription_rel system table. The subscriber also
> > > checks for newly added sequences in the publisher and synchronizes
> > > their values from the publisher using the steps outlined in the
> > > subscription creation process. It's worth noting that previously
> > > synchronized sequences won't be synchronized again; the sequence sync
> > > will occur solely for the newly added sequences.
> > >
> > > 4) Introducing a new command for refreshing all sequences: ALTER
> > > SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES.
> > > The subscriber will remove stale sequences and add newly added
> > > sequences from the publisher. Following this, it will re-synchronize
> > > the sequence values for all sequences in the updated list from the
> > > publisher, following the steps outlined in the subscription creation
> > > process.
> >
> > The difference between 3) and 4) is whether or not to re-synchronize
> > the previously synchronized sequences. Do we really want to introduce
> > a new command for 4)? I felt that we can invent an option say
> > copy_all_sequence for the REFRESH PUBLICATION command to cover the 4)
> > case.
> >
>
> Yeah, that is also an option but it could confuse along with copy_data
> option. Say the user has selected copy_data = false but
> copy_all_sequences = true then the first option indicates to *not*
> copy the data of table and sequences and the second option indicates
> to copy the sequences data which sounds contradictory. The other idea
> is to have an option copy_existing_sequences (which indicates to copy
> existing sequence values) but that also has somewhat the same drawback
> as copy_all_sequences but to a lesser degree.

Good point. And I understood that the REFRESH PUBLICATION SEQUENCES
command would be helpful when users want to synchronize sequences
between two nodes before upgrading.

>
> > >
> > > 5) Incorporate the pg_sequence_state function to fetch the sequence
> > > value from the publisher, along with the page LSN. Incorporate
> > > SetSequence function, which will procure a new relfilenode for the
> > > sequence and set the new relfilenode with the specified value. This
> > > will facilitate rollback in case of any failures.
> >
> > Does it mean that we create a new relfilenode for every update of the value?
> >
>
> We need it for initial sync so that if there is an error both the
> sequence state in pg_subscription_rel and sequence values can be
> rolled back together.

Agreed.

> However, it is unclear whether we need to create
> a new relfilenode while copying existing sequences (say during ALTER
> SUBSCRIPTION .. REFRESH PUBLICATION SEQUENCES, or whatever command we
> decide)? Probably the answer lies in how we want to implement this
> command. If we want to copy all sequence values during the command
> itself then it is probably okay but if we want to handover this task
> to the sequence-sync worker then we need some state management and a
> new relfilenode so that on error both state and sequence values are
> rolled back.

What state transition of pg_subscription_rel entries for sequences do
we need while copying sequences values? For example, we insert an
entry with 'init' state at CREATE SUBSCRIPTION and then the
sequence-sync worker updates to 'ready' and copies the sequence data.
And at REFRESH PUBLICATION SEQUENCES, we update the state back to
'init' again so that the sequence-sync worker can process it? Given
REFRESH PUBLICATION SEQUENCES won't be executed very frequently, it
might be acceptable to transactionally update sequence values.


Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Thu, Jun 13, 2024 at 1:09 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Jun 12, 2024 at 6:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Yeah, starting with a single worker sounds good for now. Do you think
> > we should sync all the sequences in a single transaction or have some
> > threshold value above which a different transaction would be required
> > or maybe a different sequence sync worker altogether? Now, having
> > multiple sequence-sync workers requires some synchronization so that
> > only a single worker is allocated for one sequence.
> >
> > The simplest thing is to use a single sequence sync worker that syncs
> > all sequences in one transaction but with a large number of sequences,
> > it could be inefficient. OTOH, I am not sure if it would be a problem
> > in reality.
>
> I think that we can start with using a single worker and one
> transaction, and measure the performance with a large number of
> sequences.
>

Fair enough. However, this raises the question Dilip and Vignesh are
discussing whether we need a new relfilenode for sequence update even
during initial sync? As per my understanding, the idea is that similar
to tables, the CREATE SUBSCRIPTION command (with copy_data = true)
will create the new sequence entries in pg_subscription_rel with the
state as 'i'. Then the sequence-sync worker would start a transaction
and one-by-one copy the latest sequence values for each sequence (that
has state as 'i' in pg_subscription_rel) and mark its state as ready
'r' and commit the transaction. Now if there is an error during this
operation it will restart the entire operation. The idea of creating a
new relfilenode is to handle the error so that if there is a rollback,
the sequence state will be rolled back to 'i' and the sequence value
will also be rolled back. The other option could be that we update the
sequence value without a new relfilenode and if the transaction rolled
back then only the sequence's state will be rolled back to 'i'. This
would work with a minor inconsistency that sequence values will be
up-to-date even when the sequence state is 'i' in pg_subscription_rel.
I am not sure if that matters because anyway, they can quickly be
out-of-sync with the publisher again.

Now, say we don't want to maintain the state of sequences for initial
sync at all then after the error how will we detect if there are any
pending sequences to be synced? One possibility is that we maintain a
subscription level flag 'subsequencesync' in 'pg_subscription' to
indicate whether sequences need sync. This flag would indicate whether
to sync all the sequences in pg_susbcription_rel. This would mean that
if there is an error while syncing the sequences we will resync all
the sequences again. This could be acceptable considering the chances
of error during sequence sync are low. The benefit is that both the
REFRESH PUBLICATION SEQUENCES and CREATE SUBSCRIPTION can use the same
idea and sync all sequences without needing a new relfilenode. Users
can always refer 'subsequencesync' flag in 'pg_subscription' to see if
all the sequences are synced after executing the command.

> > > Or yet another idea I came up with is that a tablesync worker will
> > > synchronize both the table and sequences owned by the table. That is,
> > > after the tablesync worker caught up with the apply worker, the
> > > tablesync worker synchronizes sequences associated with the target
> > > table as well. One benefit would be that at the time of initial table
> > > sync being completed, the table and its sequence data are consistent.
>
> Correction; it's not guaranteed that the sequence data and table data
> are consistent even in this case since the tablesync worker could get
> on-disk sequence data that might have already been updated.
>

The benefit of this approach is not clear to me. Our aim is to sync
all sequences before the upgrade, so not sure if this helps because
anyway both table values and corresponding sequences can again be
out-of-sync very quickly.

> >
> > > >
> > > > 3) Refreshing the sequence can be achieved through the existing
> > > > command: ALTER SUBSCRIPTION ... REFRESH PUBLICATION(no syntax change
> > > > here).
> > > > The subscriber identifies stale sequences, meaning sequences present
> > > > in pg_subscription_rel but absent from the publication, and removes
> > > > them from the pg_subscription_rel system table. The subscriber also
> > > > checks for newly added sequences in the publisher and synchronizes
> > > > their values from the publisher using the steps outlined in the
> > > > subscription creation process. It's worth noting that previously
> > > > synchronized sequences won't be synchronized again; the sequence sync
> > > > will occur solely for the newly added sequences.
> > > >
> > > > 4) Introducing a new command for refreshing all sequences: ALTER
> > > > SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES.
> > > > The subscriber will remove stale sequences and add newly added
> > > > sequences from the publisher. Following this, it will re-synchronize
> > > > the sequence values for all sequences in the updated list from the
> > > > publisher, following the steps outlined in the subscription creation
> > > > process.
> > >
> > > The difference between 3) and 4) is whether or not to re-synchronize
> > > the previously synchronized sequences. Do we really want to introduce
> > > a new command for 4)? I felt that we can invent an option say
> > > copy_all_sequence for the REFRESH PUBLICATION command to cover the 4)
> > > case.
> > >
> >
> > Yeah, that is also an option but it could confuse along with copy_data
> > option. Say the user has selected copy_data = false but
> > copy_all_sequences = true then the first option indicates to *not*
> > copy the data of table and sequences and the second option indicates
> > to copy the sequences data which sounds contradictory. The other idea
> > is to have an option copy_existing_sequences (which indicates to copy
> > existing sequence values) but that also has somewhat the same drawback
> > as copy_all_sequences but to a lesser degree.
>
> Good point. And I understood that the REFRESH PUBLICATION SEQUENCES
> command would be helpful when users want to synchronize sequences
> between two nodes before upgrading.
>

Right.

> >
> > > >
> > > > 5) Incorporate the pg_sequence_state function to fetch the sequence
> > > > value from the publisher, along with the page LSN. Incorporate
> > > > SetSequence function, which will procure a new relfilenode for the
> > > > sequence and set the new relfilenode with the specified value. This
> > > > will facilitate rollback in case of any failures.
> > >
> > > Does it mean that we create a new relfilenode for every update of the value?
> > >
> >
> > We need it for initial sync so that if there is an error both the
> > sequence state in pg_subscription_rel and sequence values can be
> > rolled back together.
>
> Agreed.
>
> > However, it is unclear whether we need to create
> > a new relfilenode while copying existing sequences (say during ALTER
> > SUBSCRIPTION .. REFRESH PUBLICATION SEQUENCES, or whatever command we
> > decide)? Probably the answer lies in how we want to implement this
> > command. If we want to copy all sequence values during the command
> > itself then it is probably okay but if we want to handover this task
> > to the sequence-sync worker then we need some state management and a
> > new relfilenode so that on error both state and sequence values are
> > rolled back.
>
> What state transition of pg_subscription_rel entries for sequences do
> we need while copying sequences values? For example, we insert an
> entry with 'init' state at CREATE SUBSCRIPTION and then the
> sequence-sync worker updates to 'ready' and copies the sequence data.
> And at REFRESH PUBLICATION SEQUENCES, we update the state back to
> 'init' again so that the sequence-sync worker can process it? Given
> REFRESH PUBLICATION SEQUENCES won't be executed very frequently, it
> might be acceptable to transactionally update sequence values.
>

Do you mean that sync the sequences during the REFRESH PUBLICATION
SEQUENCES command itself? If so, there is an argument that we can do
the same during CREATE SUBSCRIPTION. It would be beneficial to keep
the method to sync the sequences same for both the CREATE and REFRESH
commands. I have speculated on one idea above and would be happy to
see your thoughts.

--
With Regards,
Amit Kapila.

On Thu, Jun 13, 2024 at 7:06 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jun 13, 2024 at 1:09 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Wed, Jun 12, 2024 at 6:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Yeah, starting with a single worker sounds good for now. Do you think
> > > we should sync all the sequences in a single transaction or have some
> > > threshold value above which a different transaction would be required
> > > or maybe a different sequence sync worker altogether? Now, having
> > > multiple sequence-sync workers requires some synchronization so that
> > > only a single worker is allocated for one sequence.
> > >
> > > The simplest thing is to use a single sequence sync worker that syncs
> > > all sequences in one transaction but with a large number of sequences,
> > > it could be inefficient. OTOH, I am not sure if it would be a problem
> > > in reality.
> >
> > I think that we can start with using a single worker and one
> > transaction, and measure the performance with a large number of
> > sequences.
> >
>
> Fair enough. However, this raises the question Dilip and Vignesh are
> discussing whether we need a new relfilenode for sequence update even
> during initial sync? As per my understanding, the idea is that similar
> to tables, the CREATE SUBSCRIPTION command (with copy_data = true)
> will create the new sequence entries in pg_subscription_rel with the
> state as 'i'. Then the sequence-sync worker would start a transaction
> and one-by-one copy the latest sequence values for each sequence (that
> has state as 'i' in pg_subscription_rel) and mark its state as ready
> 'r' and commit the transaction. Now if there is an error during this
> operation it will restart the entire operation. The idea of creating a
> new relfilenode is to handle the error so that if there is a rollback,
> the sequence state will be rolled back to 'i' and the sequence value
> will also be rolled back. The other option could be that we update the
> sequence value without a new relfilenode and if the transaction rolled
> back then only the sequence's state will be rolled back to 'i'. This
> would work with a minor inconsistency that sequence values will be
> up-to-date even when the sequence state is 'i' in pg_subscription_rel.
> I am not sure if that matters because anyway, they can quickly be
> out-of-sync with the publisher again.

I think it would be fine in many cases even if the sequence value is
up-to-date even when the sequence state is 'i' in pg_subscription_rel.
But the case we would like to avoid is where suppose the sequence-sync
worker does both synchronizing sequence values and updating the
sequence states for all sequences in one transaction, and if there is
an error we end up retrying the synchronization for all sequences.

>
> Now, say we don't want to maintain the state of sequences for initial
> sync at all then after the error how will we detect if there are any
> pending sequences to be synced? One possibility is that we maintain a
> subscription level flag 'subsequencesync' in 'pg_subscription' to
> indicate whether sequences need sync. This flag would indicate whether
> to sync all the sequences in pg_susbcription_rel. This would mean that
> if there is an error while syncing the sequences we will resync all
> the sequences again. This could be acceptable considering the chances
> of error during sequence sync are low. The benefit is that both the
> REFRESH PUBLICATION SEQUENCES and CREATE SUBSCRIPTION can use the same
> idea and sync all sequences without needing a new relfilenode. Users
> can always refer 'subsequencesync' flag in 'pg_subscription' to see if
> all the sequences are synced after executing the command.

I think that REFRESH PUBLICATION {SEQUENCES} can be executed even
while the sequence-sync worker is synchronizing sequences. In this
case, the worker might not see new sequences added by the concurrent
REFRESH PUBLICATION {SEQUENCES} command since it's already running.
The worker could end up marking the subsequencesync as completed while
not synchronizing these new sequences.

>
> > > > Or yet another idea I came up with is that a tablesync worker will
> > > > synchronize both the table and sequences owned by the table. That is,
> > > > after the tablesync worker caught up with the apply worker, the
> > > > tablesync worker synchronizes sequences associated with the target
> > > > table as well. One benefit would be that at the time of initial table
> > > > sync being completed, the table and its sequence data are consistent.
> >
> > Correction; it's not guaranteed that the sequence data and table data
> > are consistent even in this case since the tablesync worker could get
> > on-disk sequence data that might have already been updated.
> >
>
> The benefit of this approach is not clear to me. Our aim is to sync
> all sequences before the upgrade, so not sure if this helps because
> anyway both table values and corresponding sequences can again be
> out-of-sync very quickly.

Right.

Given that our aim is to sync all sequences before the upgrade, do we
need to synchronize sequences even at CREATE SUBSCRIPTION time? In
cases where there are a large number of sequences, synchronizing
sequences in addition to tables could be overhead and make less sense,
because sequences can again be out-of-sync quickly and typically
CREATE SUBSCRIPTION is not created just before the upgrade.

>
> > >
> > > > >
> > > > > 5) Incorporate the pg_sequence_state function to fetch the sequence
> > > > > value from the publisher, along with the page LSN. Incorporate
> > > > > SetSequence function, which will procure a new relfilenode for the
> > > > > sequence and set the new relfilenode with the specified value. This
> > > > > will facilitate rollback in case of any failures.
> > > >
> > > > Does it mean that we create a new relfilenode for every update of the value?
> > > >
> > >
> > > We need it for initial sync so that if there is an error both the
> > > sequence state in pg_subscription_rel and sequence values can be
> > > rolled back together.
> >
> > Agreed.
> >
> > > However, it is unclear whether we need to create
> > > a new relfilenode while copying existing sequences (say during ALTER
> > > SUBSCRIPTION .. REFRESH PUBLICATION SEQUENCES, or whatever command we
> > > decide)? Probably the answer lies in how we want to implement this
> > > command. If we want to copy all sequence values during the command
> > > itself then it is probably okay but if we want to handover this task
> > > to the sequence-sync worker then we need some state management and a
> > > new relfilenode so that on error both state and sequence values are
> > > rolled back.
> >
> > What state transition of pg_subscription_rel entries for sequences do
> > we need while copying sequences values? For example, we insert an
> > entry with 'init' state at CREATE SUBSCRIPTION and then the
> > sequence-sync worker updates to 'ready' and copies the sequence data.
> > And at REFRESH PUBLICATION SEQUENCES, we update the state back to
> > 'init' again so that the sequence-sync worker can process it? Given
> > REFRESH PUBLICATION SEQUENCES won't be executed very frequently, it
> > might be acceptable to transactionally update sequence values.
> >
>
> Do you mean that sync the sequences during the REFRESH PUBLICATION
> SEQUENCES command itself? If so, there is an argument that we can do
> the same during CREATE SUBSCRIPTION. It would be beneficial to keep
> the method to sync the sequences same for both the CREATE and REFRESH
> commands. I have speculated on one idea above and would be happy to
> see your thoughts.

I meant that the REFRESH PUBLICATION SEQUENCES command updates all
sequence states in pg_subscription_rel to 'init' state, and the
sequence-sync worker can do the synchronization work. We use the same
method for both the CREATE SUBSCRIPTION and REFRESH PUBLICATION
{SEQUENCES} commands.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Thu, Jun 13, 2024 at 03:36:05PM +0530, Amit Kapila wrote:
> Fair enough. However, this raises the question Dilip and Vignesh are
> discussing whether we need a new relfilenode for sequence update even
> during initial sync? As per my understanding, the idea is that similar
> to tables, the CREATE SUBSCRIPTION command (with copy_data = true)
> will create the new sequence entries in pg_subscription_rel with the
> state as 'i'. Then the sequence-sync worker would start a transaction
> and one-by-one copy the latest sequence values for each sequence (that
> has state as 'i' in pg_subscription_rel) and mark its state as ready
> 'r' and commit the transaction. Now if there is an error during this
> operation it will restart the entire operation.

Hmm.  You mean to use only one transaction for all the sequences?
I've heard about deployments with a lot of them.  Could it be a
problem to process them in batches, as well?  If you maintain a state
for each one of them in pg_subscription_rel, it does not strike me as
an issue, while being more flexible than an all-or-nothing.

> The idea of creating a
> new relfilenode is to handle the error so that if there is a rollback,
> the sequence state will be rolled back to 'i' and the sequence value
> will also be rolled back. The other option could be that we update the
> sequence value without a new relfilenode and if the transaction rolled
> back then only the sequence's state will be rolled back to 'i'. This
> would work with a minor inconsistency that sequence values will be
> up-to-date even when the sequence state is 'i' in pg_subscription_rel.
> I am not sure if that matters because anyway, they can quickly be
> out-of-sync with the publisher again.

Seeing a mention to relfilenodes specifically for sequences freaks me
out a bit, because there's some work I have been doing in this area
and sequences may not have a need for a physical relfilenode at all.
But I guess that you refer to the fact that like tables, relfilenodes
would only be created as required because anything you'd do in the
apply worker path would just call some of the routines of sequence.h,
right?

> Now, say we don't want to maintain the state of sequences for initial
> sync at all then after the error how will we detect if there are any
> pending sequences to be synced? One possibility is that we maintain a
> subscription level flag 'subsequencesync' in 'pg_subscription' to
> indicate whether sequences need sync. This flag would indicate whether
> to sync all the sequences in pg_susbcription_rel. This would mean that
> if there is an error while syncing the sequences we will resync all
> the sequences again. This could be acceptable considering the chances
> of error during sequence sync are low.

There could be multiple subscriptions to a single database that point
to the same set of sequences.  Is there any conflict issue to worry
about here?

> The benefit is that both the
> REFRESH PUBLICATION SEQUENCES and CREATE SUBSCRIPTION can use the same
> idea and sync all sequences without needing a new relfilenode. Users
> can always refer 'subsequencesync' flag in 'pg_subscription' to see if
> all the sequences are synced after executing the command.

That would be cheaper, indeed.  Isn't a boolean too limiting?
Isn't that something you'd want to track with a LSN as "the point in
WAL where all the sequences have been synced"?

The approach of doing all the sync work from the subscriber, while
having a command that can be kicked from the subscriber side is a good
user experience.
--
Michael

On Thu, Jun 13, 2024 at 6:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Thu, Jun 13, 2024 at 7:06 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Jun 13, 2024 at 1:09 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Wed, Jun 12, 2024 at 6:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > >
> > > > Yeah, starting with a single worker sounds good for now. Do you think
> > > > we should sync all the sequences in a single transaction or have some
> > > > threshold value above which a different transaction would be required
> > > > or maybe a different sequence sync worker altogether? Now, having
> > > > multiple sequence-sync workers requires some synchronization so that
> > > > only a single worker is allocated for one sequence.
> > > >
> > > > The simplest thing is to use a single sequence sync worker that syncs
> > > > all sequences in one transaction but with a large number of sequences,
> > > > it could be inefficient. OTOH, I am not sure if it would be a problem
> > > > in reality.
> > >
> > > I think that we can start with using a single worker and one
> > > transaction, and measure the performance with a large number of
> > > sequences.
> > >
> >
> > Fair enough. However, this raises the question Dilip and Vignesh are
> > discussing whether we need a new relfilenode for sequence update even
> > during initial sync? As per my understanding, the idea is that similar
> > to tables, the CREATE SUBSCRIPTION command (with copy_data = true)
> > will create the new sequence entries in pg_subscription_rel with the
> > state as 'i'. Then the sequence-sync worker would start a transaction
> > and one-by-one copy the latest sequence values for each sequence (that
> > has state as 'i' in pg_subscription_rel) and mark its state as ready
> > 'r' and commit the transaction. Now if there is an error during this
> > operation it will restart the entire operation. The idea of creating a
> > new relfilenode is to handle the error so that if there is a rollback,
> > the sequence state will be rolled back to 'i' and the sequence value
> > will also be rolled back. The other option could be that we update the
> > sequence value without a new relfilenode and if the transaction rolled
> > back then only the sequence's state will be rolled back to 'i'. This
> > would work with a minor inconsistency that sequence values will be
> > up-to-date even when the sequence state is 'i' in pg_subscription_rel.
> > I am not sure if that matters because anyway, they can quickly be
> > out-of-sync with the publisher again.
>
> I think it would be fine in many cases even if the sequence value is
> up-to-date even when the sequence state is 'i' in pg_subscription_rel.
> But the case we would like to avoid is where suppose the sequence-sync
> worker does both synchronizing sequence values and updating the
> sequence states for all sequences in one transaction, and if there is
> an error we end up retrying the synchronization for all sequences.
>

The one idea to avoid this is to update sequences in chunks (say 100
or some threshold number of sequences in one transaction). Then we
would only redo the sync for the last and pending set of sequences.

> >
> > Now, say we don't want to maintain the state of sequences for initial
> > sync at all then after the error how will we detect if there are any
> > pending sequences to be synced? One possibility is that we maintain a
> > subscription level flag 'subsequencesync' in 'pg_subscription' to
> > indicate whether sequences need sync. This flag would indicate whether
> > to sync all the sequences in pg_susbcription_rel. This would mean that
> > if there is an error while syncing the sequences we will resync all
> > the sequences again. This could be acceptable considering the chances
> > of error during sequence sync are low. The benefit is that both the
> > REFRESH PUBLICATION SEQUENCES and CREATE SUBSCRIPTION can use the same
> > idea and sync all sequences without needing a new relfilenode. Users
> > can always refer 'subsequencesync' flag in 'pg_subscription' to see if
> > all the sequences are synced after executing the command.
>
> I think that REFRESH PUBLICATION {SEQUENCES} can be executed even
> while the sequence-sync worker is synchronizing sequences. In this
> case, the worker might not see new sequences added by the concurrent
> REFRESH PUBLICATION {SEQUENCES} command since it's already running.
> The worker could end up marking the subsequencesync as completed while
> not synchronizing these new sequences.
>

This is possible but we could avoid REFRESH PUBLICATION {SEQUENCES} by
not allowing to change the subsequencestate during the time
sequence-worker is syncing the sequences. This could be restrictive
but there doesn't seem to be cases where user would like to
immediately refresh sequences after creating the subscription.

> >
> > > > > Or yet another idea I came up with is that a tablesync worker will
> > > > > synchronize both the table and sequences owned by the table. That is,
> > > > > after the tablesync worker caught up with the apply worker, the
> > > > > tablesync worker synchronizes sequences associated with the target
> > > > > table as well. One benefit would be that at the time of initial table
> > > > > sync being completed, the table and its sequence data are consistent.
> > >
> > > Correction; it's not guaranteed that the sequence data and table data
> > > are consistent even in this case since the tablesync worker could get
> > > on-disk sequence data that might have already been updated.
> > >
> >
> > The benefit of this approach is not clear to me. Our aim is to sync
> > all sequences before the upgrade, so not sure if this helps because
> > anyway both table values and corresponding sequences can again be
> > out-of-sync very quickly.
>
> Right.
>
> Given that our aim is to sync all sequences before the upgrade, do we
> need to synchronize sequences even at CREATE SUBSCRIPTION time? In
> cases where there are a large number of sequences, synchronizing
> sequences in addition to tables could be overhead and make less sense,
> because sequences can again be out-of-sync quickly and typically
> CREATE SUBSCRIPTION is not created just before the upgrade.
>

I think for the upgrade one should be creating a subscription just
before the upgrade. Isn't something similar is done even in the
upgrade steps you shared once [1]? Typically users should get all the
data from the publisher before the upgrade of the publisher via
creating a subscription. Also, it would be better to keep the
implementation of sequences close to tables wherever possible. Having
said that, I understand your point as well and if you strongly feel
that we don't need to sync sequences at the time of CREATE
SUBSCRIPTION and others also don't see any problem with it then we can
consider that as well.

> > >
> >
> > Do you mean that sync the sequences during the REFRESH PUBLICATION
> > SEQUENCES command itself? If so, there is an argument that we can do
> > the same during CREATE SUBSCRIPTION. It would be beneficial to keep
> > the method to sync the sequences same for both the CREATE and REFRESH
> > commands. I have speculated on one idea above and would be happy to
> > see your thoughts.
>
> I meant that the REFRESH PUBLICATION SEQUENCES command updates all
> sequence states in pg_subscription_rel to 'init' state, and the
> sequence-sync worker can do the synchronization work. We use the same
> method for both the CREATE SUBSCRIPTION and REFRESH PUBLICATION
> {SEQUENCES} commands.
>

Marking the state as 'init' when we would have already synced the
sequences sounds a bit odd but otherwise, this could also work if we
accept that even if the sequences are synced and value could remain in
'init' state (on rollbacks).


[1] - https://knock.app/blog/zero-downtime-postgres-upgrades

--
With Regards,
Amit Kapila.

On Fri, Jun 14, 2024 at 5:16 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Thu, Jun 13, 2024 at 03:36:05PM +0530, Amit Kapila wrote:
> > Fair enough. However, this raises the question Dilip and Vignesh are
> > discussing whether we need a new relfilenode for sequence update even
> > during initial sync? As per my understanding, the idea is that similar
> > to tables, the CREATE SUBSCRIPTION command (with copy_data = true)
> > will create the new sequence entries in pg_subscription_rel with the
> > state as 'i'. Then the sequence-sync worker would start a transaction
> > and one-by-one copy the latest sequence values for each sequence (that
> > has state as 'i' in pg_subscription_rel) and mark its state as ready
> > 'r' and commit the transaction. Now if there is an error during this
> > operation it will restart the entire operation.
>
> Hmm.  You mean to use only one transaction for all the sequences?
> I've heard about deployments with a lot of them.  Could it be a
> problem to process them in batches, as well?

I don't think so. We can even sync one sequence per transaction but
then it would be resource and time consuming without much gain. As
mentioned in a previous email, we might want to sync 100 or some other
threshold number of sequences per transaction. The other possibility
is to make a subscription-level option for this batch size but I don't
see much advantage in doing so as it won't be convenient for users to
set it. I feel we should pick some threshold number that is neither
too low nor too high and if we later see any problem with it, we can
make it a configurable knob.

>
> > The idea of creating a
> > new relfilenode is to handle the error so that if there is a rollback,
> > the sequence state will be rolled back to 'i' and the sequence value
> > will also be rolled back. The other option could be that we update the
> > sequence value without a new relfilenode and if the transaction rolled
> > back then only the sequence's state will be rolled back to 'i'. This
> > would work with a minor inconsistency that sequence values will be
> > up-to-date even when the sequence state is 'i' in pg_subscription_rel.
> > I am not sure if that matters because anyway, they can quickly be
> > out-of-sync with the publisher again.
>
> Seeing a mention to relfilenodes specifically for sequences freaks me
> out a bit, because there's some work I have been doing in this area
> and sequences may not have a need for a physical relfilenode at all.
> But I guess that you refer to the fact that like tables, relfilenodes
> would only be created as required because anything you'd do in the
> apply worker path would just call some of the routines of sequence.h,
> right?
>

Yes, I think so. The only thing the patch expects is a way to rollback
the sequence changes if the transaction rolls back during the initial
sync. But I am not sure if we need such a behavior. The discussion for
the same is in progress. Let's wait for the outcome.

> > Now, say we don't want to maintain the state of sequences for initial
> > sync at all then after the error how will we detect if there are any
> > pending sequences to be synced? One possibility is that we maintain a
> > subscription level flag 'subsequencesync' in 'pg_subscription' to
> > indicate whether sequences need sync. This flag would indicate whether
> > to sync all the sequences in pg_susbcription_rel. This would mean that
> > if there is an error while syncing the sequences we will resync all
> > the sequences again. This could be acceptable considering the chances
> > of error during sequence sync are low.
>
> There could be multiple subscriptions to a single database that point
> to the same set of sequences.  Is there any conflict issue to worry
> about here?
>

I don't think so. In the worst case, the same value would be copied
twice. The same scenario in case of tables could lead to duplicate
data or unique key violation ERRORs which is much worse. So, I expect
users to be careful about the same.

> > The benefit is that both the
> > REFRESH PUBLICATION SEQUENCES and CREATE SUBSCRIPTION can use the same
> > idea and sync all sequences without needing a new relfilenode. Users
> > can always refer 'subsequencesync' flag in 'pg_subscription' to see if
> > all the sequences are synced after executing the command.
>
> That would be cheaper, indeed.  Isn't a boolean too limiting?
>

In this idea, we only need a flag to say whether the sequence sync is
required or not.

> Isn't that something you'd want to track with a LSN as "the point in
> WAL where all the sequences have been synced"?
>

It won't be any better for the required purpose because after CREATE
SUBSCRIPTION, if REFERESH wants to toggle the flag to indicate the
sequences need sync again then using LSN would mean we need to set it
to Invalid value.

> The approach of doing all the sync work from the subscriber, while
> having a command that can be kicked from the subscriber side is a good
> user experience.
>

Thank you for endorsing the idea.

--
With Regards,
Amit Kapila.

On Fri, Jun 14, 2024 at 4:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jun 13, 2024 at 6:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Thu, Jun 13, 2024 at 7:06 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, Jun 13, 2024 at 1:09 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Wed, Jun 12, 2024 at 6:59 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > >
> > > > > Yeah, starting with a single worker sounds good for now. Do you think
> > > > > we should sync all the sequences in a single transaction or have some
> > > > > threshold value above which a different transaction would be required
> > > > > or maybe a different sequence sync worker altogether? Now, having
> > > > > multiple sequence-sync workers requires some synchronization so that
> > > > > only a single worker is allocated for one sequence.
> > > > >
> > > > > The simplest thing is to use a single sequence sync worker that syncs
> > > > > all sequences in one transaction but with a large number of sequences,
> > > > > it could be inefficient. OTOH, I am not sure if it would be a problem
> > > > > in reality.
> > > >
> > > > I think that we can start with using a single worker and one
> > > > transaction, and measure the performance with a large number of
> > > > sequences.
> > > >
> > >
> > > Fair enough. However, this raises the question Dilip and Vignesh are
> > > discussing whether we need a new relfilenode for sequence update even
> > > during initial sync? As per my understanding, the idea is that similar
> > > to tables, the CREATE SUBSCRIPTION command (with copy_data = true)
> > > will create the new sequence entries in pg_subscription_rel with the
> > > state as 'i'. Then the sequence-sync worker would start a transaction
> > > and one-by-one copy the latest sequence values for each sequence (that
> > > has state as 'i' in pg_subscription_rel) and mark its state as ready
> > > 'r' and commit the transaction. Now if there is an error during this
> > > operation it will restart the entire operation. The idea of creating a
> > > new relfilenode is to handle the error so that if there is a rollback,
> > > the sequence state will be rolled back to 'i' and the sequence value
> > > will also be rolled back. The other option could be that we update the
> > > sequence value without a new relfilenode and if the transaction rolled
> > > back then only the sequence's state will be rolled back to 'i'. This
> > > would work with a minor inconsistency that sequence values will be
> > > up-to-date even when the sequence state is 'i' in pg_subscription_rel.
> > > I am not sure if that matters because anyway, they can quickly be
> > > out-of-sync with the publisher again.
> >
> > I think it would be fine in many cases even if the sequence value is
> > up-to-date even when the sequence state is 'i' in pg_subscription_rel.
> > But the case we would like to avoid is where suppose the sequence-sync
> > worker does both synchronizing sequence values and updating the
> > sequence states for all sequences in one transaction, and if there is
> > an error we end up retrying the synchronization for all sequences.
> >
>
> The one idea to avoid this is to update sequences in chunks (say 100
> or some threshold number of sequences in one transaction). Then we
> would only redo the sync for the last and pending set of sequences.

That could be one idea.

>
> > >
> > > Now, say we don't want to maintain the state of sequences for initial
> > > sync at all then after the error how will we detect if there are any
> > > pending sequences to be synced? One possibility is that we maintain a
> > > subscription level flag 'subsequencesync' in 'pg_subscription' to
> > > indicate whether sequences need sync. This flag would indicate whether
> > > to sync all the sequences in pg_susbcription_rel. This would mean that
> > > if there is an error while syncing the sequences we will resync all
> > > the sequences again. This could be acceptable considering the chances
> > > of error during sequence sync are low. The benefit is that both the
> > > REFRESH PUBLICATION SEQUENCES and CREATE SUBSCRIPTION can use the same
> > > idea and sync all sequences without needing a new relfilenode. Users
> > > can always refer 'subsequencesync' flag in 'pg_subscription' to see if
> > > all the sequences are synced after executing the command.
> >
> > I think that REFRESH PUBLICATION {SEQUENCES} can be executed even
> > while the sequence-sync worker is synchronizing sequences. In this
> > case, the worker might not see new sequences added by the concurrent
> > REFRESH PUBLICATION {SEQUENCES} command since it's already running.
> > The worker could end up marking the subsequencesync as completed while
> > not synchronizing these new sequences.
> >
>
> This is possible but we could avoid REFRESH PUBLICATION {SEQUENCES} by
> not allowing to change the subsequencestate during the time
> sequence-worker is syncing the sequences. This could be restrictive
> but there doesn't seem to be cases where user would like to
> immediately refresh sequences after creating the subscription.

I'm concerned that users would not be able to add sequences during the
time the sequence-worker is syncing the sequences. For example,
suppose we have 10000 sequences and execute REFRESH PUBLICATION
{SEQUENCES} to synchronize 10000 sequences. Now if we add one sequence
to the publication and want to synchronize it to the subscriber, we
have to wait for the current REFRESH PUBLICATION {SEQUENCES} to
complete, and then execute it again, synchronizing 10001 sequences,
instead of synchronizing only the new one.

>
> > >
> > > > > > Or yet another idea I came up with is that a tablesync worker will
> > > > > > synchronize both the table and sequences owned by the table. That is,
> > > > > > after the tablesync worker caught up with the apply worker, the
> > > > > > tablesync worker synchronizes sequences associated with the target
> > > > > > table as well. One benefit would be that at the time of initial table
> > > > > > sync being completed, the table and its sequence data are consistent.
> > > >
> > > > Correction; it's not guaranteed that the sequence data and table data
> > > > are consistent even in this case since the tablesync worker could get
> > > > on-disk sequence data that might have already been updated.
> > > >
> > >
> > > The benefit of this approach is not clear to me. Our aim is to sync
> > > all sequences before the upgrade, so not sure if this helps because
> > > anyway both table values and corresponding sequences can again be
> > > out-of-sync very quickly.
> >
> > Right.
> >
> > Given that our aim is to sync all sequences before the upgrade, do we
> > need to synchronize sequences even at CREATE SUBSCRIPTION time? In
> > cases where there are a large number of sequences, synchronizing
> > sequences in addition to tables could be overhead and make less sense,
> > because sequences can again be out-of-sync quickly and typically
> > CREATE SUBSCRIPTION is not created just before the upgrade.
> >
>
> I think for the upgrade one should be creating a subscription just
> before the upgrade. Isn't something similar is done even in the
> upgrade steps you shared once [1]?

I might be missing something but in the blog post they created
subscriptions in various ways, waited for the initial table data sync
to complete, and then set the sequence values with a buffer based on
the old cluster. What I imagined with this sequence synchronization
feature is that after the initial table sync completes, we stop to
execute further transactions on the publisher, synchronize sequences
using REFRESH PUBLICATION {SEQUENCES}, and resume the application to
execute transactions on the subscriber. So a subscription would be
created just before the upgrade, but sequence synchronization would
not necessarily happen at the same time of the initial table data
synchronization.

> Typically users should get all the
> data from the publisher before the upgrade of the publisher via
> creating a subscription. Also, it would be better to keep the
> implementation of sequences close to tables wherever possible. Having
> said that, I understand your point as well and if you strongly feel
> that we don't need to sync sequences at the time of CREATE
> SUBSCRIPTION and others also don't see any problem with it then we can
> consider that as well.

I see your point that it's better to keep the implementation of
sequences close to the table one. So I agree that we can start with
this approach, and we will see how it works in practice and consider
other options later.

>
> > > >
> > >
> > > Do you mean that sync the sequences during the REFRESH PUBLICATION
> > > SEQUENCES command itself? If so, there is an argument that we can do
> > > the same during CREATE SUBSCRIPTION. It would be beneficial to keep
> > > the method to sync the sequences same for both the CREATE and REFRESH
> > > commands. I have speculated on one idea above and would be happy to
> > > see your thoughts.
> >
> > I meant that the REFRESH PUBLICATION SEQUENCES command updates all
> > sequence states in pg_subscription_rel to 'init' state, and the
> > sequence-sync worker can do the synchronization work. We use the same
> > method for both the CREATE SUBSCRIPTION and REFRESH PUBLICATION
> > {SEQUENCES} commands.
> >
>
> Marking the state as 'init' when we would have already synced the
> sequences sounds a bit odd but otherwise, this could also work if we
> accept that even if the sequences are synced and value could remain in
> 'init' state (on rollbacks).

I mean that it's just for identifying sequences that need to be
synced. With the idea of using sequence states in pg_subscription_rel,
the REFRESH PUBLICATION SEQUENCES command needs to change states to
something so that the sequence-sync worker can identify which sequence
needs to be synced. If 'init' sounds odd, we can invent a new state
for sequences, say 'needs-to-be-syned'.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

On Tue, Jun 18, 2024 at 7:30 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Fri, Jun 14, 2024 at 4:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Jun 13, 2024 at 6:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> >
> > > >
> > > > Now, say we don't want to maintain the state of sequences for initial
> > > > sync at all then after the error how will we detect if there are any
> > > > pending sequences to be synced? One possibility is that we maintain a
> > > > subscription level flag 'subsequencesync' in 'pg_subscription' to
> > > > indicate whether sequences need sync. This flag would indicate whether
> > > > to sync all the sequences in pg_susbcription_rel. This would mean that
> > > > if there is an error while syncing the sequences we will resync all
> > > > the sequences again. This could be acceptable considering the chances
> > > > of error during sequence sync are low. The benefit is that both the
> > > > REFRESH PUBLICATION SEQUENCES and CREATE SUBSCRIPTION can use the same
> > > > idea and sync all sequences without needing a new relfilenode. Users
> > > > can always refer 'subsequencesync' flag in 'pg_subscription' to see if
> > > > all the sequences are synced after executing the command.
> > >
> > > I think that REFRESH PUBLICATION {SEQUENCES} can be executed even
> > > while the sequence-sync worker is synchronizing sequences. In this
> > > case, the worker might not see new sequences added by the concurrent
> > > REFRESH PUBLICATION {SEQUENCES} command since it's already running.
> > > The worker could end up marking the subsequencesync as completed while
> > > not synchronizing these new sequences.
> > >
> >
> > This is possible but we could avoid REFRESH PUBLICATION {SEQUENCES} by
> > not allowing to change the subsequencestate during the time
> > sequence-worker is syncing the sequences. This could be restrictive
> > but there doesn't seem to be cases where user would like to
> > immediately refresh sequences after creating the subscription.
>
> I'm concerned that users would not be able to add sequences during the
> time the sequence-worker is syncing the sequences. For example,
> suppose we have 10000 sequences and execute REFRESH PUBLICATION
> {SEQUENCES} to synchronize 10000 sequences. Now if we add one sequence
> to the publication and want to synchronize it to the subscriber, we
> have to wait for the current REFRESH PUBLICATION {SEQUENCES} to
> complete, and then execute it again, synchronizing 10001 sequences,
> instead of synchronizing only the new one.
>

I see your point and it could hurt such scenarios even though they
won't be frequent. So, let's focus on our other approach of
maintaining the flag at a per-sequence level in pg_subscription_rel.

> >
> > > >
> > > > > > > Or yet another idea I came up with is that a tablesync worker will
> > > > > > > synchronize both the table and sequences owned by the table. That is,
> > > > > > > after the tablesync worker caught up with the apply worker, the
> > > > > > > tablesync worker synchronizes sequences associated with the target
> > > > > > > table as well. One benefit would be that at the time of initial table
> > > > > > > sync being completed, the table and its sequence data are consistent.
> > > > >
> > > > > Correction; it's not guaranteed that the sequence data and table data
> > > > > are consistent even in this case since the tablesync worker could get
> > > > > on-disk sequence data that might have already been updated.
> > > > >
> > > >
> > > > The benefit of this approach is not clear to me. Our aim is to sync
> > > > all sequences before the upgrade, so not sure if this helps because
> > > > anyway both table values and corresponding sequences can again be
> > > > out-of-sync very quickly.
> > >
> > > Right.
> > >
> > > Given that our aim is to sync all sequences before the upgrade, do we
> > > need to synchronize sequences even at CREATE SUBSCRIPTION time? In
> > > cases where there are a large number of sequences, synchronizing
> > > sequences in addition to tables could be overhead and make less sense,
> > > because sequences can again be out-of-sync quickly and typically
> > > CREATE SUBSCRIPTION is not created just before the upgrade.
> > >
> >
> > I think for the upgrade one should be creating a subscription just
> > before the upgrade. Isn't something similar is done even in the
> > upgrade steps you shared once [1]?
>
> I might be missing something but in the blog post they created
> subscriptions in various ways, waited for the initial table data sync
> to complete, and then set the sequence values with a buffer based on
> the old cluster. What I imagined with this sequence synchronization
> feature is that after the initial table sync completes, we stop to
> execute further transactions on the publisher, synchronize sequences
> using REFRESH PUBLICATION {SEQUENCES}, and resume the application to
> execute transactions on the subscriber. So a subscription would be
> created just before the upgrade, but sequence synchronization would
> not necessarily happen at the same time of the initial table data
> synchronization.
>

It depends on the exact steps of the upgrade. For example, if one
stops the publisher before adding sequences to a subscription either
via create subscription or alter subscription add/set command then
there won't be a need for a separate refresh but OTOH, if one follows
the steps you mentioned then the refresh would be required. As you are
okay, with syncing the sequences while creating a subscription in the
below part of the email, there is not much point in arguing about this
further.

> > Typically users should get all the
> > data from the publisher before the upgrade of the publisher via
> > creating a subscription. Also, it would be better to keep the
> > implementation of sequences close to tables wherever possible. Having
> > said that, I understand your point as well and if you strongly feel
> > that we don't need to sync sequences at the time of CREATE
> > SUBSCRIPTION and others also don't see any problem with it then we can
> > consider that as well.
>
> I see your point that it's better to keep the implementation of
> sequences close to the table one. So I agree that we can start with
> this approach, and we will see how it works in practice and consider
> other options later.
>

makes sense.

> >
> > Marking the state as 'init' when we would have already synced the
> > sequences sounds a bit odd but otherwise, this could also work if we
> > accept that even if the sequences are synced and value could remain in
> > 'init' state (on rollbacks).
>
> I mean that it's just for identifying sequences that need to be
> synced. With the idea of using sequence states in pg_subscription_rel,
> the REFRESH PUBLICATION SEQUENCES command needs to change states to
> something so that the sequence-sync worker can identify which sequence
> needs to be synced. If 'init' sounds odd, we can invent a new state
> for sequences, say 'needs-to-be-syned'.
>

Agreed and I am not sure which is better because there is a value in
keeping the state name the same for both sequences and tables. We
probably need more comments in code and doc updates to make the
behavior clear. We can start with the sequence state as 'init' for
'needs-to-be-sycned' and 'ready' for 'synced' and can change if others
feel so during the review.

--
With Regards,
Amit Kapila.

On Tue, 18 Jun 2024 at 16:10, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
>
> Agreed and I am not sure which is better because there is a value in
> keeping the state name the same for both sequences and tables. We
> probably need more comments in code and doc updates to make the
> behavior clear. We can start with the sequence state as 'init' for
> 'needs-to-be-sycned' and 'ready' for 'synced' and can change if others
> feel so during the review.

Here is a patch which does the sequence synchronization in the
following lines from the above discussion:
This commit introduces sequence synchronization during 1) creation of
subscription for initial sync of sequences 2) refresh publication to
synchronize the sequences for the newly created sequences 3) refresh
publication sequences for synchronizing all the sequences.
1) During subscription creation with CREATE SUBSCRIPTION (no syntax change):
   - The subscriber retrieves sequences associated with publications.
   - Sequences  are added in the 'init' state to the pg_subscription_rel table.
   - Sequence synchronization worker will be started if there are any
sequences to be synchronized
   - A new sequence synchronization worker handles synchronization in
batches of 100 sequences:
     a) Retrieves sequence values using pg_sequence_state from the publisher.
     b) Sets sequence values accordingly.
     c) Updates sequence state to 'READY' in pg_susbcripion_rel
     d) Commits batches of 100 synchronized sequences.
2) Refreshing sequences with ALTER SUBSCRIPTION ... REFRESH
PUBLICATION (no syntax change):
   - Stale sequences are removed from pg_subscription_rel.
   - Newly added sequences in the publisher are added in 'init' state
to pg_subscription_rel.
   - Sequence synchronization will be done by sequence sync worker as
listed in subscription creation process.
   - Sequence synchronization occurs for newly added sequences only.
3) Introduce new command ALTER SUBSCRIPTION ... REFRESH PUBLICATION
SEQUENCES  for refreshing all sequences:
   - Removes stale sequences and adds newly added sequences from the
publisher to pg_subscription_rel.
   - Resets all sequences in pg_subscription_rel to 'init' state.
   - Initiates sequence synchronization for all sequences by sequence
sync worker as listed in subscription creation process.

Regards,
Vignesh

On Wed, 19 Jun 2024 at 20:33, vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 18 Jun 2024 at 16:10, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Agreed and I am not sure which is better because there is a value in
> > keeping the state name the same for both sequences and tables. We
> > probably need more comments in code and doc updates to make the
> > behavior clear. We can start with the sequence state as 'init' for
> > 'needs-to-be-sycned' and 'ready' for 'synced' and can change if others
> > feel so during the review.
>
> Here is a patch which does the sequence synchronization in the
> following lines from the above discussion:
> This commit introduces sequence synchronization during 1) creation of
> subscription for initial sync of sequences 2) refresh publication to
> synchronize the sequences for the newly created sequences 3) refresh
> publication sequences for synchronizing all the sequences.
> 1) During subscription creation with CREATE SUBSCRIPTION (no syntax change):
>    - The subscriber retrieves sequences associated with publications.
>    - Sequences  are added in the 'init' state to the pg_subscription_rel table.
>    - Sequence synchronization worker will be started if there are any
> sequences to be synchronized
>    - A new sequence synchronization worker handles synchronization in
> batches of 100 sequences:
>      a) Retrieves sequence values using pg_sequence_state from the publisher.
>      b) Sets sequence values accordingly.
>      c) Updates sequence state to 'READY' in pg_susbcripion_rel
>      d) Commits batches of 100 synchronized sequences.
> 2) Refreshing sequences with ALTER SUBSCRIPTION ... REFRESH
> PUBLICATION (no syntax change):
>    - Stale sequences are removed from pg_subscription_rel.
>    - Newly added sequences in the publisher are added in 'init' state
> to pg_subscription_rel.
>    - Sequence synchronization will be done by sequence sync worker as
> listed in subscription creation process.
>    - Sequence synchronization occurs for newly added sequences only.
> 3) Introduce new command ALTER SUBSCRIPTION ... REFRESH PUBLICATION
> SEQUENCES  for refreshing all sequences:
>    - Removes stale sequences and adds newly added sequences from the
> publisher to pg_subscription_rel.
>    - Resets all sequences in pg_subscription_rel to 'init' state.
>    - Initiates sequence synchronization for all sequences by sequence
> sync worker as listed in subscription creation process.

Here is an updated patch with a few fixes to remove an unused
function, changed a few references of table to sequence and added one
CHECK_FOR_INTERRUPTS in the sequence sync worker loop.

Regards,
Vignesh

On Wed, Jun 19, 2024 at 8:33 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 18 Jun 2024 at 16:10, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Agreed and I am not sure which is better because there is a value in
> > keeping the state name the same for both sequences and tables. We
> > probably need more comments in code and doc updates to make the
> > behavior clear. We can start with the sequence state as 'init' for
> > 'needs-to-be-sycned' and 'ready' for 'synced' and can change if others
> > feel so during the review.
>
> Here is a patch which does the sequence synchronization in the
> following lines from the above discussion:
>

Thanks for summarizing the points discussed. I would like to confirm
whether the patch replicates new sequences that are created
implicitly/explicitly for a publication defined as ALL SEQUENCES.

--
With Regards,
Amit Kapila.

On Thu, 20 Jun 2024 at 18:45, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jun 19, 2024 at 8:33 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Tue, 18 Jun 2024 at 16:10, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Agreed and I am not sure which is better because there is a value in
> > > keeping the state name the same for both sequences and tables. We
> > > probably need more comments in code and doc updates to make the
> > > behavior clear. We can start with the sequence state as 'init' for
> > > 'needs-to-be-sycned' and 'ready' for 'synced' and can change if others
> > > feel so during the review.
> >
> > Here is a patch which does the sequence synchronization in the
> > following lines from the above discussion:
> >
>
> Thanks for summarizing the points discussed. I would like to confirm
> whether the patch replicates new sequences that are created
> implicitly/explicitly for a publication defined as ALL SEQUENCES.

Currently, FOR ALL SEQUENCES publication both explicitly created
sequences and implicitly created sequences will be synchronized during
the creation of subscriptions (using CREATE SUBSCRIPTION) and
refreshing publication sequences(using ALTER SUBSCRIPTION ... REFRESH
PUBLICATION SEQUENCES).
Therefore, the explicitly created sequence seq1:
CREATE SEQUENCE seq1;
and the implicitly created sequence seq_test2_c2_seq for seq_test2 table:
CREATE TABLE seq_test2 (c1 int, c2 SERIAL);
will both be synchronized.

Regards,
Vignesh

On Thu, 20 Jun 2024 at 18:24, vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 19 Jun 2024 at 20:33, vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Tue, 18 Jun 2024 at 16:10, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Agreed and I am not sure which is better because there is a value in
> > > keeping the state name the same for both sequences and tables. We
> > > probably need more comments in code and doc updates to make the
> > > behavior clear. We can start with the sequence state as 'init' for
> > > 'needs-to-be-sycned' and 'ready' for 'synced' and can change if others
> > > feel so during the review.
> >
> > Here is a patch which does the sequence synchronization in the
> > following lines from the above discussion:
> > This commit introduces sequence synchronization during 1) creation of
> > subscription for initial sync of sequences 2) refresh publication to
> > synchronize the sequences for the newly created sequences 3) refresh
> > publication sequences for synchronizing all the sequences.
> > 1) During subscription creation with CREATE SUBSCRIPTION (no syntax change):
> >    - The subscriber retrieves sequences associated with publications.
> >    - Sequences  are added in the 'init' state to the pg_subscription_rel table.
> >    - Sequence synchronization worker will be started if there are any
> > sequences to be synchronized
> >    - A new sequence synchronization worker handles synchronization in
> > batches of 100 sequences:
> >      a) Retrieves sequence values using pg_sequence_state from the publisher.
> >      b) Sets sequence values accordingly.
> >      c) Updates sequence state to 'READY' in pg_susbcripion_rel
> >      d) Commits batches of 100 synchronized sequences.
> > 2) Refreshing sequences with ALTER SUBSCRIPTION ... REFRESH
> > PUBLICATION (no syntax change):
> >    - Stale sequences are removed from pg_subscription_rel.
> >    - Newly added sequences in the publisher are added in 'init' state
> > to pg_subscription_rel.
> >    - Sequence synchronization will be done by sequence sync worker as
> > listed in subscription creation process.
> >    - Sequence synchronization occurs for newly added sequences only.
> > 3) Introduce new command ALTER SUBSCRIPTION ... REFRESH PUBLICATION
> > SEQUENCES  for refreshing all sequences:
> >    - Removes stale sequences and adds newly added sequences from the
> > publisher to pg_subscription_rel.
> >    - Resets all sequences in pg_subscription_rel to 'init' state.
> >    - Initiates sequence synchronization for all sequences by sequence
> > sync worker as listed in subscription creation process.
>
> Here is an updated patch with a few fixes to remove an unused
> function, changed a few references of table to sequence and added one
> CHECK_FOR_INTERRUPTS in the sequence sync worker loop.

Hi Vignesh,

I have reviewed the patches and I have following comments:

===== tablesync.c ======
1. process_syncing_sequences_for_apply can crash with:
2024-06-21 15:25:17.208 IST [3681269] LOG:  logical replication apply
worker for subscription "test1" has started
2024-06-21 15:28:10.127 IST [3682329] LOG:  logical replication
sequences synchronization worker for subscription "test1" has started
2024-06-21 15:28:10.146 IST [3682329] LOG:  logical replication
synchronization for subscription "test1", sequence "s1" has finished
2024-06-21 15:28:10.149 IST [3682329] LOG:  logical replication
synchronization for subscription "test1", sequence "s2" has finished
2024-06-21 15:28:10.149 IST [3682329] LOG:  logical replication
sequences synchronization worker for subscription "test1" has finished
2024-06-21 15:29:53.535 IST [3682767] LOG:  logical replication
sequences synchronization worker for subscription "test1" has started
TRAP: failed Assert("nestLevel > 0 && (nestLevel <= GUCNestLevel ||
(nestLevel == GUCNestLevel + 1 && !isCommit))"), File: "guc.c", Line:
2273, PID: 3682767
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (ExceptionalCondition+0xbb)[0x5b2a61861c99]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (AtEOXact_GUC+0x7b)[0x5b2a618bddfa]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (RestoreUserContext+0xc7)[0x5b2a618a6937]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (+0x1ff7dfa)[0x5b2a61115dfa]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (+0x1ff7eb4)[0x5b2a61115eb4]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (SequencesyncWorkerMain+0x33)[0x5b2a61115fe7]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (BackgroundWorkerMain+0x4ad)[0x5b2a61029cae]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (postmaster_child_launch+0x236)[0x5b2a6102fb36]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (+0x1f1d12a)[0x5b2a6103b12a]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (+0x1f1df0f)[0x5b2a6103bf0f]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (+0x1f1bf71)[0x5b2a61039f71]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (+0x1f16f73)[0x5b2a61034f73]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (PostmasterMain+0x18fb)[0x5b2a61034445]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (+0x1ab1ab8)[0x5b2a60bcfab8]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7b76bc629d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7b76bc629e40]
postgres: logical replication sequencesync worker for subscription
16389 sync 0 (_start+0x25)[0x5b2a601491a5]

Analysis:
Suppose there are two sequences (s1, s2) on publisher.
SO, during initial sync.
in loop,
+ foreach(lc, table_states_not_ready)

table_states_not_ready -> it contains both s1 and s2.
So, for s1 a sequence sync will be started. It will sync all sequences
and the sequence sync worker will exit.
Now, for s2 again a sequence sync will start. It will give the above error.

Is this loop required? Instead we can just use a bool like
'is_any_sequence_not_ready'. Thoughts?

===== sequencesync.c =====
2. function name should be 'LogicalRepSyncSequences' instead of
'LogicalRepSyncSeqeunces'

3. In function 'LogicalRepSyncSeqeunces'
    sequencerel = table_open(seqinfo->relid, RowExclusiveLock);\
    There is a extra '\' symbol

4. In function LogicalRepSyncSeqeunces:
+ ereport(LOG,
+ errmsg("logical replication synchronization for subscription \"%s\",
sequence \"%s\" has finished",
+   get_subscription_name(subid, false), RelationGetRelationName(sequencerel)));
+ table_close(sequencerel, NoLock);
+
+ currseq++;
+
+ if (currseq % MAX_SEQUENCES_SYNC_PER_BATCH == 0 || currseq ==
list_length(sequences))
+ CommitTransactionCommand();


The above message gets logged even if the changes are not committed.
Suppose the sequence worker exits before commit due to some reason.
Thought the log will show that sequence is synced, the sequence will
be in 'init' state. I think this is not desirable.
Maybe we should log the synced sequences at commit time? Thoughts?

===== General ====
5. We can use other macros like 'foreach_ptr' instead of 'foreach'

Thanks and Regards,
Shlok Kyal

On Tue, 25 Jun 2024 at 17:53, Shlok Kyal <shlok.kyal.oss@gmail.com> wrote:
>
> On Thu, 20 Jun 2024 at 18:24, vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, 19 Jun 2024 at 20:33, vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Tue, 18 Jun 2024 at 16:10, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > >
> > > > Agreed and I am not sure which is better because there is a value in
> > > > keeping the state name the same for both sequences and tables. We
> > > > probably need more comments in code and doc updates to make the
> > > > behavior clear. We can start with the sequence state as 'init' for
> > > > 'needs-to-be-sycned' and 'ready' for 'synced' and can change if others
> > > > feel so during the review.
> > >
> > > Here is a patch which does the sequence synchronization in the
> > > following lines from the above discussion:
> > > This commit introduces sequence synchronization during 1) creation of
> > > subscription for initial sync of sequences 2) refresh publication to
> > > synchronize the sequences for the newly created sequences 3) refresh
> > > publication sequences for synchronizing all the sequences.
> > > 1) During subscription creation with CREATE SUBSCRIPTION (no syntax change):
> > >    - The subscriber retrieves sequences associated with publications.
> > >    - Sequences  are added in the 'init' state to the pg_subscription_rel table.
> > >    - Sequence synchronization worker will be started if there are any
> > > sequences to be synchronized
> > >    - A new sequence synchronization worker handles synchronization in
> > > batches of 100 sequences:
> > >      a) Retrieves sequence values using pg_sequence_state from the publisher.
> > >      b) Sets sequence values accordingly.
> > >      c) Updates sequence state to 'READY' in pg_susbcripion_rel
> > >      d) Commits batches of 100 synchronized sequences.
> > > 2) Refreshing sequences with ALTER SUBSCRIPTION ... REFRESH
> > > PUBLICATION (no syntax change):
> > >    - Stale sequences are removed from pg_subscription_rel.
> > >    - Newly added sequences in the publisher are added in 'init' state
> > > to pg_subscription_rel.
> > >    - Sequence synchronization will be done by sequence sync worker as
> > > listed in subscription creation process.
> > >    - Sequence synchronization occurs for newly added sequences only.
> > > 3) Introduce new command ALTER SUBSCRIPTION ... REFRESH PUBLICATION
> > > SEQUENCES  for refreshing all sequences:
> > >    - Removes stale sequences and adds newly added sequences from the
> > > publisher to pg_subscription_rel.
> > >    - Resets all sequences in pg_subscription_rel to 'init' state.
> > >    - Initiates sequence synchronization for all sequences by sequence
> > > sync worker as listed in subscription creation process.
> >
> > Here is an updated patch with a few fixes to remove an unused
> > function, changed a few references of table to sequence and added one
> > CHECK_FOR_INTERRUPTS in the sequence sync worker loop.
>
> Hi Vignesh,
>
> I have reviewed the patches and I have following comments:
>
> ===== tablesync.c ======
> 1. process_syncing_sequences_for_apply can crash with:
> 2024-06-21 15:25:17.208 IST [3681269] LOG:  logical replication apply
> worker for subscription "test1" has started
> 2024-06-21 15:28:10.127 IST [3682329] LOG:  logical replication
> sequences synchronization worker for subscription "test1" has started
> 2024-06-21 15:28:10.146 IST [3682329] LOG:  logical replication
> synchronization for subscription "test1", sequence "s1" has finished
> 2024-06-21 15:28:10.149 IST [3682329] LOG:  logical replication
> synchronization for subscription "test1", sequence "s2" has finished
> 2024-06-21 15:28:10.149 IST [3682329] LOG:  logical replication
> sequences synchronization worker for subscription "test1" has finished
> 2024-06-21 15:29:53.535 IST [3682767] LOG:  logical replication
> sequences synchronization worker for subscription "test1" has started
> TRAP: failed Assert("nestLevel > 0 && (nestLevel <= GUCNestLevel ||
> (nestLevel == GUCNestLevel + 1 && !isCommit))"), File: "guc.c", Line:
> 2273, PID: 3682767
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (ExceptionalCondition+0xbb)[0x5b2a61861c99]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (AtEOXact_GUC+0x7b)[0x5b2a618bddfa]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (RestoreUserContext+0xc7)[0x5b2a618a6937]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (+0x1ff7dfa)[0x5b2a61115dfa]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (+0x1ff7eb4)[0x5b2a61115eb4]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (SequencesyncWorkerMain+0x33)[0x5b2a61115fe7]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (BackgroundWorkerMain+0x4ad)[0x5b2a61029cae]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (postmaster_child_launch+0x236)[0x5b2a6102fb36]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (+0x1f1d12a)[0x5b2a6103b12a]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (+0x1f1df0f)[0x5b2a6103bf0f]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (+0x1f1bf71)[0x5b2a61039f71]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (+0x1f16f73)[0x5b2a61034f73]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (PostmasterMain+0x18fb)[0x5b2a61034445]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (+0x1ab1ab8)[0x5b2a60bcfab8]
> /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7b76bc629d90]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7b76bc629e40]
> postgres: logical replication sequencesync worker for subscription
> 16389 sync 0 (_start+0x25)[0x5b2a601491a5]
>
> Analysis:
> Suppose there are two sequences (s1, s2) on publisher.
> SO, during initial sync.
> in loop,
> + foreach(lc, table_states_not_ready)
>
> table_states_not_ready -> it contains both s1 and s2.
> So, for s1 a sequence sync will be started. It will sync all sequences
> and the sequence sync worker will exit.
> Now, for s2 again a sequence sync will start. It will give the above error.
>
> Is this loop required? Instead we can just use a bool like
> 'is_any_sequence_not_ready'. Thoughts?
>
> ===== sequencesync.c =====
> 2. function name should be 'LogicalRepSyncSequences' instead of
> 'LogicalRepSyncSeqeunces'
>
> 3. In function 'LogicalRepSyncSeqeunces'
>     sequencerel = table_open(seqinfo->relid, RowExclusiveLock);\
>     There is a extra '\' symbol
>
> 4. In function LogicalRepSyncSeqeunces:
> + ereport(LOG,
> + errmsg("logical replication synchronization for subscription \"%s\",
> sequence \"%s\" has finished",
> +   get_subscription_name(subid, false), RelationGetRelationName(sequencerel)));
> + table_close(sequencerel, NoLock);
> +
> + currseq++;
> +
> + if (currseq % MAX_SEQUENCES_SYNC_PER_BATCH == 0 || currseq ==
> list_length(sequences))
> + CommitTransactionCommand();
>
>
> The above message gets logged even if the changes are not committed.
> Suppose the sequence worker exits before commit due to some reason.
> Thought the log will show that sequence is synced, the sequence will
> be in 'init' state. I think this is not desirable.
> Maybe we should log the synced sequences at commit time? Thoughts?
>
> ===== General ====
> 5. We can use other macros like 'foreach_ptr' instead of 'foreach'

Thanks for the comments, the attached patch has the fixes for the same.

Regards,
Vignesh

Here are my initial review comments for the first patch v20240625-0001.

======
General

1. Missing docs?

Section 9.17. "Sequence Manipulation Functions" [1] describes some
functions. Shouldn't your new function be documented here also?

~~~

2. Missing tests?

Shouldn't there be some test code that at least executes your new
pg_sequence_state function to verify that sane values are returned?

======
Commit Message

3.
This patch introduces new functionalities to PostgreSQL:
- pg_sequence_state allows retrieval of sequence values using LSN.
- SetSequence enables updating sequences with user-specified values.

~

3a.
I didn't understand why this says "using LSN" because IIUC 'lsn' is an
output parameter of that function. Don't you mean "... retrieval of
sequence values including LSN"?

~

3b.
Does "user-specified" make sense? Is this going to be exposed to a
user? How about just "specified"?

======
src/backend/commands/sequence.c

4. SetSequence:

+void
+SetSequence(Oid seq_relid, int64 value)

Would 'new_last_value' be a better parameter name here?

~~~

5.
This new function logic looks pretty similar to the do_setval()
function. Can you explain (maybe in the function comment) some info
about how and why it differs from that other function?

~~~

6.
I saw that RelationNeedsWAL() is called 2 times. It may make no sense,
but is it possible to assign that to a variable 1st time so you don't
need to call it 2nd time within the critical section?

~~~

NITPICK - remove junk (') char in comment

NITPICK - missing periods (.) in multi-sentence comment

~~~

7.
-read_seq_tuple(Relation rel, Buffer *buf, HeapTuple seqdatatuple)
+read_seq_tuple(Relation rel, Buffer *buf, HeapTuple seqdatatuple,
+    XLogRecPtr *lsn)

7a.
The existing parameters were described in the function comment. So,
the new 'lsn' parameter should be described here also.

~

7b.
Maybe the new parameter name should be 'lsn_res' or 'lsn_out' or
similar to emphasise that this is a returned value.

~~

NITPICK - tweaked comment. YMMV.

~~~

8. pg_sequence_state:

Should you give descriptions of the output parameters in the function
header comment? Otherwise, where are they described so called knows
what they mean?

~~~

NITPICK - /relid/seq_relid/

NITPICK - declare the variables in the same order as the output parameters

NITPICK - An alternative to the memset for nulls is just to use static
initialisation
"bool nulls[4] = {false, false, false, false};"

======
+extern void SetSequence(Oid seq_relid, int64 value);

9.
Would 'SetSequenceLastValue' be a better name for what this function is doing?

======

99.
See also my attached diff which is a top-up patch implementing those
nitpicks mentioned above. Please apply any of these that you agree
with.

======
[1] https://www.postgresql.org/docs/devel/functions-sequence.html

Kind Regards,
Peter Smith.
Fujitsu Australia

Here are some review comments for the patch v20240625-0002

======
Commit Message

1.
This commit enhances logical replication by enabling the inclusion of all
sequences in publications. This improvement facilitates seamless
synchronization of sequence data during operations such as
CREATE SUBSCRIPTION, REFRESH PUBLICATION, and REFRESH PUBLICATION SEQUENCES.

~

Isn't this description getting ahead of the functionality a bit? For
example, it talks about operations like REFRESH PUBLICATION SEQUENCES
but AFAIK that syntax does not exist just yet.

~~~

2.
The commit message should mention that you are only introducing new
syntax for "FOR ALL SEQUENCES" here, but syntax for "FOR SEQUENCE" is
being deferred to some later patch. Without such a note it is not
clear why the gram.y syntax and docs seemed only half done.

======
doc/src/sgml/ref/create_publication.sgml

3.
    <varlistentry id="sql-createpublication-params-for-all-tables">
     <term><literal>FOR ALL TABLES</literal></term>
+    <term><literal>FOR ALL SEQUENCES</literal></term>
     <listitem>
      <para>
-      Marks the publication as one that replicates changes for all tables in
-      the database, including tables created in the future.
+      Marks the publication as one that replicates changes for all tables or
+      sequences in the database, including tables created in the future.

It might be better here to keep descriptions for "ALL TABLES" and "ALL
SEQUENCES" separated, otherwise the wording does not quite seem
appropriate for sequences (e.g. where it says "including tables
created in the future").

~~~

NITPICK - missing spaces
NITPICK - removed Oxford commas since previously there were none

~~~

4.
+   If <literal>FOR TABLE</literal>, <literal>FOR ALL TABLES</literal>,
+   <literal>FOR ALL SEQUENCES</literal>,or <literal>FOR TABLES IN
SCHEMA</literal>
+   are not specified, then the publication starts out with an empty set of
+   tables.  That is useful if tables or schemas are to be added later.

It seems like "FOR ALL SEQUENCES" is out of place since it is jammed
between other clauses referring to TABLES. Would it be better to
mention SEQUENCES last in the list?

~~~

5.
+   rights on the table.  The <command>FOR ALL TABLES</command>,
+   <command>FOR ALL SEQUENCES</command>, and
    <command>FOR TABLES IN SCHEMA</command> clauses require the invoking

ditto of #4 above.

======
src/backend/catalog/pg_publication.c

GetAllSequencesPublicationRelations:

NITPICK - typo /relation/relations/

======
src/backend/commands/publicationcmds.c

6.
+ foreach(lc, stmt->for_all_objects)
+ {
+ char    *val = strVal(lfirst(lc));
+
+ if (strcmp(val, "tables") == 0)
+ for_all_tables = true;
+ else if (strcmp(val, "sequences") == 0)
+ for_all_sequences = true;
+ }

Consider the foreach_ptr macro to slightly simplify this code.
Actually, this whole logic seems cumbersome -- can’t the parser assign
flags automatically. Please see my more detailed comment #10 below
about this in gram.y

~~~

7.
  /* FOR ALL TABLES requires superuser */
- if (stmt->for_all_tables && !superuser())
+ if (for_all_tables && !superuser())
  ereport(ERROR,
  (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
  errmsg("must be superuser to create FOR ALL TABLES publication")));

+ /* FOR ALL SEQUENCES requires superuser */
+ if (for_all_sequences && !superuser())
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be superuser to create FOR ALL SEQUENCES publication")));
+

The current code is easy to read, but I wonder if it should try harder
to share common code, or at least a common translatable message like
"must be superuser to create %s publication".

~~~

8.
- else
+
+ /*
+ * If the publication might have either tables or sequences (directly or
+ * through a schema), process that.
+ */
+ if (!for_all_tables || !for_all_sequences)

I did not understand why this code cannot just say "else" like before,
because the direct or through-schema syntax cannot be specified at the
same time as "FOR ALL ...", so why is the more complicated condition
necessary? Also, the similar code in AlterPublicationOptions() was not
changed to be like this.

======
src/backend/parser/gram.y

9. comment

  *
  * CREATE PUBLICATION FOR ALL TABLES [WITH options]
  *
+ * CREATE PUBLICATION FOR ALL SEQUENCES [WITH options]
+ *
  * CREATE PUBLICATION FOR pub_obj [, ...] [WITH options]

The comment is not quite correct because actually you are allowing
simultaneous FOR ALL TABLES, SEQUENCES. It should be more like:

CREATE PUBLICATION FOR ALL pub_obj_type [,...] [WITH options]

pub_obj_type is one of:
TABLES
SEQUENCES

~~~

10.
+pub_obj_type: TABLES
+ { $$ = (Node *) makeString("tables"); }
+ | SEQUENCES
+ { $$ = (Node *) makeString("sequences"); }
+ ;
+
+pub_obj_type_list: pub_obj_type
+ { $$ = list_make1($1); }
+ | pub_obj_type_list ',' pub_obj_type
+ { $$ = lappend($1, $3); }
+ ;

IIUC the only thing you need is a flag to say if FOR ALL TABLE is in
effect and another flag to say if FOR ALL SEQUENCES is in effect. So,
It seemed clunky to build up a temporary list of "tables" or
"sequences" strings here, which is subsequently scanned by
CreatePublication to be turned back into booleans.

Can't we just change the CreatePublicationStmt field to have:

A) a 'for_all_types' bitmask instead of a list:
0x0000 means FOR ALL is not specified
0x0001 means ALL TABLES
0x0010 means ALL SEQUENCES

Or, B) have 2 boolean fields ('for_all_tables' and 'for_all_sequences')

...where the gram.y code can be written to assign the flag/s values directly?

======
src/bin/pg_dump/pg_dump.c

11.
  if (pubinfo->puballtables)
  appendPQExpBufferStr(query, " FOR ALL TABLES");

+ if (pubinfo->puballsequences)
+ appendPQExpBufferStr(query, " FOR ALL SEQUENCES");
+

Hmm. Is that correct? It looks like a possible bug, because if both
flags are true it will give invalid syntax like "FOR ALL TABLES FOR
ALL SEQUENCES" instead of "FOR ALL TABLES, SEQUENCES"

======
src/bin/pg_dump/t/002_pg_dump.pl

12.
This could also try the test scenario of both FOR ALL being
simultaneously set ("FOR ALL TABLES, SEQUENCES") to check for bugs
like the suspected one in dump.c review comment #11 above.

======
src/bin/psql/describe.c

13.
+ if (pset.sversion >= 170000)
+ printfPQExpBuffer(&buf,
+   "SELECT pubname AS \"%s\",\n"
+   "  pg_catalog.pg_get_userbyid(pubowner) AS \"%s\",\n"
+   "  puballtables AS \"%s\",\n"
+   "  puballsequences AS \"%s\",\n"
+   "  pubinsert AS \"%s\",\n"
+   "  pubupdate AS \"%s\",\n"
+   "  pubdelete AS \"%s\"",
+   gettext_noop("Name"),
+   gettext_noop("Owner"),
+   gettext_noop("All tables"),
+   gettext_noop("All sequences"),
+   gettext_noop("Inserts"),
+   gettext_noop("Updates"),
+   gettext_noop("Deletes"));
+ else
+ printfPQExpBuffer(&buf,
+   "SELECT pubname AS \"%s\",\n"
+   "  pg_catalog.pg_get_userbyid(pubowner) AS \"%s\",\n"
+   "  puballtables AS \"%s\",\n"
+   "  pubinsert AS \"%s\",\n"
+   "  pubupdate AS \"%s\",\n"
+   "  pubdelete AS \"%s\"",
+   gettext_noop("Name"),
+   gettext_noop("Owner"),
+   gettext_noop("All tables"),
+   gettext_noop("Inserts"),
+   gettext_noop("Updates"),
+   gettext_noop("Deletes"));
+

IMO this should be coded differently so that only the
"puballsequences" column is guarded by the (pset.sversion >= 170000),
and everything else is the same as before. This suggested way would
also be consistent with the existing code version checks (e.g. for
"pubtruncate" or for "pubviaroot").

~~~

NITPICK - Add blank lines
NITPICK - space in "ncols ++"

======
src/bin/psql/tab-complete.c

14.
Hmm. When I tried this, it didn't seem to be working properly.

For example "CREATE PUBLICATION pub1 FOR ALL" only completes with
"TABLES" but not "SEQUENCES".
For example "CREATE PUBLICATION pub1 FOR ALL SEQ" doesn't complete
"SEQUENCES" properly

======
src/include/catalog/pg_publication.h

NITPICK - move the extern to be adjacent to others like it.

======
src/include/nodes/parsenodes.h

15.
- bool for_all_tables; /* Special publication for all tables in db */
+ List    *for_all_objects; /* Special publication for all objects in
+ * db */
 } CreatePublicationStmt;

I felt this List logic is a bit strange. See my comment #10 in gram.y
for more details.

~~~

16.
- bool for_all_tables; /* Special publication for all tables in db */
+ List    *for_all_objects; /* Special publication for all objects in
+ * db */

Ditto comment #15 in AlterPublicationStmt

======
src/test/regress/sql/publication.sql

17.
+CREATE SEQUENCE testpub_seq0;
+CREATE SEQUENCE pub_test.testpub_seq1;
+
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_forallsequences FOR ALL SEQUENCES;
+RESET client_min_messages;
+
+SELECT pubname, puballtables, puballsequences FROM pg_publication
WHERE pubname = 'testpub_forallsequences';
+\d+ pub_test.testpub_seq1

Should you also do "\d+ tespub_seq0" here? Otherwise what was the
point of defining the seq0 sequence being in this test?

~~~

18.
Maybe there are missing test cases for different syntax combinations like:

FOR ALL TABLES, SEQUENCES
FOR ALL SEQUENCES, TABLES

Note that the current list logic of this patch even considers my
following bogus statement syntax is OK.

test_pub=# CREATE PUBLICATION pub_silly FOR ALL TABLES, SEQUENCES,
TABLES, TABLES, TABLES, SEQUENCES;
CREATE PUBLICATION
test_pub=#

======
99.
Please also refer to the attached nitpicks patch which implements all
the cosmetic issues identified above as NITPICKS.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Wed, 26 Jun 2024 at 14:41, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my initial review comments for the first patch v20240625-0001.
>
> ======
> General
>
> 6.
> I saw that RelationNeedsWAL() is called 2 times. It may make no sense,
> but is it possible to assign that to a variable 1st time so you don't
> need to call it 2nd time within the critical section?
>

I felt this is ok, we do similarly in other places also like
fill_seq_fork_with_data function in the same file.

I have fixed the other comments and merged the nitpicks changes. The
attached patch has the changes for the same.

Regards,
Vignesh

Here are my comments for patch v20240702-0001

They are all cosmetic and/or typos. Apart from these the 0001 patch LGTM.

======
doc/src/sgml/func.sgml

Section 9.17. Sequence Manipulation Functions

pg_sequence_state:
nitpick - typo /whethere/whether/
nitpick - reworded slightly using a ChatGPT suggestion. (YMMV, so it
is fine also if you prefer the current wording)

======
src/backend/commands/sequence.c

SetSequenceLastValue:
nitpick - typo in function comment /diffrent/different/

pg_sequence_state:
nitpick - function comment wording: /page LSN/the page LSN/
nitpick - moved some comment details about 'lsn_ret' into the function header
nitpick - rearranged variable assignments to have consistent order
with the values
nitpick - tweaked comments
nitpick - typo /whethere/whether/

======
99.
Please see the attached diffs patch which implements all those
nitpicks mentioned above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Mon, 1 Jul 2024 at 12:57, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for the patch v20240625-0002
>
> ======
> Commit Message
>
> 1.
> This commit enhances logical replication by enabling the inclusion of all
> sequences in publications. This improvement facilitates seamless
> synchronization of sequence data during operations such as
> CREATE SUBSCRIPTION, REFRESH PUBLICATION, and REFRESH PUBLICATION SEQUENCES.
>
> ~
>
> Isn't this description getting ahead of the functionality a bit? For
> example, it talks about operations like REFRESH PUBLICATION SEQUENCES
> but AFAIK that syntax does not exist just yet.
>
> ~~~
>
> 2.
> The commit message should mention that you are only introducing new
> syntax for "FOR ALL SEQUENCES" here, but syntax for "FOR SEQUENCE" is
> being deferred to some later patch. Without such a note it is not
> clear why the gram.y syntax and docs seemed only half done.
>
> ======
> doc/src/sgml/ref/create_publication.sgml
>
> 3.
>     <varlistentry id="sql-createpublication-params-for-all-tables">
>      <term><literal>FOR ALL TABLES</literal></term>
> +    <term><literal>FOR ALL SEQUENCES</literal></term>
>      <listitem>
>       <para>
> -      Marks the publication as one that replicates changes for all tables in
> -      the database, including tables created in the future.
> +      Marks the publication as one that replicates changes for all tables or
> +      sequences in the database, including tables created in the future.
>
> It might be better here to keep descriptions for "ALL TABLES" and "ALL
> SEQUENCES" separated, otherwise the wording does not quite seem
> appropriate for sequences (e.g. where it says "including tables
> created in the future").
>
> ~~~
>
> NITPICK - missing spaces
> NITPICK - removed Oxford commas since previously there were none
>
> ~~~
>
> 4.
> +   If <literal>FOR TABLE</literal>, <literal>FOR ALL TABLES</literal>,
> +   <literal>FOR ALL SEQUENCES</literal>,or <literal>FOR TABLES IN
> SCHEMA</literal>
> +   are not specified, then the publication starts out with an empty set of
> +   tables.  That is useful if tables or schemas are to be added later.
>
> It seems like "FOR ALL SEQUENCES" is out of place since it is jammed
> between other clauses referring to TABLES. Would it be better to
> mention SEQUENCES last in the list?
>
> ~~~
>
> 5.
> +   rights on the table.  The <command>FOR ALL TABLES</command>,
> +   <command>FOR ALL SEQUENCES</command>, and
>     <command>FOR TABLES IN SCHEMA</command> clauses require the invoking
>
> ditto of #4 above.
>
> ======
> src/backend/catalog/pg_publication.c
>
> GetAllSequencesPublicationRelations:
>
> NITPICK - typo /relation/relations/
>
> ======
> src/backend/commands/publicationcmds.c
>
> 6.
> + foreach(lc, stmt->for_all_objects)
> + {
> + char    *val = strVal(lfirst(lc));
> +
> + if (strcmp(val, "tables") == 0)
> + for_all_tables = true;
> + else if (strcmp(val, "sequences") == 0)
> + for_all_sequences = true;
> + }
>
> Consider the foreach_ptr macro to slightly simplify this code.
> Actually, this whole logic seems cumbersome -- can’t the parser assign
> flags automatically. Please see my more detailed comment #10 below
> about this in gram.y
>
> ~~~
>
> 7.
>   /* FOR ALL TABLES requires superuser */
> - if (stmt->for_all_tables && !superuser())
> + if (for_all_tables && !superuser())
>   ereport(ERROR,
>   (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
>   errmsg("must be superuser to create FOR ALL TABLES publication")));
>
> + /* FOR ALL SEQUENCES requires superuser */
> + if (for_all_sequences && !superuser())
> + ereport(ERROR,
> + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> + errmsg("must be superuser to create FOR ALL SEQUENCES publication")));
> +
>
> The current code is easy to read, but I wonder if it should try harder
> to share common code, or at least a common translatable message like
> "must be superuser to create %s publication".
>
> ~~~
>
> 8.
> - else
> +
> + /*
> + * If the publication might have either tables or sequences (directly or
> + * through a schema), process that.
> + */
> + if (!for_all_tables || !for_all_sequences)
>
> I did not understand why this code cannot just say "else" like before,
> because the direct or through-schema syntax cannot be specified at the
> same time as "FOR ALL ...", so why is the more complicated condition
> necessary? Also, the similar code in AlterPublicationOptions() was not
> changed to be like this.
>
> ======
> src/backend/parser/gram.y
>
> 9. comment
>
>   *
>   * CREATE PUBLICATION FOR ALL TABLES [WITH options]
>   *
> + * CREATE PUBLICATION FOR ALL SEQUENCES [WITH options]
> + *
>   * CREATE PUBLICATION FOR pub_obj [, ...] [WITH options]
>
> The comment is not quite correct because actually you are allowing
> simultaneous FOR ALL TABLES, SEQUENCES. It should be more like:
>
> CREATE PUBLICATION FOR ALL pub_obj_type [,...] [WITH options]
>
> pub_obj_type is one of:
> TABLES
> SEQUENCES
>
> ~~~
>
> 10.
> +pub_obj_type: TABLES
> + { $$ = (Node *) makeString("tables"); }
> + | SEQUENCES
> + { $$ = (Node *) makeString("sequences"); }
> + ;
> +
> +pub_obj_type_list: pub_obj_type
> + { $$ = list_make1($1); }
> + | pub_obj_type_list ',' pub_obj_type
> + { $$ = lappend($1, $3); }
> + ;
>
> IIUC the only thing you need is a flag to say if FOR ALL TABLE is in
> effect and another flag to say if FOR ALL SEQUENCES is in effect. So,
> It seemed clunky to build up a temporary list of "tables" or
> "sequences" strings here, which is subsequently scanned by
> CreatePublication to be turned back into booleans.
>
> Can't we just change the CreatePublicationStmt field to have:
>
> A) a 'for_all_types' bitmask instead of a list:
> 0x0000 means FOR ALL is not specified
> 0x0001 means ALL TABLES
> 0x0010 means ALL SEQUENCES
>
> Or, B) have 2 boolean fields ('for_all_tables' and 'for_all_sequences')
>
> ...where the gram.y code can be written to assign the flag/s values directly?
>
> ======
> src/bin/pg_dump/pg_dump.c
>
> 11.
>   if (pubinfo->puballtables)
>   appendPQExpBufferStr(query, " FOR ALL TABLES");
>
> + if (pubinfo->puballsequences)
> + appendPQExpBufferStr(query, " FOR ALL SEQUENCES");
> +
>
> Hmm. Is that correct? It looks like a possible bug, because if both
> flags are true it will give invalid syntax like "FOR ALL TABLES FOR
> ALL SEQUENCES" instead of "FOR ALL TABLES, SEQUENCES"
>
> ======
> src/bin/pg_dump/t/002_pg_dump.pl
>
> 12.
> This could also try the test scenario of both FOR ALL being
> simultaneously set ("FOR ALL TABLES, SEQUENCES") to check for bugs
> like the suspected one in dump.c review comment #11 above.
>
> ======
> src/bin/psql/describe.c
>
> 13.
> + if (pset.sversion >= 170000)
> + printfPQExpBuffer(&buf,
> +   "SELECT pubname AS \"%s\",\n"
> +   "  pg_catalog.pg_get_userbyid(pubowner) AS \"%s\",\n"
> +   "  puballtables AS \"%s\",\n"
> +   "  puballsequences AS \"%s\",\n"
> +   "  pubinsert AS \"%s\",\n"
> +   "  pubupdate AS \"%s\",\n"
> +   "  pubdelete AS \"%s\"",
> +   gettext_noop("Name"),
> +   gettext_noop("Owner"),
> +   gettext_noop("All tables"),
> +   gettext_noop("All sequences"),
> +   gettext_noop("Inserts"),
> +   gettext_noop("Updates"),
> +   gettext_noop("Deletes"));
> + else
> + printfPQExpBuffer(&buf,
> +   "SELECT pubname AS \"%s\",\n"
> +   "  pg_catalog.pg_get_userbyid(pubowner) AS \"%s\",\n"
> +   "  puballtables AS \"%s\",\n"
> +   "  pubinsert AS \"%s\",\n"
> +   "  pubupdate AS \"%s\",\n"
> +   "  pubdelete AS \"%s\"",
> +   gettext_noop("Name"),
> +   gettext_noop("Owner"),
> +   gettext_noop("All tables"),
> +   gettext_noop("Inserts"),
> +   gettext_noop("Updates"),
> +   gettext_noop("Deletes"));
> +
>
> IMO this should be coded differently so that only the
> "puballsequences" column is guarded by the (pset.sversion >= 170000),
> and everything else is the same as before. This suggested way would
> also be consistent with the existing code version checks (e.g. for
> "pubtruncate" or for "pubviaroot").
>
> ~~~
>
> NITPICK - Add blank lines
> NITPICK - space in "ncols ++"
>
> ======
> src/bin/psql/tab-complete.c
>
> 14.
> Hmm. When I tried this, it didn't seem to be working properly.
>
> For example "CREATE PUBLICATION pub1 FOR ALL" only completes with
> "TABLES" but not "SEQUENCES".
> For example "CREATE PUBLICATION pub1 FOR ALL SEQ" doesn't complete
> "SEQUENCES" properly
>
> ======
> src/include/catalog/pg_publication.h
>
> NITPICK - move the extern to be adjacent to others like it.
>
> ======
> src/include/nodes/parsenodes.h
>
> 15.
> - bool for_all_tables; /* Special publication for all tables in db */
> + List    *for_all_objects; /* Special publication for all objects in
> + * db */
>  } CreatePublicationStmt;
>
> I felt this List logic is a bit strange. See my comment #10 in gram.y
> for more details.
>
> ~~~
>
> 16.
> - bool for_all_tables; /* Special publication for all tables in db */
> + List    *for_all_objects; /* Special publication for all objects in
> + * db */
>
> Ditto comment #15 in AlterPublicationStmt
>
> ======
> src/test/regress/sql/publication.sql
>
> 17.
> +CREATE SEQUENCE testpub_seq0;
> +CREATE SEQUENCE pub_test.testpub_seq1;
> +
> +SET client_min_messages = 'ERROR';
> +CREATE PUBLICATION testpub_forallsequences FOR ALL SEQUENCES;
> +RESET client_min_messages;
> +
> +SELECT pubname, puballtables, puballsequences FROM pg_publication
> WHERE pubname = 'testpub_forallsequences';
> +\d+ pub_test.testpub_seq1
>
> Should you also do "\d+ tespub_seq0" here? Otherwise what was the
> point of defining the seq0 sequence being in this test?
>
> ~~~
>
> 18.
> Maybe there are missing test cases for different syntax combinations like:
>
> FOR ALL TABLES, SEQUENCES
> FOR ALL SEQUENCES, TABLES
>
> Note that the current list logic of this patch even considers my
> following bogus statement syntax is OK.
>
> test_pub=# CREATE PUBLICATION pub_silly FOR ALL TABLES, SEQUENCES,
> TABLES, TABLES, TABLES, SEQUENCES;
> CREATE PUBLICATION
> test_pub=#
>
> ======
> 99.
> Please also refer to the attached nitpicks patch which implements all
> the cosmetic issues identified above as NITPICKS.

Thank you for your feedback. I have addressed all the comments in the
attached patch.

Regards,
Vignesh

On Wed, 3 Jul 2024 at 08:24, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my comments for patch v20240702-0001
>
> They are all cosmetic and/or typos. Apart from these the 0001 patch LGTM.
>
> ======
> doc/src/sgml/func.sgml
>
> Section 9.17. Sequence Manipulation Functions
>
> pg_sequence_state:
> nitpick - typo /whethere/whether/
> nitpick - reworded slightly using a ChatGPT suggestion. (YMMV, so it
> is fine also if you prefer the current wording)
>
> ======
> src/backend/commands/sequence.c
>
> SetSequenceLastValue:
> nitpick - typo in function comment /diffrent/different/
>
> pg_sequence_state:
> nitpick - function comment wording: /page LSN/the page LSN/
> nitpick - moved some comment details about 'lsn_ret' into the function header
> nitpick - rearranged variable assignments to have consistent order
> with the values
> nitpick - tweaked comments
> nitpick - typo /whethere/whether/
>
> ======
> 99.
> Please see the attached diffs patch which implements all those
> nitpicks mentioned above.

Thank you for your feedback. I have addressed all the comments in the
v20240703 version patch attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm0mSSrvHNRnC67f0HWMpoLW9UzxGVXimhwbRtKjE7Aa-Q%40mail.gmail.com

Regards,
Vignesh

Hi Vignesh. Here are my comments for the latest patch v20240703-0001.

======
doc/src/sgml/func.sgml

nitpick - /lsn/LSN/ (all other doc pages I found use uppercase for this acronym)

======
src/backend/commands/sequence.c

nitpick - /lsn/LSN/

======
Please see attached nitpicks diff.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Thu, 4 Jul 2024 at 06:40, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh. Here are my comments for the latest patch v20240703-0001.
>
> ======
> doc/src/sgml/func.sgml
>
> nitpick - /lsn/LSN/ (all other doc pages I found use uppercase for this acronym)
>
> ======
> src/backend/commands/sequence.c
>
> nitpick - /lsn/LSN/

Thanks for the comments, the attached patch has the changes for the same.

Regards,
Vignesh

Here are my review comments for the patch v20240703-0002

======
doc/src/sgml/ref/create_publication.sgml

nitpick - consider putting the "FOR ALL SEQUENCES" para last, because
eventually when more sequence syntax is added IMO it will be better to
describe all the TABLES together, and then describe all the SEQUENCES
together.

nitpick - /synchronizing changes/synchronizes changes/

Question: Was there a reason you chose wording "synchronizes changes"
instead of having same "replicates changes" wording of FOR ALL TABLES?

======
src/backend/catalog/system_views.sql

1.
Should there be some new test for the view? Otherwise, AFAICT this
patch has no tests that will exercise the new function
pg_get_publication_sequences.

======
src/backend/commands/publicationcmds.c

2.
+ errmsg("must be superuser to create FOR ALL %s publication",
+ stmt->for_all_tables ? "TABLES" : "SEQUENCES")));

nitpick - the combined error message may be fine, but I think
translators will prefer the substitution to be the full "FOR ALL
TABLES" and "FOR ALL SEQUENCES" instead of just the keywords that are
different.

======
src/backend/parser/gram.y

3.
Some of these new things maybe could be named better?

'preprocess_allpubobjtype_list' => 'preprocess_pub_all_objtype_list'

'AllPublicationObjSpec *allpublicationobjectspec;' =>
'PublicationAllObjSpec *publicationallobjectspec;'

(I didn't include these in nitpicks diffs because you probably have
better ideas than I do for good names)

~~~

nitpick - typo in comment /SCHEMAS/SEQUENCES/

preprocess_allpubobjtype_list:
nitpick - typo /allbjects_list/all_objects_list/
nitpick - simplify /allpubob/obj/
nitpick - add underscores in the enums

======
src/bin/pg_dump/pg_dump.c

4.
+ if (pubinfo->puballtables || pubinfo->puballsequences)
+ {
+ appendPQExpBufferStr(query, " FOR ALL");
+ if (pubinfo->puballtables &&  pubinfo->puballsequences)
+ appendPQExpBufferStr(query, " TABLES, SEQUENCES");
+ else if (pubinfo->puballtables)
+ appendPQExpBufferStr(query, " TABLES");
+ else
+ appendPQExpBufferStr(query, " SEQUENCES");
+ }

nitpick - it seems over-complicated; See nitpicks diff for my suggestion.

======
src/include/nodes/parsenodes.h

nitpick - put underscores in the enum values

~~

5.
- bool for_all_tables; /* Special publication for all tables in db */
+ List    *for_all_objects; /* Special publication for all objects in
+ * db */

Is this OK? Saying "for all objects" seemed misleading.

======
src/test/regress/sql/publication.sql

nitpick - some small changes to comments, e.g. writing keywords in uppercase

~~~

6.
I asked this before in a previous review [1-#17] -- I didn't
understand the point of the sequence 'testpub_seq0' since nobody seems
to be doing anything with it. Should it just be removed? Or is there a
missing test case to use it?

~~~

7.
Other things to consider:

(I didn't include these in my attached diff)

* could use a single CREATE SEQUENCE stmt instead of multiple

* could use a single DROP PUBLICATION stmt instead of multiple

* shouldn't all publication names ideally have a 'regress_' prefix?

======
99.
Please refer to the attached nitpicks diff which has implementation
for the nitpicks cited above.

======
[1] https://www.postgresql.org/message-id/CAHut%2BPvrk75vSDkaXJVmhhZuuqQSY98btWJV%3DBMZAnyTtKRB4g%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

The latest (v20240704) patch 0001 LGTM

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi Vignesh.

After applying the v20240703-0003 patch, I was always getting errors
when running the subscription TAP tests.

# +++ tap check in src/test/subscription +++
t/001_rep_changes.pl ............... ok
t/002_types.pl ..................... ok
t/003_constraints.pl ............... ok
t/004_sync.pl ...................... ok
t/005_encoding.pl .................. ok
t/006_rewrite.pl ................... ok
t/007_ddl.pl ....................... 3/?
#   Failed test 'Alter subscription set publication throws warning for
non-existent publication'
#   at t/007_ddl.pl line 67.
Bailout called.  Further testing stopped:  pg_ctl stop failed
# Tests were run but no plan was declared and done_testing() was not seen.
FAILED--Further testing stopped: pg_ctl stop failed
make: *** [check] Error 255

~~~

The publisher log shows an Assert TRAP occurred:

2024-07-04 18:15:40.089 AEST [745] mysub1 LOG:  statement: SELECT
DISTINCT s.schemaname, s.sequencename
  FROM pg_catalog.pg_publication_sequences s
WHERE s.pubname IN ('mypub', 'non_existent_pub', 'non_existent_pub1',
'non_existent_pub2')
TRAP: failed Assert("IsA(list, OidList)"), File:
"../../../src/include/nodes/pg_list.h", Line: 323, PID: 745

~~~

A debugging backtrace looks like below:

Core was generated by `postgres: publisher: walsender postgres
postgres [local] SELECT               '.
Program terminated with signal 6, Aborted.
#0  0x00007f36f44f02c7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-260.el7_6.6.x86_64 pcre-8.32-17.el7.x86_64
(gdb) bt
#0  0x00007f36f44f02c7 in raise () from /lib64/libc.so.6
#1  0x00007f36f44f19b8 in abort () from /lib64/libc.so.6
#2  0x0000000000bb8be1 in ExceptionalCondition (conditionName=0xc7aa6c
"IsA(list, OidList)",
    fileName=0xc7aa10 "../../../src/include/nodes/pg_list.h",
lineNumber=323) at assert.c:66
#3  0x00000000005f2c57 in list_nth_oid (list=0x27948f0, n=0) at
../../../src/include/nodes/pg_list.h:323
#4  0x00000000005f5491 in pg_get_publication_sequences
(fcinfo=0x2796a00) at pg_publication.c:1334
#5  0x0000000000763d10 in ExecMakeTableFunctionResult
(setexpr=0x27b2fd8, econtext=0x27b2ef8, argContext=0x2796900,
...

Something goes wrong indexing into that 'sequences' list.

1329 funcctx = SRF_PERCALL_SETUP();
1330 sequences = (List *) funcctx->user_fctx;
1331
1332 if (funcctx->call_cntr < list_length(sequences))
1333 {
1334 Oid relid = list_nth_oid(sequences, funcctx->call_cntr);
1335
1336 SRF_RETURN_NEXT(funcctx, ObjectIdGetDatum(relid));
1337 }

======

Perhaps now it is time to create a CF entry for this thread because
the cfbot could have detected the error earlier.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Thu, 4 Jul 2024 at 12:44, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are my review comments for the patch v20240703-0002
>
> ======
> doc/src/sgml/ref/create_publication.sgml
>
>
> Question: Was there a reason you chose wording "synchronizes changes"
> instead of having same "replicates changes" wording of FOR ALL TABLES?

Since at this point we are only supporting sync of sequences, there
are no incremental changes being replicated to subscribers. I thought
synchronization is better suited here.

> ======
> src/backend/catalog/system_views.sql
>
> 1.
> Should there be some new test for the view? Otherwise, AFAICT this
> patch has no tests that will exercise the new function
> pg_get_publication_sequences.

pg_publication_sequences view uses pg_get_publication_sequences which
will be tested with 3rd patch while creating subscription/refreshing
publication sequences. I felt it is ok not to have a test here.

> 5.
> - bool for_all_tables; /* Special publication for all tables in db */
> + List    *for_all_objects; /* Special publication for all objects in
> + * db */
>
> Is this OK? Saying "for all objects" seemed misleading.

This change is not required, reverting it.

> 6.
> I asked this before in a previous review [1-#17] -- I didn't
> understand the point of the sequence 'testpub_seq0' since nobody seems
> to be doing anything with it. Should it just be removed? Or is there a
> missing test case to use it?

Since we are having all sequences published I wanted to have a
sequence in another schema also. Adding describe for it too.

> ~~~
>
> 7.
> Other things to consider:
>
> (I didn't include these in my attached diff)
>
> * could use a single CREATE SEQUENCE stmt instead of multiple

CREATE SEQUENCE does not support specifying multiple sequences in one
statement, skipping this.

The rest of the comments are fixed, the attached v20240705 version
patch has the changes for the same.

Regards,
Vignesh

On Fri, 5 Jul 2024 at 09:46, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh.
>
> After applying the v20240703-0003 patch, I was always getting errors
> when running the subscription TAP tests.
>
> # +++ tap check in src/test/subscription +++
> t/001_rep_changes.pl ............... ok
> t/002_types.pl ..................... ok
> t/003_constraints.pl ............... ok
> t/004_sync.pl ...................... ok
> t/005_encoding.pl .................. ok
> t/006_rewrite.pl ................... ok
> t/007_ddl.pl ....................... 3/?
> #   Failed test 'Alter subscription set publication throws warning for
> non-existent publication'
> #   at t/007_ddl.pl line 67.
> Bailout called.  Further testing stopped:  pg_ctl stop failed
> # Tests were run but no plan was declared and done_testing() was not seen.
> FAILED--Further testing stopped: pg_ctl stop failed
> make: *** [check] Error 255
>
> ~~~
>
> The publisher log shows an Assert TRAP occurred:
>
> 2024-07-04 18:15:40.089 AEST [745] mysub1 LOG:  statement: SELECT
> DISTINCT s.schemaname, s.sequencename
>   FROM pg_catalog.pg_publication_sequences s
> WHERE s.pubname IN ('mypub', 'non_existent_pub', 'non_existent_pub1',
> 'non_existent_pub2')
> TRAP: failed Assert("IsA(list, OidList)"), File:
> "../../../src/include/nodes/pg_list.h", Line: 323, PID: 745
>
> ~~~
>
> A debugging backtrace looks like below:
>
> Core was generated by `postgres: publisher: walsender postgres
> postgres [local] SELECT               '.
> Program terminated with signal 6, Aborted.
> #0  0x00007f36f44f02c7 in raise () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.17-260.el7_6.6.x86_64 pcre-8.32-17.el7.x86_64
> (gdb) bt
> #0  0x00007f36f44f02c7 in raise () from /lib64/libc.so.6
> #1  0x00007f36f44f19b8 in abort () from /lib64/libc.so.6
> #2  0x0000000000bb8be1 in ExceptionalCondition (conditionName=0xc7aa6c
> "IsA(list, OidList)",
>     fileName=0xc7aa10 "../../../src/include/nodes/pg_list.h",
> lineNumber=323) at assert.c:66
> #3  0x00000000005f2c57 in list_nth_oid (list=0x27948f0, n=0) at
> ../../../src/include/nodes/pg_list.h:323
> #4  0x00000000005f5491 in pg_get_publication_sequences
> (fcinfo=0x2796a00) at pg_publication.c:1334
> #5  0x0000000000763d10 in ExecMakeTableFunctionResult
> (setexpr=0x27b2fd8, econtext=0x27b2ef8, argContext=0x2796900,
> ...
>
> Something goes wrong indexing into that 'sequences' list.
>
> 1329 funcctx = SRF_PERCALL_SETUP();
> 1330 sequences = (List *) funcctx->user_fctx;
> 1331
> 1332 if (funcctx->call_cntr < list_length(sequences))
> 1333 {
> 1334 Oid relid = list_nth_oid(sequences, funcctx->call_cntr);
> 1335
> 1336 SRF_RETURN_NEXT(funcctx, ObjectIdGetDatum(relid));
> 1337 }

I was not able to reproduce this issue after several runs, but looks
like sequences need to be initialized here.

> Perhaps now it is time to create a CF entry for this thread because
> the cfbot could have detected the error earlier.

I have added a commitfest entry for the same at [1].

The v20240705 version patch attached at [2] has the change for the same.

[1] - https://commitfest.postgresql.org/49/5111/
[2] - https://www.postgresql.org/message-id/CALDaNm3WvLUesGq54JagEkbBh4CBfMoT84Rw7HjL8KML_BSzPw%40mail.gmail.com

Regards,
Vignesh

On Fri, Jul 5, 2024 at 9:58 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 4 Jul 2024 at 12:44, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > 1.
> > Should there be some new test for the view? Otherwise, AFAICT this
> > patch has no tests that will exercise the new function
> > pg_get_publication_sequences.
>
> pg_publication_sequences view uses pg_get_publication_sequences which
> will be tested with 3rd patch while creating subscription/refreshing
> publication sequences. I felt it is ok not to have a test here.
>

OTOH, if there had been such a test here then the ("sequence = NIL")
bug in patch 0002 code would have been caught earlier in patch 0002
testing instead of later in patch 0003 testing. In general, I think
each patch should be self-contained w.r.t. to testing all of its new
code, but if you think another test here is overkill then I am fine
with that too.

//////////

Meanwhile, here are my review comments for patch v20240705-0002

======
doc/src/sgml/ref/create_publication.sgml

1.
The CREATE PUBLICATION page has many examples showing many different
combinations of syntax. I think it would not hurt to add another one
showing SEQUENCES being used.

======
src/backend/commands/publicationcmds.c

2.
+ if (form->puballsequences && !superuser_arg(newOwnerId))
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("permission denied to change owner of publication \"%s\"",
+ NameStr(form->pubname)),
+ errhint("The owner of a FOR ALL SEQUENCES publication must be a
superuser.")));

You might consider combining this with the previous error in the same
way that the "FOR ALL TABLES" and "FOR ALL SEQUENCES" errors were
combined in CreatePublication. The result would be less code. But, I
also think your current code is fine, so I am just putting this out as
an idea in case you prefer it.

======
src/backend/parser/gram.y

nitpick - added a space in the comment
nitpick - changed the call order slightly because $6 comes before $7

======
src/bin/pg_dump/pg_dump.c

3. getPublications

- if (fout->remoteVersion >= 130000)
+ if (fout->remoteVersion >= 170000)

This should be 180000.

======
src/bin/psql/describe.c

4. describeOneTableDetails

+ /* print any publications */
+ if (pset.sversion >= 170000)
+ {

This should be 180000.

~~~

describeOneTableDetails:
nitpick - removed a redundant "else"
nitpick - simplified the "Publications:" header logic slightly

~~~

5. listPublications

+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+   ",\n  puballsequences AS \"%s\"",
+   gettext_noop("All sequences"));

This should be 180000.

~~~

6. describePublications

+ has_pubsequence = (pset.sversion >= 170000);

This should be 180000.

~

nitpick - remove some blank lines for consistency with nearby code

======
src/include/nodes/parsenodes.h

nitpick - minor change to comment for PublicationAllObjType
nitpick - the meanings of the enums are self-evident; I didn't think
comments were very useful

======
src/test/regress/sql/publication.sql

7.
I think it will also be helpful to arrange for a SEQUENCE to be
published by *multiple* publications. This would test that they get
listed as expected in the "Publications:" part of the "describe" (\d+)
for the sequence.

======
99.
Please also see the attached diffs patch which implements any nitpicks
mentioned above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Here are a few comments for patch v20240705-0003.

(This is a WIP. I have only looked at the docs so far.)

======
doc/src/sgml/config.sgml

nitpick - max_logical_replication_workers: /and sequence
synchornization worker/and a sequence synchornization worker/

======
doc/src/sgml/logical-replication.sgml

nitpick - max_logical_replication_workers: re-order list of workers to
be consistent with other docs 1-apply,2-parallel,3-tablesync,4-seqsync

======
doc/src/sgml/ref/alter_subscription.sgml

1.
IIUC the existing "REFRESH PUBLICATION" command will fetch and sync
all new sequences, etc., and/or remove old ones no longer in the
publication. But current docs do not say anything at all about
sequences here. It should say something about sequence behaviour.

~~~

2.
For the existing "REFRESH PUBLICATION" there is a sub-option
"copy_data=true/false". Won't this need some explanation about how it
behaves for sequences? Or will there be another option
"copy_sequences=true/false".

~~~

3.
IIUC the main difference between REFRESH PUBLICATION and REFRESH
PUBLICATION SEQUENCES is that the 2nd command will try synchronize
with all the *existing* sequences to bring them to the same point as
on the publisher, but otherwise, they are the same command. If that is
correct understanding I don't think that distinction is made very
clear in the current docs.

~~~

nitpick - the synopsis is misplaced. It should not be between ENABLE
and DISABLE. I moved it. Also, it should say "REFRESH PUBLICATION
SEQUENCES" because that is how the new syntax is defined in gram.y

nitpick - REFRESH SEQUENCES. Renamed to "REFRESH PUBLICATION
SEQUENCES". And, shouldn't "from the publisher" say "with the
publisher"?

nitpick - changed the varlistentry "id".

======
99.
Please also see the attached diffs patch which implements any nitpicks
mentioned above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi Vignesh. Here are the rest of my comments for patch v20240705-0003.

(Apologies for the length of this post; but it was unavoidable due to
this being the 1st review of a very large large 1700-line patch)

======
src/backend/catalog/pg_subscription.c

1. GetSubscriptionSequences

+/*
+ * Get the sequences for the subscription.
+ *
+ * The returned list is palloc'ed in the current memory context.
+ */

Is that comment right? The palloc seems to be done in
CacheMemoryContext, not in the current context.

~

2.
The code is very similar to the other function
GetSubscriptionRelations(). In fact I did not understand how the 2
functions know what they are returning:

E.g. how does GetSubscriptionRelations not return sequences too?
E.g. how does GetSubscriptionSequences not return relations too?

======
src/backend/commands/subscriptioncmds.c

CreateSubscription:
nitpick - put the sequence logic *after* the relations logic because
that is the order that seems used everywhere else.

~~~

3. AlterSubscription_refresh

- logicalrep_worker_stop(sub->oid, relid);
+ /* Stop the worker if relation kind is not sequence*/
+ if (relkind != RELKIND_SEQUENCE)
+ logicalrep_worker_stop(sub->oid, relid);

Can you give more reasons in the comment why skip the stop for sequence worker?

~

nitpick - period and space in the comment

~~~

4.
  for (off = 0; off < remove_rel_len; off++)
  {
  if (sub_remove_rels[off].state != SUBREL_STATE_READY &&
- sub_remove_rels[off].state != SUBREL_STATE_SYNCDONE)
+ sub_remove_rels[off].state != SUBREL_STATE_SYNCDONE &&
+ get_rel_relkind(sub_remove_rels[off].relid) != RELKIND_SEQUENCE)
  {
Would this new logic perhaps be better written as:

if (get_rel_relkind(sub_remove_rels[off].relid) == RELKIND_SEQUENCE)
  continue;

~~~

AlterSubscription_refreshsequences:
nitpick - rename AlterSubscription_refresh_sequences

~
5.
There is significant code overlap between the existing
AlterSubscription_refresh and the new function
AlterSubscription_refreshsequences. I wonder if it is better to try to
combine the logic and just pass another parameter to
AlterSubscription_refresh saying to update the existing sequences if
necessary. Particularly since the AlterSubscription_refresh is already
tweaked to work for sequences. Of course, the resulting combined
function would be large and complex, but maybe that would still be
better than having giant slabs of nearly identical cut/paste code.
Thoughts?

~~~

check_publications_origin:
nitpick - move variable declarations
~~~

fetch_sequence_list:
nitpick - change /tablelist/seqlist/
nitpick - tweak the spaces of the SQL for alignment (similar to
fetch_table_list)

~

6.
+    " WHERE s.pubname IN (");
+ first = true;
+ foreach_ptr(String, pubname, publications)
+ {
+ if (first)
+ first = false;
+ else
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_literal_cstr(pubname->sval));
+ }
+ appendStringInfoChar(&cmd, ')');

IMO this can be written much better by using get_publications_str()
function to do all this list work.

======
src/backend/replication/logical/launcher.c

7. logicalrep_worker_find

/*
 * Walks the workers array and searches for one that matches given
 * subscription id and relid.
 *
 * We are only interested in the leader apply worker or table sync worker.
 */

The above function comment (not in the patch 0003) is stale because
this AFAICT this is also going to return sequence workers if it finds
one.

~~~

8. logicalrep_sequence_sync_worker_find

+/*
+ * Walks the workers array and searches for one that matches given
+ * subscription id.
+ *
+ * We are only interested in the sequence sync worker.
+ */
+LogicalRepWorker *
+logicalrep_sequence_sync_worker_find(Oid subid, bool only_running)

There are other similar functions for walking the workers array to
search for a worker. Instead of having different functions for
different cases, wouldn't it be cleaner to combine these into a single
function, where you pass a parameter (e.g. a mask of worker types that
you are interested in finding)?

~

nitpick - declare a for loop variable 'i'

~~~

9. logicalrep_apply_worker_find

+static LogicalRepWorker *
+logicalrep_apply_worker_find(Oid subid, bool only_running)

All the other find* functions assume the lock is already held
(Assert(LWLockHeldByMe(LogicalRepWorkerLock));). But this one is
different. IMO it might be better to acquire the lock in the caller to
make all the find* functions look the same. Anyway, that will help to
combine everything into 1 "find" worker as suggested in the previous
review comment #8.

~

nitpick - declare a for loop variable 'i'
nitpick - removed unnecessary parens in condition.

~~~

10. logicalrep_worker_launch

/*----------
* Sanity checks:
* - must be valid worker type
* - tablesync workers are only ones to have relid
* - parallel apply worker is the only kind of subworker
*/

The above code-comment (not in the 0003 patch) seems stale. This
should now also mention sequence sync workers, right?

~~~

11.
- Assert(is_tablesync_worker == OidIsValid(relid));
+ Assert(is_tablesync_worker == OidIsValid(relid) ||
is_sequencesync_worker == OidIsValid(relid));

IIUC there is only a single sequence sync worker for handling all the
sequences. So, what does the 'relid' actually mean here when there are
multiple sequences?

~~~

12. logicalrep_seqsyncworker_failuretime

+/*
+ * Set the sequence sync worker failure time
+ *
+ * Called on sequence sync worker failure exit.
+ */

12a.
The comment should be improved to make it more clear that the failure
time of the sync worker information is stored with the *apply* worker.
See also other review comments in this post about this area -- perhaps
all this can be removed?

~

12b.
Curious if this had to be a separate exit handler or if may this could
have been handled by the existing logicalrep_worker_onexit handler.
See also other review comments int this post about this area --
perhaps all this can be removed?

======
.../replication/logical/sequencesync.c

13. fetch_sequence_data

13a.
The function comment has no explanation of what exactly the returned
value means. It seems like it is what you will assign as 'last_value'
on the subscriber-side.

~

13b.
Some of the table functions like this are called like
'fetch_remote_table_info()'. Maybe it is better to do similar here
(e.g. include the word "remote" in the function name).

~

14.
The reason for the addition logic "(last_value + log_cnt)" is not
obvious. I am guessing it might be related to code from
'nextval_internal' (fetch = log = fetch + SEQ_LOG_VALS;) but it is
complicated. It is unfortunate that the field 'log_cnt' seems hardly
commented anywhere at all.

Also, I am not 100% sure if I trust the logic in the first place. The
caller of this function is doing:
sequence_value = fetch_sequence_data(conn, remoteid, &lsn);
/* sets the sequence with sequence_value */
SetSequenceLastValue(RelationGetRelid(rel), sequence_value);

Won't that mean you can get to a situation where subscriber-side
result of lastval('s') can be *ahead* from lastval('s') on the
publisher? That doesn't seem good.

~~~

copy_sequence:

nitpick - ERROR message. Reword "for table..." to be more like the 2nd
error message immediately below.
nitpick - /RelationGetRelationName(rel)/relname/
nitpick - moved the Assert for 'relkind' to be nearer the assignment.

~

15.
+ /*
+ * Logical replication of sequences is based on decoding WAL records,
+ * describing the "next" state of the sequence the current state in the
+ * relfilenode is yet to reach. But during the initial sync we read the
+ * current state, so we need to reconstruct the WAL record logged when we
+ * started the current batch of sequence values.
+ *
+ * Otherwise we might get duplicate values (on subscriber) if we failed
+ * over right after the sync.
+ */
+ sequence_value = fetch_sequence_data(conn, remoteid, &lsn);
+
+ /* sets the sequence with sequence_value */
+ SetSequenceLastValue(RelationGetRelid(rel), sequence_value);

(This is related to some earlier review comment #14 above). IMO all
this tricky commentary belongs in the function header of
"fetch_sequence_data", where it should be describing that function's
return value.

~~~

LogicalRepSyncSequences:
nitpick - declare void param
nitpick indentation
nitpick - wrapping
nitpick - /sequencerel/sequence_rel/
nitpick - blank lines

~

16.
+ if (check_enable_rls(RelationGetRelid(sequencerel), InvalidOid,
false) == RLS_ENABLED)
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("user \"%s\" cannot replicate into relation with row-level
security enabled: \"%s\"",
+ GetUserNameFromId(GetUserId(), true),
+ RelationGetRelationName(sequencerel)));

This should be reworded to refer to sequences instead of relations. Maybe like:
user \"%s\" cannot replicate into sequence \"%s\" with row-level
security enabled"

~

17.
The Calculations involving the BATCH size seem a bit tricky.
e.g. in 1st place it is doing: (curr_seq % MAX_SEQUENCES_SYNC_PER_BATCH == 0)
e.g. in 2nd place it is doing: (next_seq % MAX_SEQUENCES_SYNC_PER_BATCH) == 0)

Maybe this batch logic can be simplified somehow using a bool variable
for the calculation?

Also, where does the number 100 come from? Why not 1000? Why not 10?
Why have batching at all? Maybe there should be some comment to
describe the reason and the chosen value.

~

18.
+ next_seq = curr_seq + 1;
+ if (((next_seq % MAX_SEQUENCES_SYNC_PER_BATCH) == 0) || next_seq == seq_count)
+ {
+ /* LOG all the sequences synchronized during current batch. */
+ int i = curr_seq - (curr_seq % MAX_SEQUENCES_SYNC_PER_BATCH);
+ for (; i <= curr_seq; i++)
+ {
+ SubscriptionRelState *done_seq;
+ done_seq = (SubscriptionRelState *) lfirst(list_nth_cell(sequences, i));
+ ereport(LOG,
+ errmsg("logical replication synchronization for subscription \"%s\",
sequence \"%s\" has finished",
+    get_subscription_name(subid, false), get_rel_name(done_seq->relid)));
+ }
+
+ CommitTransactionCommand();
+ }
+
+ curr_seq++;

I feel this batching logic needs more comments describing what you are
doing here.

~~~

SequencesyncWorkerMain:
nitpick - spaces in the function comment

======
src/backend/replication/logical/tablesync.c

19. finish_sync_worker

-finish_sync_worker(void)
+finish_sync_worker(bool istable)

IMO, for better readability (here and in the callers) the new
parameter should be the enum LogicalRepWorkerType. Since we have that
enum, might as well make good use of it.

~

nitpick - /sequences synchronization worker/sequence synchronization worker/
nitpick - comment tweak

~

20.
+ char relkind;
+
+ if (!started_tx)
+ {
+ StartTransactionCommand();
+ started_tx = true;
+ }
+
+ relkind = get_rel_relkind(rstate->relid);
+ if (relkind == RELKIND_SEQUENCE)
+ continue;

I am wondering is it possible to put the relkind check can come
*before* the TX code here, because in case there are *only* sequences
then maybe every would be skipped and there would have been no need
for any TX at all in the first place.

~~~

process_syncing_sequences_for_apply:

nitpick - fix typo and slight reword function header comment. Also
/last start time/last failure time/
nitpick - tweak comments
nitpick - blank lines

~

21.
+ if (!started_tx)
+ {
+ StartTransactionCommand();
+ started_tx = true;
+ }
+
+ relkind = get_rel_relkind(rstate->relid);
+ if (relkind != RELKIND_SEQUENCE || rstate->state != SUBREL_STATE_INIT)
+ continue;

Wondering (like in review comment #20) if it is possible to swap those
because maybe there was no reason for any TX if the other condition
would always continue.

~~~

22.
+ if (nsyncworkers < max_sync_workers_per_subscription)
+ {
+ TimestampTz now = GetCurrentTimestamp();
+ if (!MyLogicalRepWorker->sequencesync_failure_time ||
+ TimestampDifferenceExceeds(MyLogicalRepWorker->sequencesync_failure_time,
+    now, wal_retrieve_retry_interval))
+ {
+ MyLogicalRepWorker->sequencesync_failure_time = 0;

It seems to me that storing 'sequencesync_failure_time' logic may be
unnecessarily complicated. Can't the same "throttling" be achieved by
storing the synchronization worker 'start time' instead of 'fail
time', in which case then you won't have to mess around with
considering if the sync worker failed or just exited normally etc? You
might also be able to remove all the
logicalrep_seqsyncworker_failuretime() exit handler code.

~~~

process_syncing_tables:
nitpick - let's process tables before sequences (because all other
code is generally in this same order)
nitpick - removed some excessive comments about code that is not
supposed to happen

======
src/backend/replication/logical/worker.c

should_apply_changes_for_rel:
nitpick - IMO there were excessive comments for something that is not
going to happen

~~~

23. InitializeLogRepWorker

/*
 * Common initialization for leader apply worker, parallel apply worker and
 * tablesync worker.
 *
 * Initialize the database connection, in-memory subscription and necessary
 * config options.
 */

That comment (not part of patch 0003) is stale; it should now mention
the sequence sync worker as well, right?

~

nitpick - Tweak plural /sequences sync worker/sequence sync worker/

~~~

24. SetupApplyOrSyncWorker

/* Common function to setup the leader apply or tablesync worker. */

That comment (not part of patch 0003) is stale; it should now mention
the sequence sync worker as well, right?

======
src/include/nodes/parsenodes.h

25.
  ALTER_SUBSCRIPTION_ADD_PUBLICATION,
  ALTER_SUBSCRIPTION_DROP_PUBLICATION,
  ALTER_SUBSCRIPTION_REFRESH,
+ ALTER_SUBSCRIPTION_REFRESH_PUBLICATION_SEQUENCES,

For consistency with your new enum it would be better to also change
the existing enum name ALTER_SUBSCRIPTION_REFRESH ==>
ALTER_SUBSCRIPTION_REFRESH_PUBLICATION.

======
src/include/replication/logicalworker.h

nitpick - IMO should change the function name
/SequencesyncWorkerMain/SequenceSyncWorkerMain/, and in passing make
the same improvement to the TablesyncWorkerMain function name.

======
src/include/replication/worker_internal.h

26.
  WORKERTYPE_PARALLEL_APPLY,
+ WORKERTYPE_SEQUENCESYNC,
 } LogicalRepWorkerType;

AFAIK the enum order should not matter here so it would be better to
put the WORKERTYPE_SEQUENCESYNC directly after the
WORKERTYPE_TABLESYNC to keep the similar things together.

~

nitpick - IMO change the macro name
/isSequencesyncWorker/isSequenceSyncWorker/, and in passing make the
same improvement to the isTablesyncWorker macro name.

======
src/test/subscription/t/034_sequences.pl

nitpick - Copyright year
nitpick - Modify the "Create subscriber node" comment for consistency
nitpick - Modify comments slightly for the setup structure parts
nitpick - Add or remove various blank lines
nitpick - Since you have sequences 's2' and 's3', IMO it makes more
sense to call the original sequence 's1' instead of just 's'
nitpick - Rearrange so the CREATE PUBLICATION/SUBSCRIPTION can stay together
nitpick - Modified some comment styles to clearly delineate all the
main "TEST" scenarios
nitpick - In the REFRESH PUBLICATION test the create new sequence and
update existing can be combined (like you do in a later test).
nitpick - Changed some of the test messages for REFRESH PUBLICATION
which seemed wrong
nitpick - Added another test for 's1' in REFRESH PUBLICATION SEQUENCES
nitpick - Changed some of the test messages for REFRESH PUBLICATION
SEQUENCES which seemed wrong

~

27.
IIUC the preferred practice is to give these test object names a
'regress_' prefix.

~

28.
+# Check the data on subscriber
+$result = $node_subscriber->safe_psql(
+ 'postgres', qq(
+ SELECT * FROM s;
+));
+
+is($result, '132|0|t', 'initial test data replicated');

28a.
Maybe it is better to say "SELECT last_value, log_cnt, is_called"
instead of "SELECT *" ?
Note - this is in a couple of places.

~

28b.
Can you explain why the expected sequence value its 132, because
AFAICT you only called nextval('s') 100 times, so why isn't it 100?
My guess is that it seems to be related to code in "nextval_internal"
(fetch = log = fetch + SEQ_LOG_VALS;) but it kind of defies
expectations of the test, so if it really is correct then it needs
commentary.

Actually, I found other regression test code that deals with this:
-- log_cnt can be higher if there is a checkpoint just at the right
-- time, so just test for the expected range
SELECT last_value, log_cnt IN (31, 32) AS log_cnt_ok, is_called FROM
foo_seq_new;

Do you have to do something similar? Or is this a bug? See my other
review comments for function fetch_sequence_data in sequencesync.c

======
99.
Please also see the attached diffs patch which implements any nitpicks
mentioned above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Wed, 10 Jul 2024 at 09:34, Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Fri, Jul 5, 2024 at 9:58 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Thu, 4 Jul 2024 at 12:44, Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > 1.
> > > Should there be some new test for the view? Otherwise, AFAICT this
> > > patch has no tests that will exercise the new function
> > > pg_get_publication_sequences.
> >
> > pg_publication_sequences view uses pg_get_publication_sequences which
> > will be tested with 3rd patch while creating subscription/refreshing
> > publication sequences. I felt it is ok not to have a test here.
> >
>
> OTOH, if there had been such a test here then the ("sequence = NIL")
> bug in patch 0002 code would have been caught earlier in patch 0002
> testing instead of later in patch 0003 testing. In general, I think
> each patch should be self-contained w.r.t. to testing all of its new
> code, but if you think another test here is overkill then I am fine
> with that too.

Moved these changes to 0003 patch where it is actually required.

> //////////
>
> Meanwhile, here are my review comments for patch v20240705-0002

All the comments are fixed and the attached v20240720 version patch
has the changes for the same.

Regards,
Vignesh

On Wed, 10 Jul 2024 at 13:46, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are a few comments for patch v20240705-0003.
>
> (This is a WIP. I have only looked at the docs so far.)
>
> ======
> doc/src/sgml/config.sgml
>
> nitpick - max_logical_replication_workers: /and sequence
> synchornization worker/and a sequence synchornization worker/
>
> ======
> doc/src/sgml/logical-replication.sgml
>
> nitpick - max_logical_replication_workers: re-order list of workers to
> be consistent with other docs 1-apply,2-parallel,3-tablesync,4-seqsync
>
> ======
> doc/src/sgml/ref/alter_subscription.sgml
>
> 1.
> IIUC the existing "REFRESH PUBLICATION" command will fetch and sync
> all new sequences, etc., and/or remove old ones no longer in the
> publication. But current docs do not say anything at all about
> sequences here. It should say something about sequence behaviour.
>
> ~~~
>
> 2.
> For the existing "REFRESH PUBLICATION" there is a sub-option
> "copy_data=true/false". Won't this need some explanation about how it
> behaves for sequences? Or will there be another option
> "copy_sequences=true/false".
>
> ~~~
>
> 3.
> IIUC the main difference between REFRESH PUBLICATION and REFRESH
> PUBLICATION SEQUENCES is that the 2nd command will try synchronize
> with all the *existing* sequences to bring them to the same point as
> on the publisher, but otherwise, they are the same command. If that is
> correct understanding I don't think that distinction is made very
> clear in the current docs.
>
> ~~~
>
> nitpick - the synopsis is misplaced. It should not be between ENABLE
> and DISABLE. I moved it. Also, it should say "REFRESH PUBLICATION
> SEQUENCES" because that is how the new syntax is defined in gram.y
>
> nitpick - REFRESH SEQUENCES. Renamed to "REFRESH PUBLICATION
> SEQUENCES". And, shouldn't "from the publisher" say "with the
> publisher"?
>
> nitpick - changed the varlistentry "id".
>
> ======
> 99.
> Please also see the attached diffs patch which implements any nitpicks
> mentioned above.

All these comments are handled in the v20240720 version patch attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm2vuO7Ya4QVTZKR9jY_mkFFcE_hKUJiXx4KUknPgGFjSg%40mail.gmail.com

Regards,
Vignesh

On Fri, 12 Jul 2024 at 08:22, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh. Here are the rest of my comments for patch v20240705-0003.
> ======
> src/backend/catalog/pg_subscription.c
>
> 1. GetSubscriptionSequences
>
> +/*
> + * Get the sequences for the subscription.
> + *
> + * The returned list is palloc'ed in the current memory context.
> + */
>
> Is that comment right? The palloc seems to be done in
> CacheMemoryContext, not in the current context.

This function is removed and GetSubscriptionRelations is being used instead.

> ~
>
> 2.
> The code is very similar to the other function
> GetSubscriptionRelations(). In fact I did not understand how the 2
> functions know what they are returning:
>
> E.g. how does GetSubscriptionRelations not return sequences too?
> E.g. how does GetSubscriptionSequences not return relations too?

GetSubscriptionRelations can be used, so removed the
GetSubscriptionSequences function.

>
> 3. AlterSubscription_refresh
>
> - logicalrep_worker_stop(sub->oid, relid);
> + /* Stop the worker if relation kind is not sequence*/
> + if (relkind != RELKIND_SEQUENCE)
> + logicalrep_worker_stop(sub->oid, relid);
>
> Can you give more reasons in the comment why skip the stop for sequence worker?
>
> ~
>
> nitpick - period and space in the comment
>
> ~~~
>
> 8. logicalrep_sequence_sync_worker_find
>
> +/*
> + * Walks the workers array and searches for one that matches given
> + * subscription id.
> + *
> + * We are only interested in the sequence sync worker.
> + */
> +LogicalRepWorker *
> +logicalrep_sequence_sync_worker_find(Oid subid, bool only_running)
>
> There are other similar functions for walking the workers array to
> search for a worker. Instead of having different functions for
> different cases, wouldn't it be cleaner to combine these into a single
> function, where you pass a parameter (e.g. a mask of worker types that
> you are interested in finding)?

I will address this in a future version once the patch has become more stable.
> ~~~
>
> 11.
> - Assert(is_tablesync_worker == OidIsValid(relid));
> + Assert(is_tablesync_worker == OidIsValid(relid) ||
> is_sequencesync_worker == OidIsValid(relid));
>
> IIUC there is only a single sequence sync worker for handling all the
> sequences. So, what does the 'relid' actually mean here when there are
> multiple sequences?

Sequence sync workers will not have relid, modified the assert.

> ~~~
>
> 12. logicalrep_seqsyncworker_failuretime
> 12b.
> Curious if this had to be a separate exit handler or if may this could
> have been handled by the existing logicalrep_worker_onexit handler.
> See also other review comments int this post about this area --
> perhaps all this can be removed?

This function cannot be combined with logicalrep_worker_onexit as this
function should be called only in failure case and this exit handler
should be removed in case of success case.

This cannot be removed because of the following reason:
Consider the following situation: a sequence sync worker starts and
then encounters a failure while syncing sequences. At the same time, a
user initiates a "refresh publication sequences" operation. Given only
the start time, it's not possible to distinguish whether the sequence
sync worker failed or completed successfully. This is because the
"refresh publication sequences" operation would have re-added the
sequences, making it unclear whether the sync worker's failure or
success occurred.

>
> 14.
> The reason for the addition logic "(last_value + log_cnt)" is not
> obvious. I am guessing it might be related to code from
> 'nextval_internal' (fetch = log = fetch + SEQ_LOG_VALS;) but it is
> complicated. It is unfortunate that the field 'log_cnt' seems hardly
> commented anywhere at all.
>
> Also, I am not 100% sure if I trust the logic in the first place. The
> caller of this function is doing:
> sequence_value = fetch_sequence_data(conn, remoteid, &lsn);
> /* sets the sequence with sequence_value */
> SetSequenceLastValue(RelationGetRelid(rel), sequence_value);
>
> Won't that mean you can get to a situation where subscriber-side
> result of lastval('s') can be *ahead* from lastval('s') on the
> publisher? That doesn't seem good.

Added comments for "last_value + log_cnt"
Yes it can be ahead in subscribers. This will happen because every
change of the sequence is not wal logged. It is WAL logged once in
SEQ_LOG_VALS. This was discussed earlier and the sequence value being
ahead was ok.
https://www.postgresql.org/message-id/CA%2BTgmoaVLiKDD5vr1bzL-rxhMA37KCS_2xrqjbKVwGyqK%2BPCXQ%40mail.gmail.com

> 15.
> + /*
> + * Logical replication of sequences is based on decoding WAL records,
> + * describing the "next" state of the sequence the current state in the
> + * relfilenode is yet to reach. But during the initial sync we read the
> + * current state, so we need to reconstruct the WAL record logged when we
> + * started the current batch of sequence values.
> + *
> + * Otherwise we might get duplicate values (on subscriber) if we failed
> + * over right after the sync.
> + */
> + sequence_value = fetch_sequence_data(conn, remoteid, &lsn);
> +
> + /* sets the sequence with sequence_value */
> + SetSequenceLastValue(RelationGetRelid(rel), sequence_value);
>
> (This is related to some earlier review comment #14 above). IMO all
> this tricky commentary belongs in the function header of
> "fetch_sequence_data", where it should be describing that function's
> return value.

Moved it to fetch_sequence_data where pg_sequence_state is called to
avoid any confusion

> 17.
> Also, where does the number 100 come from? Why not 1000? Why not 10?
> Why have batching at all? Maybe there should be some comment to
> describe the reason and the chosen value.

Added a comment for this. I will do one round of testing with few
values and see if this value needs to be changed. I will share it
later.

> 20.
> + char relkind;
> +
> + if (!started_tx)
> + {
> + StartTransactionCommand();
> + started_tx = true;
> + }
> +
> + relkind = get_rel_relkind(rstate->relid);
> + if (relkind == RELKIND_SEQUENCE)
> + continue;
>
> I am wondering is it possible to put the relkind check can come
> *before* the TX code here, because in case there are *only* sequences
> then maybe every would be skipped and there would have been no need
> for any TX at all in the first place.

We need to start the transaction before calling get_rel_relkind, else
it will assert in SearchCatCacheInternal. So Skipping this.

> 21.
> + if (!started_tx)
> + {
> + StartTransactionCommand();
> + started_tx = true;
> + }
> +
> + relkind = get_rel_relkind(rstate->relid);
> + if (relkind != RELKIND_SEQUENCE || rstate->state != SUBREL_STATE_INIT)
> + continue;
>
> Wondering (like in review comment #20) if it is possible to swap those
> because maybe there was no reason for any TX if the other condition
> would always continue.

As transaction is required before calling get_rel_relkind, this cannot
be changed. So skipping this.

> ~~~
>
> 22.
> + if (nsyncworkers < max_sync_workers_per_subscription)
> + {
> + TimestampTz now = GetCurrentTimestamp();
> + if (!MyLogicalRepWorker->sequencesync_failure_time ||
> + TimestampDifferenceExceeds(MyLogicalRepWorker->sequencesync_failure_time,
> +    now, wal_retrieve_retry_interval))
> + {
> + MyLogicalRepWorker->sequencesync_failure_time = 0;
>
> It seems to me that storing 'sequencesync_failure_time' logic may be
> unnecessarily complicated. Can't the same "throttling" be achieved by
> storing the synchronization worker 'start time' instead of 'fail
> time', in which case then you won't have to mess around with
> considering if the sync worker failed or just exited normally etc? You
> might also be able to remove all the
> logicalrep_seqsyncworker_failuretime() exit handler code.

Consider the following situation: a sequence sync worker starts and
then encounters a failure while syncing sequences. At the same time, a
user initiates a "refresh publication sequences" operation. Given only
the start time, it's not possible to distinguish whether the sequence
sync worker failed or completed successfully. This is because the
"refresh publication sequences" operation would have re-added the
sequences, making it unclear whether the sync worker's failure or
success occurred.

> 28b.
> Can you explain why the expected sequence value its 132, because
> AFAICT you only called nextval('s') 100 times, so why isn't it 100?
> My guess is that it seems to be related to code in "nextval_internal"
> (fetch = log = fetch + SEQ_LOG_VALS;) but it kind of defies
> expectations of the test, so if it really is correct then it needs
> commentary.

I felt adding comments for one of the tests should be enough, So I did
not add the comment for all of the tests.

> Actually, I found other regression test code that deals with this:
> -- log_cnt can be higher if there is a checkpoint just at the right
> -- time, so just test for the expected range
> SELECT last_value, log_cnt IN (31, 32) AS log_cnt_ok, is_called FROM
> foo_seq_new;
>
> Do you have to do something similar? Or is this a bug? See my other
> review comments for function fetch_sequence_data in sequencesync.c

The comments in nextval_internal says:
 * If this is the first nextval after a checkpoint, we must force a new
 * WAL record to be written anyway, else replay starting from the
 * checkpoint would fail to advance the sequence past the logged values.
 * In this case we may as well fetch extra values.

I have increased the checkpoint for this test, so this issue will not occur.

All the other comments were fixed and the same is available in the
v20240720 version attached at [1].

[1] - https://www.postgresql.org/message-id/CALDaNm2vuO7Ya4QVTZKR9jY_mkFFcE_hKUJiXx4KUknPgGFjSg%40mail.gmail.com

Regards,
Vignesh

Here are some review comments for patch v20240720-0002.

======
1. Commit message:

1a.
The commit message is stale. It is still referring to functions and
views that have been moved to patch 0003.

1b.
"ALL SEQUENCES" is not a command. It is a clause of the CREATE
PUBLICATION command.

======
doc/src/sgml/ref/create_publication.sgml

nitpick - publication name in the example /allsequences/all_sequences/

======
src/bin/psql/describe.c

2. describeOneTableDetails

Although it's not the fault of this patch, this patch propagates the
confusion of 'result' versus 'res'. Basically, I did not understand
the need for the variable 'result'. There is already a "PGResult
*res", and unless I am mistaken we can just keep re-using that instead
of introducing a 2nd variable having almost the same name and purpose.

~

nitpick - comment case
nitpick - rearrange comment

======
src/test/regress/expected/publication.out

(see publication.sql)

======
src/test/regress/sql/publication.sql

nitpick - tweak comment

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Tue, 16 Jul 2024 at 06:00, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi,
>
> I was reading back through this thread to find out how the proposed new command for refreshing sequences,  came
about.The patch 0705 introduces a new command syntax for ALTER SUBSCRIPTION ... REFRESH SEQUENCES
 
>
> So now there are 2 forms of subscription refresh.
>
> #1. ALTER SUBSCRIPTION name REFRESH PUBLICATION [ WITH ( refresh_option [= value] [, ... ] ) ]

This is correct.

> #2. ALTER SUBSCRIPTION name REFRESH SEQUENCES

This is not correct, it is actually "ALTER SUBSCRIPTION name REFRESH
PUBLICATION SEQUENCES"

> ~~~~
>
> IMO, that separation seems complicated. It leaves many questions like:
> * It causes a bit of initial confusion. e.g. When I saw the REFRESH SEQUENCES I first assumed that was needed because
sequenceswere not covered by the existing REFRESH PUBLICATION
 
> * Why wasn't command #2 called ALTER SUBSCRIPTION REFRESH PUBLICATION SEQUENCES? E.g. missing keyword PUBLICATION. It
seemsinconsistent.
 

This is not correct, the existing implementation uses the key word
PUBLICATION, the actual syntax is:
"ALTER SUBSCRIPTION name REFRESH PUBLICATION SEQUENCES"

> * I expect sequence values can become stale pretty much immediately after command #1, so the user will want to use
command#2 anyway...
 

Yes

> * ... but if command #2 also does add/remove changed sequences same as command #1 then what benefit was there of
havingthe command #1 for sequences?
 
> * There is a separation of sequences (from tables) in command #2 but there is no separation currently possible in
command#1. It seemed inconsistent.
 

This can be enhanced if required. It is not included as of now because
I'm not sure if there is such a use case in case of tables.

> ~~~
>
> IIUC some of the goals I saw in the thread are to:
> * provide a way to fetch and refresh sequences that also keeps behaviors (e.g. copy_data etc.) consistent with the
refreshof subscription tables
 
> * provide a way to fetch and refresh *only* sequences
>
> I felt you could just enhance the existing refresh command syntax (command #1), instead of introducing a new one it
wouldbe simpler and it would still meet those same objectives.
 
>
> Synopsis:
> ALTER SUBSCRIPTION name REFRESH PUBLICATION [TABLES | SEQUENCES | ALL] [ WITH ( refresh_option [= value] [, ... ] )
]
>
> My only change is the introduction of the optional "[TABLES | SEQUENCES | ALL]" clause.
>
> I believe that can do everything your current patch does, plus more:
> * Can refresh *only* TABLES if that is what you want (current patch 0705 cannot do this)
> * Can refresh *only* SEQUENCES (same as current patch 0705 command #2)
> * Has better integration with refresh options like "copy_data" (current patch 0705 command #2 doesn't have options)
> * Existing REFRESH PUBLICATION syntax still works as-is. You can decide later what is PG18 default if the "[TABLES |
SEQUENCES| ALL]" is omitted.
 
>
> ~~~
>
> More examples using proposed syntax.
>
> ex1.
> ALTER SUBSCRIPTION sub REFRESH PUBLICATION TABLES WITH (copy_data = false)
> - same as PG17 functionality for ALTER SUBSCRIPTION sub REFRESH PUBLICATION WITH (copy_data = false)
>
> ex2.
> ALTER SUBSCRIPTION sub REFRESH PUBLICATION TABLES WITH (copy_data = true)
> - same as PG17 functionality for ALTER SUBSCRIPTION sub REFRESH PUBLICATION WITH (copy_data = true)
>
> ex3.
> ALTER SUBSCRIPTION sub REFRESH PUBLICATION SEQUENCES WITH (copy data = false)
> - this adds/removes only sequences to pg_subscription_rel but doesn't update their sequence values
>
> ex4.
> ALTER SUBSCRIPTION sub REFRESH PUBLICATION SEQUENCES WITH (copy data = true)
> - this adds/removes only sequences to pg_subscription_rel and also updates all sequence values.
> - this is equivalent behaviour of what your current 0705 patch is doing for command #2, ALTER SUBSCRIPTION sub
REFRESHSEQUENCES
 
>
> ex5.
> ALTER SUBSCRIPTION sub REFRESH PUBLICATION ALL WITH (copy_data = false)
> - this is equivalent behaviour of what your current 0705 patch is doing for command #1, ALTER SUBSCRIPTION sub
REFRESHPUBLICATION WITH (copy_data = false)
 
>
> ex6.
> ALTER SUBSCRIPTION sub REFRESH PUBLICATION ALL WITH (copy_data = true)
> - this adds/removes tables and sequences and updates all table initial data sequence values.- I think it is
equivalentto your current 0705 patch doing
 
> command #1 ALTER SUBSCRIPTION sub REFRESH PUBLICATION WITH (copy_data = true), followed by another command #2 ALTER
SUBSCRIPTIONsub REFRESH SEQUENCES
 
>
> ex7.
> ALTER SUBSCRIPTION sub REFRESH PUBLICATION SEQUENCES
> - Because default copy_data is true you do not need to specify options, so this is the same behaviour as your current
0705patch command #2, ALTER SUBSCRIPTION sub REFRESH SEQUENCES.
 

I felt ex:4 is equivalent to command #2 "ALTER SUBSCRIPTION name
REFRESH PUBLICATION SEQUENCES" and ex:3 just updates the
pg_subscription_rel. But I'm not seeing an equivalent for "ALTER
SUBSCRIPTION name REFRESH PUBLICATION with (copy_data = true)" which
will identify and remove the stale entries and add entries/synchronize
the sequences for the newly added sequences in the publisher.

Regards,
Vignesh

Hi, here are some review comments for patch v20240720-0003.

This review is a WIP. This post is only about the docs (*.sgml) of patch 0003.

======
doc/src/sgml/ref/alter_subscription.sgml

1. REFRESH PUBLICATION and copy_data
nitpicks:
- IMO the "synchronize the sequence data" info was misleading because
synchronization should only occur when copy_data=true.
- I also felt it was strange to mention pg_subscription_rel for
sequences, but not for tables. I modified this part too.
- Then I moved the information about re/synchronization of sequences
into the "copy_data" part.
- And added another link to ALTER SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES

Anyway, in summary, I have updated this page quite a lot according to
my understanding. Please take a look at the attached nitpick for my
suggestions.

nitpick - /The supported options are:/The only supported option is:/

~~~

2. REFRESH PUBLICATION SEQUENCES
nitpick - tweaked the wording
nitpicK - typo /syncronizes/synchronizes/

======
3. catalogs.sgml

IMO something is missing in Section "1.55. pg_subscription_rel".

Currently, this page only talks of relations/tables, but I think it
should mention "sequences" here too, particularly since now we are
linking to here from ALTER SUBSCRIPTION when talking about sequences.

======
99.
Please see the attached diffs patch which implements any nitpicks
mentioned above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Wed, Jul 24, 2024 at 9:17 AM Peter Smith <smithpb2250@gmail.com> wrote:
>

I had a look at patches v20240720* (considering these as the latest
one) and tried to do some basic testing (WIP). Few comments:

1)
I see 'last_value' is updated wrongly after create-sub.  Steps:

-----------
pub:
CREATE SEQUENCE myseq0 INCREMENT 5 START 100;
SELECT nextval('myseq0');
SELECT nextval('myseq0');
--last_value on pub is 105
select * from pg_sequences;
create publication pub1 for all tables, sequences;

Sub:
CREATE SEQUENCE myseq0 INCREMENT 5 START 100;
create subscription sub1 connection 'dbname=postgres host=localhost
user=shveta port=5433' publication pub1;

--check 'r' state is reached
select pc.relname, pr.srsubstate, pr.srsublsn from pg_subscription_rel
pr, pg_class pc where (pr.srrelid = pc.oid);

--check 'last_value', it shows some random value as 136
select * from pg_sequences;
-----------

2)
I can use 'for all sequences' only with 'for all tables' and can not
use it with the below. Shouldn't it be allowed?

create publication pub2 for tables in schema public, for all sequences;
create publication pub2 for table t1, for all sequences;

3)
preprocess_pub_all_objtype_list():
Do we need 'alltables_specified' and 'allsequences_specified' ? Can't
we make a repetition check using *alltables and *allsequences?

4) patch02's commit msg says : 'Additionally, a new system view,
pg_publication_sequences, has been introduced'
But it is not part of patch002.

thanks
Shveta

On Wed, 24 Jul 2024 at 09:17, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are some review comments for patch v20240720-0003.
>
> This review is a WIP. This post is only about the docs (*.sgml) of patch 0003.
>
> 3. catalogs.sgml
>
> IMO something is missing in Section "1.55. pg_subscription_rel".
>
> Currently, this page only talks of relations/tables, but I think it
> should mention "sequences" here too, particularly since now we are
> linking to here from ALTER SUBSCRIPTION when talking about sequences.

Modified it to mention sequences too.

I have merged the rest of the nitpicks suggested by you.
The attached v20240724 version patch has the changes for the same.

Regards,
Vignesh

On Tue, 23 Jul 2024 at 11:03, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for patch v20240720-0002.
>
> ======
> 1. Commit message:
>
> 1a.
> The commit message is stale. It is still referring to functions and
> views that have been moved to patch 0003.

Modified

> 1b.
> "ALL SEQUENCES" is not a command. It is a clause of the CREATE
> PUBLICATION command.

Modified

> src/bin/psql/describe.c
>
> 2. describeOneTableDetails
>
> Although it's not the fault of this patch, this patch propagates the
> confusion of 'result' versus 'res'. Basically, I did not understand
> the need for the variable 'result'. There is already a "PGResult
> *res", and unless I am mistaken we can just keep re-using that instead
> of introducing a 2nd variable having almost the same name and purpose.

This is intentional, we cannot clear res as it will be used in many
places of printTable like in
printTable->print_aligned_text->pg_wcssize which was earlier stored
from printTableAddCell calls.

The rest of the nitpicks comments were merged.
The v20240724 version patch attached at [1] has the changes for the same.

[1] - https://www.postgresql.org/message-id/CALDaNm1uncevCSMqo5Nk%3DtqqV_o3KNH_jwp8URiGop_nPC8BTg%40mail.gmail.com

Regards,
Vignesh

On Wed, Jul 24, 2024 at 11:52 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Wed, Jul 24, 2024 at 9:17 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
>
> I had a look at patches v20240720* (considering these as the latest
> one) and tried to do some basic testing (WIP). Few comments:
>
> 1)
> I see 'last_value' is updated wrongly after create-sub.  Steps:
>
> -----------
> pub:
> CREATE SEQUENCE myseq0 INCREMENT 5 START 100;
> SELECT nextval('myseq0');
> SELECT nextval('myseq0');
> --last_value on pub is 105
> select * from pg_sequences;
> create publication pub1 for all tables, sequences;
>
> Sub:
> CREATE SEQUENCE myseq0 INCREMENT 5 START 100;
> create subscription sub1 connection 'dbname=postgres host=localhost
> user=shveta port=5433' publication pub1;
>
> --check 'r' state is reached
> select pc.relname, pr.srsubstate, pr.srsublsn from pg_subscription_rel
> pr, pg_class pc where (pr.srrelid = pc.oid);
>
> --check 'last_value', it shows some random value as 136
> select * from pg_sequences;

Okay, I see that in fetch_remote_sequence_data(), we are inserting
'last_value + log_cnt' fetched from remote as 'last_val' on subscriber
and thus leading to above behaviour. I did not understand why this is
done? This may result into issue when we insert data into a table with
identity column on subscriber (whose internal sequence is replicated);
the identity column in this case will end up having wrong value.

thanks
Shveta

On Wed, 24 Jul 2024 at 11:53, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Wed, Jul 24, 2024 at 9:17 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
>
> I had a look at patches v20240720* (considering these as the latest
> one) and tried to do some basic testing (WIP). Few comments:
>
> 1)
> I see 'last_value' is updated wrongly after create-sub.  Steps:
>
> -----------
> pub:
> CREATE SEQUENCE myseq0 INCREMENT 5 START 100;
> SELECT nextval('myseq0');
> SELECT nextval('myseq0');
> --last_value on pub is 105
> select * from pg_sequences;
> create publication pub1 for all tables, sequences;
>
> Sub:
> CREATE SEQUENCE myseq0 INCREMENT 5 START 100;
> create subscription sub1 connection 'dbname=postgres host=localhost
> user=shveta port=5433' publication pub1;
>
> --check 'r' state is reached
> select pc.relname, pr.srsubstate, pr.srsublsn from pg_subscription_rel
> pr, pg_class pc where (pr.srrelid = pc.oid);
>
> --check 'last_value', it shows some random value as 136
> select * from pg_sequences;

Earlier I was setting sequence value with the value of publisher +
log_cnt, that is why the difference is there. On further thinking
since we are not supporting incremental replication of sequences, so
no plugin usage is involved which requires the special decoding last
value and log_count. I felt we can use the exact sequence last value
and log count to generate the similar sequence value. So Now I have
changed it to get the last_value and log_count from the publisher and
set it to the same values.

>
> 2)
> I can use 'for all sequences' only with 'for all tables' and can not
> use it with the below. Shouldn't it be allowed?
>
> create publication pub2 for tables in schema public, for all sequences;
> create publication pub2 for table t1, for all sequences;

I feel this can be added as part of a later version while supporting
"add/drop/set sequence and add/drop/set sequences in schema" once the
patch is stable.

> 3)
> preprocess_pub_all_objtype_list():
> Do we need 'alltables_specified' and 'allsequences_specified' ? Can't
> we make a repetition check using *alltables and *allsequences?

Modified

> 4) patch02's commit msg says : 'Additionally, a new system view,
> pg_publication_sequences, has been introduced'
> But it is not part of patch002.

This is removed now

The attached v20240725 version patch has the changes for the same.

Regards,
Vignesh

On Thu, Jul 25, 2024 at 9:06 AM vignesh C <vignesh21@gmail.com> wrote:
>
> The attached v20240725 version patch has the changes for the same.

Thank You for addressing the comments. Please review below issues:

1) Sub ahead of pub due to wrong initial sync of last_value for
non-incremented sequences. Steps at [1]
2) Sequence's min value is not honored on sub during replication. Steps at [2]

[1]:
-----------
on PUB:
CREATE SEQUENCE myseq001 INCREMENT 5 START 100;
SELECT * from pg_sequences; -->shows last_val as NULL

on SUB:
CREATE SEQUENCE myseq001 INCREMENT 5 START 100;
SELECT * from pg_sequences; -->correctly shows last_val as NULL
ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
SELECT * from pg_sequences;  -->wrongly updates last_val to 100; it is
still NULL on Pub.

Thus , SELECT nextval('myseq001') on pub gives 100, while on sub gives 105.
-----------


[2]:
-----------
Pub:
CREATE SEQUENCE myseq0 INCREMENT 5 START 10;
SELECT * from pg_sequences;

Sub:
CREATE SEQUENCE myseq0 INCREMENT 5 MINVALUE 100;

Pub:
SELECT nextval('myseq0');
SELECT nextval('myseq0');

Sub:
ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
--check 'last_value', it is 15 while min_value is 100
SELECT * from pg_sequences;
-----------

thanks
Shveta

Hi, here are more review comments for patch v20240720-0003.

======
src/backend/catalog/pg_subscription.c

(Numbers are starting at #4 because this is a continuation of the docs review)

4. GetSubscriptionRelations

nitpick - rearranged the function header comment

~

5.
TBH, I'm thinking that just passing 2 parameters:
- bool get_tables
- bool get_sequences
where one or both can be true, would have resulted in simpler code,
instead of introducing this new enum SubscriptionRelKind.

~

6.
The 'not_all_relations' parameter/logic feels really awkward. IMO it
needs a better name and reverse the meaning to remove all the "nots".

For example, commenting it and calling it like below could be much simpler.

'all_relations'
If returning sequences, if all_relations=true get all sequences,
otherwise only get sequences that are in 'init' state.
If returning tables, if all_relation=true get all tables, otherwise
only get tables that have not reached 'READY' state.

======
src/backend/commands/subscriptioncmds.c

AlterSubscription_refresh:

nitpick - this function comment is difficult to understand. I've
rearranged it a bit but it could still do with some further
improvement.
nitpick - move some code comments
nitpick - I adjusted the "stop worker" comment slightly. Please check
it is still correct.
nitpick - add a blank line

~

7.
The logic seems over-complicated. For example, why is the sequence
list *always* fetched, but the tables list is only sometimes fetched?
Furthermore, this 'refresh_all_sequences' parameter seems to have a
strange interference with tables (e.g. even though it is possible to
refresh all tables and sequences at the same time). It is as if the
meaning is 'refresh_publication_sequences' yet it is not called that
(???)

These gripes may be related to my other thread [1] about the new ALTER
syntax. (I feel that there should be the ability to refresh ALL TABLES
or ALL SEQUENCES independently if the user wants to). IIUC, it would
simplify this function logic as well as being more flexible. Anyway, I
will leave the discussion about syntax to that other thread.

~

8.
+ if (relkind != RELKIND_SEQUENCE)
+ logicalrep_worker_stop(sub->oid, relid);

  /*
  * For READY state, we would have already dropped the
  * tablesync origin.
  */
- if (state != SUBREL_STATE_READY)
+ if (state != SUBREL_STATE_READY && relkind != RELKIND_SEQUENCE)

It might be better to have a single "if (relkind != RELKIND_SEQUENCE)"
here and combine both of these codes under that.

~

9.
  ereport(DEBUG1,
- (errmsg_internal("table \"%s.%s\" removed from subscription \"%s\"",
+ (errmsg_internal("%s \"%s.%s\" removed from subscription \"%s\"",
+ get_namespace_name(get_rel_namespace(relid)),
+ get_rel_name(relid),
+ sub->name,
+ get_rel_relkind(relid) == RELKIND_SEQUENCE ? "sequence" : "table")));

IIUC prior conDitions mean get_rel_relkind(relid) == RELKIND_SEQUENCE
will be impossible here.

~~~

10. AlterSubscription

+ PreventInTransactionBlock(isTopLevel, "ALTER SUBSCRIPTION ...
REFRESH PUBLICATION SEQUENCES");

IIUC the docs page for ALTER SUBSCRIPTION was missing this information
about "REFRESH PUBLICATION SEQUENCES" in transactions. Docs need more
updates.

======
src/backend/replication/logical/launcher.c

logicalrep_worker_find:
nitpick - tweak comment to say "or" instead of "and"

~~~

11.
+/*
+ * Return the pid of the apply worker for one that matches given
+ * subscription id.
+ */
+static LogicalRepWorker *
+logicalrep_apply_worker_find(Oid subid, bool only_running)

The function comment is wrong. This is not returning a PID.

~~~

12.
+ if (is_sequencesync_worker)
+ Assert(!OidIsValid(relid));

Should we the Assert to something more like:
Assert(!is_sequencesync_worker || !OidIsValid(relid));

Otherwise, in NODEBUG current code will compile into an empty
condition statement, which is a bit odd.

~~~

logicalrep_seqsyncworker_failuretime:
nitpick - tweak function comment
nitpick - add blank line

======
.../replication/logical/sequencesync.c

13. fetch_remote_sequence_data

The "current state" mentioned in the function comment is a bit vague.
Can't tell from this comment what it is returning without looking
deeper into the function code.

~

nitpick - typo "scenarios" in comment

~~~

copy_sequence:
nitpick - typo "withe" in function comment
nitpick - typo /retreived/retrieved/
nitpick - add/remove blank lines

~~~

LogicalRepSyncSequences:
nitpick - move a comment.
nitpick - remove blank line

14.
+ /*
+ * Verify whether the current batch of sequences is synchronized or if
+ * there are no remaining sequences to synchronize.
+ */
+ if ((((curr_seq + 1) % MAX_SEQUENCES_SYNC_PER_BATCH) == 0) ||
+ (curr_seq + 1) == seq_count)


All this "curr_seq + 1" maths seems unnecessarily tricky. Can't we
just increment the cur_seq? before this calculation?

~

nitpick - simplify the comment about batching
nitpick - added a comment to the commit

======
src/backend/replication/logical/tablesync.c

finish_sync_worker:
nitpick - added an Assert so the if/else is less risky.
nitpick - modify the comment about failure time when it is a clean exit

~~~

15. process_syncing_sequences_for_apply

+ /* We need up-to-date sync state info for subscription sequences here. */
+ FetchTableStates(&started_tx, SUB_REL_KIND_ALL);

Should that say SUB_REL_KIND_SEQUENCE?

~

16.
+ /*
+ * If there are free sync worker slot(s), start a new sequence
+ * sync worker, and break from the loop.
+ */
+ if (nsyncworkers < max_sync_workers_per_subscription)

Should this "if" have some "else" code to log a warning if we have run
out of free workers? Otherwise, how will the user know that the system
may need tuning?

~~~

17. FetchTableStates

  /* Fetch all non-ready tables. */
- rstates = GetSubscriptionRelations(MySubscription->oid, true);
+ rstates = GetSubscriptionRelations(MySubscription->oid, rel_type, true);

This feels risky. IMO there needs to be some prior Assert about the
rel_type. For example, if it happened to be SUB_REL_KIND_SEQUENCE then
this function code doesn't seem to make sense.

~~~

======
src/backend/replication/logical/worker.c

18. SetupApplyOrSyncWorker

+
+ if (isSequenceSyncWorker(MyLogicalRepWorker))
+ before_shmem_exit(logicalrep_seqsyncworker_failuretime, (Datum) 0);

Probably that should be using macro am_sequencesync_worker(), right?

======
src/include/catalog/pg_subscription_rel.h

19.
+typedef enum
+{
+ SUB_REL_KIND_TABLE,
+ SUB_REL_KIND_SEQUENCE,
+ SUB_REL_KIND_ALL,
+} SubscriptionRelKind;
+

I was not sure how helpful this is; it might not be needed. e.g. see
review comment for GetSubscriptionRelations

~~~

20.
+extern List *GetSubscriptionRelations(Oid subid, SubscriptionRelKind reltype,
+   bool not_ready);

There is a mismatch with the ‘not_ready’ parameter name here and in
the function implementation

======
src/test/subscription/t/034_sequences.pl

nitpick - removed a blank line

======
99.
Please also see the attached diffs patch which implements all the
nitpicks mentioned above.

======
[1] syntax -
https://www.postgresql.org/message-id/CAHut%2BPuFH1OCj-P1UKoRQE2X4-0zMG%2BN1V7jdn%3DtOQV4RNbAbw%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

On Thu, Jul 25, 2024 at 12:08 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Thu, Jul 25, 2024 at 9:06 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > The attached v20240725 version patch has the changes for the same.
>
> Thank You for addressing the comments. Please review below issues:
>
> 1) Sub ahead of pub due to wrong initial sync of last_value for
> non-incremented sequences. Steps at [1]
> 2) Sequence's min value is not honored on sub during replication. Steps at [2]

One more issue:
3)  Sequence datatype's range is not honored on sub during
replication, while it is honored for tables.


Behaviour for tables:
---------------------
Pub: create table tab1( i integer);
Sub: create table tab1( i smallint);

Pub: insert into tab1 values(generate_series(1, 32768));

Error on sub:
2024-07-25 10:38:06.446 IST [178680] ERROR:  value "32768" is out of
range for type smallint

---------------------
Behaviour for sequences:
---------------------

Pub:
CREATE SEQUENCE myseq_i as integer INCREMENT 10000 START 1;

Sub:
CREATE SEQUENCE myseq_i as smallint INCREMENT 10000 START 1;

Pub:
SELECT nextval('myseq_i');
SELECT nextval('myseq_i');
SELECT nextval('myseq_i');
SELECT nextval('myseq_i');
SELECT nextval('myseq_i'); -->brings value to 40001

Sub:
ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
SELECT * from pg_sequences;  -->last_val reached till 40001, while the
range is till  32767.

thanks
Shveta

Here are some review comments for latest patch v20240725-0002

======
doc/src/sgml/ref/create_publication.sgml

nitpick - tweak to the description of the example.

======
src/backend/parser/gram.y

preprocess_pub_all_objtype_list:
nitpick - typo "allbjects_list"
nitpick - reword function header
nitpick - /alltables/all_tables/
nitpick - /allsequences/all_sequences/
nitpick - I think code is safe as-is because makeNode internally does
palloc0, but OTOH adding Assert would be nicer just to remove any
doubts.

======
src/bin/psql/describe.c

1.
+ /* Print any publications */
+ if (pset.sversion >= 180000)
+ {
+ int tuples = 0;

No need to assign value 0 here, because this will be unconditionally
assigned before use anyway.

~~~~

2. describePublications

  has_pubviaroot = (pset.sversion >= 130000);
+ has_pubsequence = (pset.sversion >= 18000);

That's a bug! Should be 180000, not 18000.

======

And, please see the attached diffs patch, which implements the
nitpicks mentioned above.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi Vignesh,

There are still pending changes from my previous review of the
0720-0003 patch [1], but here are some new review comments for your
latest patch v20240525-0003.

======
doc/src/sgml/catalogs.sgml

nitpick - fix plurals and tweak the description.

~~~

1.
   <para>
-   This catalog only contains tables known to the subscription after running
-   either <link linkend="sql-createsubscription"><command>CREATE
SUBSCRIPTION</command></link> or
-   <link linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION
... REFRESH
+   This catalog only contains tables and sequences known to the subscription
+   after running either
+   <link linkend="sql-createsubscription"><command>CREATE
SUBSCRIPTION</command></link>
+   or <link linkend="sql-altersubscription"><command>ALTER
SUBSCRIPTION ... REFRESH
    PUBLICATION</command></link>.
   </para>

Shouldn't this mention "REFRESH PUBLICATION SEQUENCES" too?

======
src/backend/commands/sequence.c

SetSequenceLastValue:
nitpick - maybe change: /log_cnt/new_log_cnt/ for consistency with the
other parameter, and to emphasise the old log_cnt is overwritten

======
src/backend/replication/logical/sequencesync.c

2.
+/*
+ * fetch_remote_sequence_data
+ *
+ * Fetch sequence data (current state) from the remote node, including
+ * the latest sequence value from the publisher and the Page LSN for the
+ * sequence.
+ */
+static int64
+fetch_remote_sequence_data(WalReceiverConn *conn, Oid remoteid,
+    int64 *log_cnt, XLogRecPtr *lsn)

2a.
Now you are also returning the 'log_cnt' but that is not mentioned by
the function comment.

~

2b.
Is it better to name these returned by-ref ptrs like 'ret_log_cnt',
and 'ret_lsn' to emphasise they are output variables? YMMV.

~~~

3.
+ /* Process the sequence. */
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))

This will have one-and-only-one tuple for the discovered sequence,
won't it? So, why is this a while loop?

======
src/include/commands/sequence.h

nitpick - maybe change: /log_cnt/new_log_cnt/ (same as earlier in this post)

======
src/test/subscription/t/034_sequences.pl

4.
Q. Should we be suspicious that log_cnt changes from '32' to '31', or
is there a valid explanation? It smells like some calculation is
off-by-one, but without debugging I can't tell if it is right or
wrong.

======
Please also see the attached diffs patch, which implements the
nitpicks mentioned above.

======
[1] 0720-0003 review -
https://www.postgresql.org/message-id/CAHut%2BPsfsfzyBrmo8E43qFMp9_bmen2tuCsNYN8sX%3Dfa86SdfA%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

On Thu, 25 Jul 2024 at 12:54, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi, here are more review comments for patch v20240720-0003.
> 7.
> The logic seems over-complicated. For example, why is the sequence
> list *always* fetched, but the tables list is only sometimes fetched?
> Furthermore, this 'refresh_all_sequences' parameter seems to have a
> strange interference with tables (e.g. even though it is possible to
> refresh all tables and sequences at the same time). It is as if the
> meaning is 'refresh_publication_sequences' yet it is not called that
> (???)
>
> These gripes may be related to my other thread [1] about the new ALTER
> syntax. (I feel that there should be the ability to refresh ALL TABLES
> or ALL SEQUENCES independently if the user wants to). IIUC, it would
> simplify this function logic as well as being more flexible. Anyway, I
> will leave the discussion about syntax to that other thread.

1) ALTER SUBCRIPTION ... REFRESH PUBLICATION
This command will refresh both tables and sequences. It will remove
stale tables and sequences and include newly added tables and
sequences.
2) ALTER SUBCRIPTION ... REFRESH PUBLICATION SEQUENCES
This command will refresh only sequences. It will remove stale
sequences and synchronize all sequences including the existing
sequences.
So the table will be fetched only for the first command.
I have changed refresh_publication_sequences parameter to tables,
sequences, all_relations with this the function should be easier to
understand and remove any confusions.

> ~
>
> 9.
>   ereport(DEBUG1,
> - (errmsg_internal("table \"%s.%s\" removed from subscription \"%s\"",
> + (errmsg_internal("%s \"%s.%s\" removed from subscription \"%s\"",
> + get_namespace_name(get_rel_namespace(relid)),
> + get_rel_name(relid),
> + sub->name,
> + get_rel_relkind(relid) == RELKIND_SEQUENCE ? "sequence" : "table")));
>
> IIUC prior conDitions mean get_rel_relkind(relid) == RELKIND_SEQUENCE
> will be impossible here.

Consider a scenario where logical replication is setup with sequences
seq1, seq2.
Now drop sequence seq1 and do "ALTER SUBSCRIPTION sub REFRESH PUBLICATION"
It will hit this code to generate the log:
DEBUG:  sequence "public.seq1" removed from subscription "test1"

> ======
> .../replication/logical/sequencesync.c
>
> 13. fetch_remote_sequence_data
>
> The "current state" mentioned in the function comment is a bit vague.
> Can't tell from this comment what it is returning without looking
> deeper into the function code.

Added more comments to  clarify.

> ~~~
>
> 15. process_syncing_sequences_for_apply
>
> + /* We need up-to-date sync state info for subscription sequences here. */
> + FetchTableStates(&started_tx, SUB_REL_KIND_ALL);
>
> Should that say SUB_REL_KIND_SEQUENCE?

We cannot pass SUB_REL_KIND_SEQUENCE here because the
pg_subscription_rel table is shared between sequences and tables. As
changes to either sequences or relations can affect the validity of
relation states, we update both table_states_not_ready and
sequence_states_not_ready simultaneously to ensure consistency, rather
than updating them separately. I have removed the relation kind
parameter now. Fetch tables is called to fetch all tables and
sequences before calling process_syncing_tables_for_apply and
process_syncing_sequences_for_apply now.

> ~
>
> 16.
> + /*
> + * If there are free sync worker slot(s), start a new sequence
> + * sync worker, and break from the loop.
> + */
> + if (nsyncworkers < max_sync_workers_per_subscription)
>
> Should this "if" have some "else" code to log a warning if we have run
> out of free workers? Otherwise, how will the user know that the system
> may need tuning?

I felt no need to log here else we will get a lot of log messages
which might not be required. Similar logic is used for tablesync to in
process_syncing_tables_for_apply.

> ~~~
>
> 17. FetchTableStates
>
>   /* Fetch all non-ready tables. */
> - rstates = GetSubscriptionRelations(MySubscription->oid, true);
> + rstates = GetSubscriptionRelations(MySubscription->oid, rel_type, true);
>
> This feels risky. IMO there needs to be some prior Assert about the
> rel_type. For example, if it happened to be SUB_REL_KIND_SEQUENCE then
> this function code doesn't seem to make sense.

The pg_subscription_rel table is shared between sequences and tables.
As changes to either sequences or relations can affect the validity of
relation states, we update both table_states_not_ready and
sequence_states_not_ready simultaneously to ensure consistency, rather
than updating them separately. This will update both tables and
sequences that should be synced.

The rest of the comments are fixed. The attached v20240729 version
patch has the changes for the same.

Regards,
Vignesh

On Fri, 26 Jul 2024 at 08:04, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for latest patch v20240725-0002
>
> ======
> doc/src/sgml/ref/create_publication.sgml
>
> nitpick - tweak to the description of the example.
>
> ======
> src/backend/parser/gram.y
>
> preprocess_pub_all_objtype_list:
> nitpick - typo "allbjects_list"
> nitpick - reword function header
> nitpick - /alltables/all_tables/
> nitpick - /allsequences/all_sequences/
> nitpick - I think code is safe as-is because makeNode internally does
> palloc0, but OTOH adding Assert would be nicer just to remove any
> doubts.
>
> ======
> src/bin/psql/describe.c
>
> 1.
> + /* Print any publications */
> + if (pset.sversion >= 180000)
> + {
> + int tuples = 0;
>
> No need to assign value 0 here, because this will be unconditionally
> assigned before use anyway.
>
> ~~~~
>
> 2. describePublications
>
>   has_pubviaroot = (pset.sversion >= 130000);
> + has_pubsequence = (pset.sversion >= 18000);
>
> That's a bug! Should be 180000, not 18000.
>
> ======
>
> And, please see the attached diffs patch, which implements the
> nitpicks mentioned above.

These are handled in the v20240729 version attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm3SucGGLe-B-a_aqWNWQZ-yfxFTiAA0JyP-SwX4jq9Y3A%40mail.gmail.com

Regards,
Vignesh

On Fri, 26 Jul 2024 at 11:46, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> There are still pending changes from my previous review of the
> 0720-0003 patch [1], but here are some new review comments for your
> latest patch v20240525-0003.
> 2b.
> Is it better to name these returned by-ref ptrs like 'ret_log_cnt',
> and 'ret_lsn' to emphasise they are output variables? YMMV.

I felt this is ok as we have mentioned in function header too

> ======
> src/test/subscription/t/034_sequences.pl
>
> 4.
> Q. Should we be suspicious that log_cnt changes from '32' to '31', or
> is there a valid explanation? It smells like some calculation is
> off-by-one, but without debugging I can't tell if it is right or
> wrong.

 It works like this: for every 33 nextval we will get log_cnt as 0. So
for 33 * 6(198) log_cnt will be 0, then for 199 log_cnt will be 32 and
for 200 log_cnt will be 31. This pattern repeats, so this is ok.

These are handled in the v20240729 version attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm3SucGGLe-B-a_aqWNWQZ-yfxFTiAA0JyP-SwX4jq9Y3A%40mail.gmail.com

Regards,
Vignesh

On Thu, 25 Jul 2024 at 12:08, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Thu, Jul 25, 2024 at 9:06 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > The attached v20240725 version patch has the changes for the same.
>
> Thank You for addressing the comments. Please review below issues:
>
> 1) Sub ahead of pub due to wrong initial sync of last_value for
> non-incremented sequences. Steps at [1]
> 2) Sequence's min value is not honored on sub during replication. Steps at [2]
>
> [1]:
> -----------
> on PUB:
> CREATE SEQUENCE myseq001 INCREMENT 5 START 100;
> SELECT * from pg_sequences; -->shows last_val as NULL
>
> on SUB:
> CREATE SEQUENCE myseq001 INCREMENT 5 START 100;
> SELECT * from pg_sequences; -->correctly shows last_val as NULL
> ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
> SELECT * from pg_sequences;  -->wrongly updates last_val to 100; it is
> still NULL on Pub.
>
> Thus , SELECT nextval('myseq001') on pub gives 100, while on sub gives 105.
> -----------
>
>
> [2]:
> -----------
> Pub:
> CREATE SEQUENCE myseq0 INCREMENT 5 START 10;
> SELECT * from pg_sequences;
>
> Sub:
> CREATE SEQUENCE myseq0 INCREMENT 5 MINVALUE 100;
>
> Pub:
> SELECT nextval('myseq0');
> SELECT nextval('myseq0');
>
> Sub:
> ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
> --check 'last_value', it is 15 while min_value is 100
> SELECT * from pg_sequences;

Thanks for reporting this, these issues are fixed in the attached
v20240730_2 version patch.

Regards,
Vignesh

On Thu, 25 Jul 2024 at 15:41, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Thu, Jul 25, 2024 at 12:08 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Thu, Jul 25, 2024 at 9:06 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > The attached v20240725 version patch has the changes for the same.
> >
> > Thank You for addressing the comments. Please review below issues:
> >
> > 1) Sub ahead of pub due to wrong initial sync of last_value for
> > non-incremented sequences. Steps at [1]
> > 2) Sequence's min value is not honored on sub during replication. Steps at [2]
>
> One more issue:
> 3)  Sequence datatype's range is not honored on sub during
> replication, while it is honored for tables.
>
>
> Behaviour for tables:
> ---------------------
> Pub: create table tab1( i integer);
> Sub: create table tab1( i smallint);
>
> Pub: insert into tab1 values(generate_series(1, 32768));
>
> Error on sub:
> 2024-07-25 10:38:06.446 IST [178680] ERROR:  value "32768" is out of
> range for type smallint
>
> ---------------------
> Behaviour for sequences:
> ---------------------
>
> Pub:
> CREATE SEQUENCE myseq_i as integer INCREMENT 10000 START 1;
>
> Sub:
> CREATE SEQUENCE myseq_i as smallint INCREMENT 10000 START 1;
>
> Pub:
> SELECT nextval('myseq_i');
> SELECT nextval('myseq_i');
> SELECT nextval('myseq_i');
> SELECT nextval('myseq_i');
> SELECT nextval('myseq_i'); -->brings value to 40001
>
> Sub:
> ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
> SELECT * from pg_sequences;  -->last_val reached till 40001, while the
> range is till  32767.

This issue is addressed in the v20240730_2 version patch attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm3%2BXzHAbgyn8gmbBLK5goyv_uyGgHEsTQxRZ8bVk6nAEg%40mail.gmail.com

Regards,
Vignesh

Hi Vignesh,

Here are my review comments for your latest 0730_2* patches.

Patch v20240730_2-0001 looks good to me.

Patch v20240730_2-0002 looks good to me.

My comments for the v20240730_2-0003 patch are below:

//////////

GENERAL

1. Inconsistent terms

I've noticed there are many variations of how the sequence sync worker is known:
- "sequencesync worker"
- "sequence sync worker"
- "sequence-sync worker"
- "sequence synchronization worker"
- more?

We must settle on some standardized name.

AFAICT we generally use "table synchronization worker" in the docs,
and "tablesync worker" in the code and comments.  IMO, we should do
same as that for sequences -- e.g. "sequence synchronization worker"
in the docs, and "sequencesync worker" in the code and comments.

======
doc/src/sgml/catalogs.sgml

nitpick - the links should jump directly to REFRESH PUBLICATION or
REFRESH PUBLICATION SEQUENCES. Currently they go to the top of the
ALTER SUBSCRIPTION page which is not as useful.

======
src/backend/commands/sequence.c

do_setval:
nitpick - minor wording in the function header
nitpick - change some param names to more closely resemble the fields
they get assigned to (/logcnt/log_cnt/, /iscalled/is_called/)

~

2.
  seq->is_called = iscalled;
- seq->log_cnt = 0;
+ seq->log_cnt = (logcnt == SEQ_LOG_CNT_INVALID) ? 0: logcnt;

The logic here for SEQ_LOG_CNT_INVALID seemed strange. Why not just
#define SEQ_LOG_CNT_INVALID as 0 in the first place if that is what
you will assign for invalid? Then you won't need to do anything here
except seq->log_cnt = log_cnt;

======
src/backend/catalog/pg_subscription.c

HasSubscriptionRelations:
nitpick - I think the comment "If even a single tuple exists..." is
not quite accurate. e.g. It also has to be the right kind of tuple.

~~

GetSubscriptionRelations:
nitpick - Give more description in the function header about the other
parameters.
nitpick - I felt that a better name for 'all_relations' is all_states.
Because in my mind *all relations* sounds more like when both
'all_tables' and 'all_sequences' are true.
nitpick - IMO add an Assert to be sure something is being fetched.
Assert(get_tables || get_sequences);
nitpick - Rephrase the "skip the tables" and "skip the sequences"
comments to be more aligned with the code condition.

~

3.
- if (not_ready)
+ /* Get the relations that are not in ready state */
+ if (get_tables && !all_relations)
  ScanKeyInit(&skey[nkeys++],
  Anum_pg_subscription_rel_srsubstate,
  BTEqualStrategyNumber, F_CHARNE,
  CharGetDatum(SUBREL_STATE_READY));
+ /* Get the sequences that are in init state */
+ else if (get_sequences && !all_relations)
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_rel_srsubstate,
+ BTEqualStrategyNumber, F_CHAREQ,
+ CharGetDatum(SUBREL_STATE_INIT));

This is quite tricky, using multiple flags (get_tables and
get_sequences) in such a way. It might even be a bug -- e.g. Is the
'else' keyword correct? Otherwise, when both get_tables and
get_sequences are true, and all_relations is false, then the sequence
part wouldn't even get executed (???).

======
src/backend/commands/subscriptioncmds.c

CreateSubscription:
nitpick - let's move the 'tables' declaration to be beside the
'sequences' var for consistency. (in passing move other vars too)
nitpick - it's not strictly required for the patch, but let's change
the 'tables' loop to be consistent with the new sequences loop.

~~~

4. AlterSubscription_refresh

My first impression (from the function comment) is that these function
parameters are a bit awkward. For example,
- It says:  If 'copy_data' parameter is true, the function will set
the state to "init"; otherwise, it will set the state to "ready".
- It also says: "If 'all_relations' is true, mark all objects with
"init" state..."
Those statements seem to clash. e.g. if copy_data is false but
all_relations is true, then what (???)

~

nitpick - tweak function comment wording.
nitpick - introduce a 'relkind' variable to avoid multiple calls of
get_rel_relkind(relid)
nitpick - use an existing 'relkind' variable instead of calling
get_rel_relkind(relid);
nitpick - add another comment about skipping (for dropping tablesync slots)

~

5.
+ /*
+ * If all the relations should be re-synchronized, then set the
+ * state to init for re-synchronization. This is currently
+ * supported only for sequences.
+ */
+ else if (all_relations)
+ {
+ ereport(DEBUG1,
+ (errmsg_internal("sequence \"%s.%s\" of subscription \"%s\" set to
INIT state",
  get_namespace_name(get_rel_namespace(relid)),
  get_rel_name(relid),
  sub->name)));
+ UpdateSubscriptionRelState(sub->oid, relid, SUBREL_STATE_INIT,
+    InvalidXLogRecPtr);

(This is a continuation of my doubts regarding 'all_relations' in the
previous review comment #4 above)

Here are some more questions about it:

~

5a. Why is this an 'else' of the !bsearch? It needs more explanation
what this case means.

~

5b. Along with more description, it might be better to reverse the
!bsearch condition, so this ('else') code is not so distantly
separated from the condition.

~

5c. Saying "only supported for sequences" seems strange: e.g. what
would it even mean to "re-synchronize" tables? They would all have to
be truncated first -- so if re-sync for tables has no meaning maybe
the parameter is misnamed and should just be 'resync_all_sequences' or
similar? In any case, an Assert here might be good.

======
src/backend/replication/logical/launcher.c

logicalrep_worker_find:

nitpick - I feel the function comment "We are only interested in..."
is now redundant since you are passing the exact worker type you want.
nitpick - I added an Assert for the types you are expecting to look for
nitpick - The comment "Search for attached worker..." is stale now
because there are more search criteria
nitpick - IMO the "Skip parallel apply workers." code is no longer
needed now that you are matching the worker type.

~~~

6. logicalrep_worker_launch

  * - must be valid worker type
  * - tablesync workers are only ones to have relid
  * - parallel apply worker is the only kind of subworker
+ * - sequencesync workers will not have relid
  */
  Assert(wtype != WORKERTYPE_UNKNOWN);
  Assert(is_tablesync_worker == OidIsValid(relid));
  Assert(is_parallel_apply_worker == (subworker_dsm != DSM_HANDLE_INVALID));
+ Assert(!is_sequencesync_worker || !OidIsValid(relid));

On further reflection, is that added comment and added Assert even
needed? I think they can be removed because saying "tablesync workers
are only ones to have relid" seems to already cover what we needed to
say/assert.

~~~

logicalrep_worker_stop:
nitpick - /type/wtype/ for readability

~~~

7.
/*
 * Count the number of registered (not necessarily running) sync workers
 * for a subscription.
 */
int
logicalrep_sync_worker_count(Oid subid)

~

I thought this function should count the sequencesync worker as well.

======
.../replication/logical/sequencesync.c

fetch_remote_sequence_data:
nitpick - tweaked function comment
nitpick - /value/last_value/ for readability

~

8.
+ *lsn = DatumGetInt64(slot_getattr(slot, 4, &isnull));
+ Assert(!isnull);

Should that be DatumGetUInt64?

~~~

copy_sequence:
nitpick - tweak function header.
nitpick - renamed the sequence vars for consistency, and declared them
all together.

======
src/backend/replication/logical/tablesync.c

9.
 void
 invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
 {
- table_states_validity = SYNC_TABLE_STATE_NEEDS_REBUILD;
+ relation_states_validity = SYNC_TABLE_STATE_NEEDS_REBUILD;
 }

I assume you changed the 'table_states_validity' name because this is
no longer exclusively for tables. So, should the function name also be
similarly changed?

~~~

process_syncing_sequences_for_apply:
nitpick - tweaked the function comment
nitpick - cannot just say "if there is not one already." a sequence
syn worker might not even be needed.
nitpick - added blank line for readability

~

10.
+ if (syncworker)
+ {
+ /* Now safe to release the LWLock */
+ LWLockRelease(LogicalRepWorkerLock);
+ break;
+ }
+ else
+ {

This 'else' can be removed if you wish to pull back all the indentation.

~~~

11.
process_syncing_tables(XLogRecPtr current_lsn)

Is the function name still OK given that is is now also syncing for sequences?

~~~

FetchTableStates:
nitpick - Reworded some of the function comment
nitpick - Function comment is stale because it is still referring to
the function parameter which this patch removed.
nitpick - tweak a comment

======
src/include/commands/sequence.h

12.
+#define SEQ_LOG_CNT_INVALID (-1)

See a previous review comment (#2 above) where I wondered why not use
value 0 for this.

~~~

13.
 extern void SequenceChangePersistence(Oid relid, char newrelpersistence);
 extern void DeleteSequenceTuple(Oid relid);
 extern void ResetSequence(Oid seq_relid);
+extern void do_setval(Oid relid, int64 next, bool iscalled, int64 logcnt);
 extern void ResetSequenceCaches(void);

do_setval() was an OK function name when it was static, but as an
exposed API it seems like a terrible name. IMO rename it to something
like 'SetSequence' to match the other API functions nearby.

~

nitpick - same change to the parameter names as suggested for the
implementation.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Mon, Jun 10, 2024 at 5:00 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, 10 Jun 2024 at 12:24, Amul Sul <sulamul@gmail.com> wrote:
> >
> >
> >
> > On Sat, Jun 8, 2024 at 6:43 PM vignesh C <vignesh21@gmail.com> wrote:
> >>
> >> On Wed, 5 Jun 2024 at 14:11, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >> [...]
> >> A new catalog table, pg_subscription_seq, has been introduced for
> >> mapping subscriptions to sequences. Additionally, the sequence LSN
> >> (Log Sequence Number) is stored, facilitating determination of
> >> sequence changes occurring before or after the returned sequence
> >> state.
> >
> >
> > Can't it be done using pg_depend? It seems a bit excessive unless I'm missing
> > something.
>
> We'll require the lsn because the sequence LSN informs the user that
> it has been synchronized up to the LSN in pg_subscription_seq. Since
> we are not supporting incremental sync, the user will be able to
> identify if he should run refresh sequences or not by checking the lsn
> of the pg_subscription_seq and the lsn of the sequence(using
> pg_sequence_state added) in the publisher.

How the user will know from seq's lsn that he needs to run refresh.
lsn indicates page_lsn and thus the sequence might advance on pub
without changing lsn and thus lsn may look the same on subscriber even
though a sequence-refresh is needed. Am I missing something here?

thanks
Shveta

On Sat, 20 Jul 2024 at 20:48, vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 12 Jul 2024 at 08:22, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Vignesh. Here are the rest of my comments for patch v20240705-0003.
> > ======
> >
> > 8. logicalrep_sequence_sync_worker_find
> >
> > +/*
> > + * Walks the workers array and searches for one that matches given
> > + * subscription id.
> > + *
> > + * We are only interested in the sequence sync worker.
> > + */
> > +LogicalRepWorker *
> > +logicalrep_sequence_sync_worker_find(Oid subid, bool only_running)
> >
> > There are other similar functions for walking the workers array to
> > search for a worker. Instead of having different functions for
> > different cases, wouldn't it be cleaner to combine these into a single
> > function, where you pass a parameter (e.g. a mask of worker types that
> > you are interested in finding)?

This is fixed in the v20240730_2 version attached at [1].

> > 17.
> > Also, where does the number 100 come from? Why not 1000? Why not 10?
> > Why have batching at all? Maybe there should be some comment to
> > describe the reason and the chosen value.

I had run some tests with 10/100 and 1000 sequences per batch for
10000 sequences. The results for it:
10 per batch   - 4.94 seconds
100 per batch  - 4.87 seconds
1000 per batch - 4.53 seconds

There is not much time difference between each of them. Currently, it
is set to 100, which seems fine since it will not generate a lot of
transactions. Additionally, the locks on the sequences will be
periodically released during the commit transaction.

I had used the test from the attached patch by changing
max_sequences_sync_per_batch to 10/100/100 in 035_sequences.pl to
verify this.

[1] - https://www.postgresql.org/message-id/CALDaNm3%2BXzHAbgyn8gmbBLK5goyv_uyGgHEsTQxRZ8bVk6nAEg%40mail.gmail.com

Regards,
Vignesh

Hi Vignesh,

I have a question about the subscriber-side behaviour of currval().

======

AFAIK it is normal for currval() to give error is nextval() has not
yet been called [1]

For example.
test_pub=# create sequence s1;
CREATE SEQUENCE
test_pub=# select * from currval('s1');
2024-08-01 07:42:48.619 AEST [24131] ERROR:  currval of sequence "s1"
is not yet defined in this session
2024-08-01 07:42:48.619 AEST [24131] STATEMENT:  select * from currval('s1');
ERROR:  currval of sequence "s1" is not yet defined in this session
test_pub=# select * from nextval('s1');
 nextval
---------
       1
(1 row)

test_pub=# select * from currval('s1');
 currval
---------
       1
(1 row)

test_pub=#

~~~

OTOH, I was hoping to be able to use currval() at the subscriber=side
to see the current sequence value after issuing ALTER .. REFRESH
PUBLICATION SEQUENCES.

Unfortunately, it has the same behaviour where currval() cannot be
used without nextval(). But, on the subscriber, you probably never
want to do an explicit nextval() independently of the publisher.

Is this currently a bug, or maybe a quirk that should be documented?

For example:

Publisher
==========

test_pub=# create sequence s1;
CREATE SEQUENCE
test_pub=# CREATE PUBLICATION pub1 FOR ALL SEQUENCES;
CREATE PUBLICATION
test_pub=# select * from nextval('s1');
 nextval
---------
       1
(1 row)

test_pub=# select * from nextval('s1');
 nextval
---------
       2
(1 row)

test_pub=# select * from nextval('s1');
 nextval
---------
       3
(1 row)

test_pub=#

Subscriber
==========

(Notice currval() always gives an error unless nextval() is used prior).

test_sub=# create sequence s1;
CREATE SEQUENCE
test_sub=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=test_pub'
PUBLICATION pub1;
2024-08-01 07:51:06.955 AEST [24325] WARNING:  subscriptions created
by regression test cases should have names starting with "regress_"
WARNING:  subscriptions created by regression test cases should have
names starting with "regress_"
NOTICE:  created replication slot "sub1" on publisher
CREATE SUBSCRIPTION
test_sub=# 2024-08-01 07:51:07.023 AEST [4211] LOG:  logical
replication apply worker for subscription "sub1" has started
2024-08-01 07:51:07.037 AEST [4213] LOG:  logical replication sequence
synchronization worker for subscription "sub1" has started
2024-08-01 07:51:07.063 AEST [4213] LOG:  logical replication
synchronization for subscription "sub1", sequence "s1" has finished
2024-08-01 07:51:07.063 AEST [4213] LOG:  logical replication sequence
synchronization worker for subscription "sub1" has finished

test_sub=# SELECT * FROM currval('s1');
2024-08-01 07:51:19.688 AEST [24325] ERROR:  currval of sequence "s1"
is not yet defined in this session
2024-08-01 07:51:19.688 AEST [24325] STATEMENT:  SELECT * FROM currval('s1');
ERROR:  currval of sequence "s1" is not yet defined in this session
test_sub=# ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
ALTER SUBSCRIPTION
test_sub=# 2024-08-01 07:51:35.298 AEST [4993] LOG:  logical
replication sequence synchronization worker for subscription "sub1"
has started

test_sub=# 2024-08-01 07:51:35.321 AEST [4993] LOG:  logical
replication synchronization for subscription "sub1", sequence "s1" has
finished
2024-08-01 07:51:35.321 AEST [4993] LOG:  logical replication sequence
synchronization worker for subscription "sub1" has finished

test_sub=#
test_sub=# SELECT * FROM currval('s1');
2024-08-01 07:51:41.438 AEST [24325] ERROR:  currval of sequence "s1"
is not yet defined in this session
2024-08-01 07:51:41.438 AEST [24325] STATEMENT:  SELECT * FROM currval('s1');
ERROR:  currval of sequence "s1" is not yet defined in this session
test_sub=#
test_sub=# SELECT * FROM nextval('s1');
 nextval
---------
       4
(1 row)

test_sub=# SELECT * FROM currval('s1');
 currval
---------
       4
(1 row)

test_sub=#

======
[1] https://www.postgresql.org/docs/current/functions-sequence.html

Kind Regards,
Peter Smith.
Fujitsu Australia.

Hi Vignesh,

I noticed that when replicating sequences (using the latest patches
0730_2*)  the subscriber-side checks the *existence* of the sequence,
but apparently it is not checking other sequence attributes.

For example, consider:

Publisher: "CREATE SEQUENCE s1 START 1 INCREMENT 2;" should be a
sequence of only odd numbers.
Subscriber: "CREATE SEQUENCE s1 START 2 INCREMENT 2;" should be a
sequence of only even numbers.

Because the names match, currently the patch allows replication of the
s1 sequence. I think that might lead to unexpected results on the
subscriber. IMO it might be safer to report ERROR unless the sequences
match properly (i.e. not just a name check).

Below is a demonstration the problem:

==========
Publisher:
==========

(publisher sequence is odd numbers)

test_pub=# create sequence s1 start 1 increment 2;
CREATE SEQUENCE
test_pub=# select * from nextval('s1');
 nextval
---------
       1
(1 row)

test_pub=# select * from nextval('s1');
 nextval
---------
       3
(1 row)

test_pub=# select * from nextval('s1');
 nextval
---------
       5
(1 row)

test_pub=# CREATE PUBLICATION pub1 FOR ALL SEQUENCES;
CREATE PUBLICATION
test_pub=#

==========
Subscriber:
==========

(subscriber sequence is even numbers)

test_sub=# create sequence s1 start 2 increment 2;
CREATE SEQUENCE
test_sub=# SELECT * FROM nextval('s1');
 nextval
---------
       2
(1 row)

test_sub=# SELECT * FROM nextval('s1');
 nextval
---------
       4
(1 row)

test_sub=# SELECT * FROM nextval('s1');
 nextval
---------
       6
(1 row)

test_sub=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=test_pub'
PUBLICATION pub1;
2024-08-01 08:43:04.198 AEST [24325] WARNING:  subscriptions created
by regression test cases should have names starting with "regress_"
WARNING:  subscriptions created by regression test cases should have
names starting with "regress_"
NOTICE:  created replication slot "sub1" on publisher
CREATE SUBSCRIPTION
test_sub=# 2024-08-01 08:43:04.294 AEST [26240] LOG:  logical
replication apply worker for subscription "sub1" has started
2024-08-01 08:43:04.309 AEST [26244] LOG:  logical replication
sequence synchronization worker for subscription "sub1" has started
2024-08-01 08:43:04.323 AEST [26244] LOG:  logical replication
synchronization for subscription "sub1", sequence "s1" has finished
2024-08-01 08:43:04.323 AEST [26244] LOG:  logical replication
sequence synchronization worker for subscription "sub1" has finished

(after the CREATE SUBSCRIPTION we are getting replicated odd values
from the publisher, even though the subscriber side sequence was
supposed to be even numbers)

test_sub=# SELECT * FROM nextval('s1');
 nextval
---------
       7
(1 row)

test_sub=# SELECT * FROM nextval('s1');
 nextval
---------
       9
(1 row)

test_sub=# SELECT * FROM nextval('s1');
 nextval
---------
      11
(1 row)

(Looking at the description you would expect odd values for this
sequence to be impossible)

test_sub=# \dS+ s1
                             Sequence "public.s1"
  Type  | Start | Minimum |       Maximum       | Increment | Cycles? | Cache
--------+-------+---------+---------------------+-----------+---------+-------
 bigint |     2 |       1 | 9223372036854775807 |         2 | no      |     1

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Thu, Aug 1, 2024 at 9:26 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Jul 29, 2024 at 4:17 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for reporting this, these issues are fixed in the attached
> > v20240730_2 version patch.
> >

I was reviewing the design of patch003, and I have a query. Do we need
to even start an apply worker and create replication slot when
subscription created is for 'sequences only'? IIUC, currently logical
replication apply worker is the one launching sequence-sync worker
whenever needed. I think it should be the launcher doing this job and
thus apply worker may even not be needed for current functionality of
sequence sync? Going forward when we implement incremental sync of
sequences, then we may have apply worker started but now it is not
needed.

thanks
Shveta

On Fri, Aug 2, 2024 at 2:24 PM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Thu, Aug 1, 2024 at 9:26 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Mon, Jul 29, 2024 at 4:17 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > Thanks for reporting this, these issues are fixed in the attached
> > > v20240730_2 version patch.
> > >
>
> I was reviewing the design of patch003, and I have a query. Do we need
> to even start an apply worker and create replication slot when
> subscription created is for 'sequences only'? IIUC, currently logical
> replication apply worker is the one launching sequence-sync worker
> whenever needed. I think it should be the launcher doing this job and
> thus apply worker may even not be needed for current functionality of
> sequence sync? Going forward when we implement incremental sync of
> sequences, then we may have apply worker started but now it is not
> needed.
>

Also, can we please mention the state change and 'who does what' atop
sequencesync.c file similar to what we have atop tablesync.c file
otherwise it is difficult to figure out the flow.

thanks
Shveta

On Wed, 31 Jul 2024 at 12:56, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> Here are my review comments for your latest 0730_2* patches.
>
> Patch v20240730_2-0001 looks good to me.
>
> Patch v20240730_2-0002 looks good to me.
>
> My comments for the v20240730_2-0003 patch are below:
> ~~~
>
> 4. AlterSubscription_refresh
>
> My first impression (from the function comment) is that these function
> parameters are a bit awkward. For example,
> - It says:  If 'copy_data' parameter is true, the function will set
> the state to "init"; otherwise, it will set the state to "ready".
> - It also says: "If 'all_relations' is true, mark all objects with
> "init" state..."
> Those statements seem to clash. e.g. if copy_data is false but
> all_relations is true, then what (???)

all_relations will be true only for "ALTER SUBSCRIPTION ... REFRESH
PUBLICATION SEQUENCES".  With option is not supported along with this
command so copy_data with false option is not possible here. Added an
assert for this.
>
> 8.
> + *lsn = DatumGetInt64(slot_getattr(slot, 4, &isnull));
> + Assert(!isnull);
>
> Should that be DatumGetUInt64?

It should be DatumGetLSN here.

The rest of the comments are fixed. The attached v20240805 version
patch has the changes for the same.

Regards,
Vignesh

On Thu, 1 Aug 2024 at 03:33, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> I have a question about the subscriber-side behaviour of currval().
>
> ======
>
> AFAIK it is normal for currval() to give error is nextval() has not
> yet been called [1]
>
> For example.
> test_pub=# create sequence s1;
> CREATE SEQUENCE
> test_pub=# select * from currval('s1');
> 2024-08-01 07:42:48.619 AEST [24131] ERROR:  currval of sequence "s1"
> is not yet defined in this session
> 2024-08-01 07:42:48.619 AEST [24131] STATEMENT:  select * from currval('s1');
> ERROR:  currval of sequence "s1" is not yet defined in this session
> test_pub=# select * from nextval('s1');
>  nextval
> ---------
>        1
> (1 row)
>
> test_pub=# select * from currval('s1');
>  currval
> ---------
>        1
> (1 row)
>
> test_pub=#
>
> ~~~
>
> OTOH, I was hoping to be able to use currval() at the subscriber=side
> to see the current sequence value after issuing ALTER .. REFRESH
> PUBLICATION SEQUENCES.
>
> Unfortunately, it has the same behaviour where currval() cannot be
> used without nextval(). But, on the subscriber, you probably never
> want to do an explicit nextval() independently of the publisher.
>
> Is this currently a bug, or maybe a quirk that should be documented?

The currval returns the most recent value obtained from the nextval
function for a given sequence within the current session. This
function is specific to the session, meaning it only provides the last
sequence value retrieved during that session. However, if you call
currval before using nextval in the same session, you'll encounter an
error stating "currval of the sequence is not yet defined in this
session." Meaning even in the publisher this value is only visible in
the current session and not in a different session. Alternatively you
can use the following to get the last_value of the sequence: SELECT
last_value FROM sequence_name. I feel this need not be documented as
the similar issue is present in the publisher and there is an "SELECT
last_value FROM sequence_name" to get the last_value.

Regards,
Vignesh

On Thu, 1 Aug 2024 at 04:25, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> I noticed that when replicating sequences (using the latest patches
> 0730_2*)  the subscriber-side checks the *existence* of the sequence,
> but apparently it is not checking other sequence attributes.
>
> For example, consider:
>
> Publisher: "CREATE SEQUENCE s1 START 1 INCREMENT 2;" should be a
> sequence of only odd numbers.
> Subscriber: "CREATE SEQUENCE s1 START 2 INCREMENT 2;" should be a
> sequence of only even numbers.
>
> Because the names match, currently the patch allows replication of the
> s1 sequence. I think that might lead to unexpected results on the
> subscriber. IMO it might be safer to report ERROR unless the sequences
> match properly (i.e. not just a name check).
>
> Below is a demonstration the problem:
>
> ==========
> Publisher:
> ==========
>
> (publisher sequence is odd numbers)
>
> test_pub=# create sequence s1 start 1 increment 2;
> CREATE SEQUENCE
> test_pub=# select * from nextval('s1');
>  nextval
> ---------
>        1
> (1 row)
>
> test_pub=# select * from nextval('s1');
>  nextval
> ---------
>        3
> (1 row)
>
> test_pub=# select * from nextval('s1');
>  nextval
> ---------
>        5
> (1 row)
>
> test_pub=# CREATE PUBLICATION pub1 FOR ALL SEQUENCES;
> CREATE PUBLICATION
> test_pub=#
>
> ==========
> Subscriber:
> ==========
>
> (subscriber sequence is even numbers)
>
> test_sub=# create sequence s1 start 2 increment 2;
> CREATE SEQUENCE
> test_sub=# SELECT * FROM nextval('s1');
>  nextval
> ---------
>        2
> (1 row)
>
> test_sub=# SELECT * FROM nextval('s1');
>  nextval
> ---------
>        4
> (1 row)
>
> test_sub=# SELECT * FROM nextval('s1');
>  nextval
> ---------
>        6
> (1 row)
>
> test_sub=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=test_pub'
> PUBLICATION pub1;
> 2024-08-01 08:43:04.198 AEST [24325] WARNING:  subscriptions created
> by regression test cases should have names starting with "regress_"
> WARNING:  subscriptions created by regression test cases should have
> names starting with "regress_"
> NOTICE:  created replication slot "sub1" on publisher
> CREATE SUBSCRIPTION
> test_sub=# 2024-08-01 08:43:04.294 AEST [26240] LOG:  logical
> replication apply worker for subscription "sub1" has started
> 2024-08-01 08:43:04.309 AEST [26244] LOG:  logical replication
> sequence synchronization worker for subscription "sub1" has started
> 2024-08-01 08:43:04.323 AEST [26244] LOG:  logical replication
> synchronization for subscription "sub1", sequence "s1" has finished
> 2024-08-01 08:43:04.323 AEST [26244] LOG:  logical replication
> sequence synchronization worker for subscription "sub1" has finished
>
> (after the CREATE SUBSCRIPTION we are getting replicated odd values
> from the publisher, even though the subscriber side sequence was
> supposed to be even numbers)
>
> test_sub=# SELECT * FROM nextval('s1');
>  nextval
> ---------
>        7
> (1 row)
>
> test_sub=# SELECT * FROM nextval('s1');
>  nextval
> ---------
>        9
> (1 row)
>
> test_sub=# SELECT * FROM nextval('s1');
>  nextval
> ---------
>       11
> (1 row)
>
> (Looking at the description you would expect odd values for this
> sequence to be impossible)
>
> test_sub=# \dS+ s1
>                              Sequence "public.s1"
>   Type  | Start | Minimum |       Maximum       | Increment | Cycles? | Cache
> --------+-------+---------+---------------------+-----------+---------+-------
>  bigint |     2 |       1 | 9223372036854775807 |         2 | no      |     1

Even if we check the sequence definition during the CREATE
SUBSCRIPTION/ALTER SUBSCRIPTION ... REFRESH PUBLICATION or ALTER
SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES commands, there's still
a chance that the sequence definition might change after the command
has been executed. Currently, there's no mechanism to lock a sequence,
and we also permit replication of table data even if the table
structures differ, such as mismatched data types like int and
smallint. I have modified it to log a warning to inform users that the
sequence options on the publisher and subscriber are not the same and
advise them to ensure that the sequence definitions are consistent
between both.
The v20240805 version patch attached at [1] has the changes for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm1Y_ot-jFRfmtwDuwmFrgSSYHjVuy28RspSopTtwzXy8w%40mail.gmail.com

Regards,
Vignesh

On Wed, 31 Jul 2024 at 14:39, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Jun 10, 2024 at 5:00 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Mon, 10 Jun 2024 at 12:24, Amul Sul <sulamul@gmail.com> wrote:
> > >
> > >
> > >
> > > On Sat, Jun 8, 2024 at 6:43 PM vignesh C <vignesh21@gmail.com> wrote:
> > >>
> > >> On Wed, 5 Jun 2024 at 14:11, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >> [...]
> > >> A new catalog table, pg_subscription_seq, has been introduced for
> > >> mapping subscriptions to sequences. Additionally, the sequence LSN
> > >> (Log Sequence Number) is stored, facilitating determination of
> > >> sequence changes occurring before or after the returned sequence
> > >> state.
> > >
> > >
> > > Can't it be done using pg_depend? It seems a bit excessive unless I'm missing
> > > something.
> >
> > We'll require the lsn because the sequence LSN informs the user that
> > it has been synchronized up to the LSN in pg_subscription_seq. Since
> > we are not supporting incremental sync, the user will be able to
> > identify if he should run refresh sequences or not by checking the lsn
> > of the pg_subscription_seq and the lsn of the sequence(using
> > pg_sequence_state added) in the publisher.
>
> How the user will know from seq's lsn that he needs to run refresh.
> lsn indicates page_lsn and thus the sequence might advance on pub
> without changing lsn and thus lsn may look the same on subscriber even
> though a sequence-refresh is needed. Am I missing something here?

When a sequence is synchronized to the subscriber, the page LSN of the
sequence from the publisher is also retrieved and stored in
pg_subscriber_rel as shown below:
--- Publisher page lsn
publisher=# select pg_sequence_state('seq1');
 pg_sequence_state
--------------------
 (0/1510E38,65,1,t)
(1 row)

--- Subscriber stores the publisher's page lsn for the sequence
subscriber=# select * from pg_subscription_rel where srrelid = 16384;
 srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
   16389 |   16384 | r          | 0/1510E38
(1 row)

If changes are made to the sequence, such as performing many nextvals,
the page LSN will be updated. Currently the sequence values are
prefetched for SEQ_LOG_VALS 32, so the lsn will not get updated for
the prefetched values, once the prefetched values are consumed the lsn
will get updated.
For example:
--- Updated LSN on the publisher (old lsn - 0/1510E38, new lsn - 0/1558CA8)
publisher=# select pg_sequence_state('seq1');
  pg_sequence_state
----------------------
 (0/1558CA8,143,22,t)
(1 row)

The user can then compare this updated value with the sequence's LSN
in pg_subscription_rel to determine when to re-synchronize the
sequence.

Regards,
Vignesh

On Thu, 1 Aug 2024 at 09:26, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Jul 29, 2024 at 4:17 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > Thanks for reporting this, these issues are fixed in the attached
> > v20240730_2 version patch.
> >
>
> Thanks for addressing the comments. Please find few comments on patch001 alone:
>
> Potential Bug:
> 1) 'last_value' returned by pg_sequence_state() is wrong initially.
>
> postgres=# create sequence myseq5;
> CREATE SEQUENCE
>
> postgres=# select * from pg_sequence_state('myseq5');
>  page_lsn  | last_value | log_cnt | is_called
> -----------+------------+---------+-----------
>  0/1579C78 |          1 |       0 | f
>
> postgres=# SELECT nextval('myseq5') ;
>  nextval
> ---------
>        1
>
> postgres=# select * from pg_sequence_state('myseq5');
>  page_lsn  | last_value | log_cnt | is_called
> -----------+------------+---------+-----------
>  0/1579FD8 |          1 |      32 | t
>
>
> Both calls returned 1. First call should have returned NULL.

I noticed the same behavior for selecting from a sequence:
postgres=# select * from myseq5;
 last_value | log_cnt | is_called
------------+---------+-----------
          1 |       0 | f
(1 row)

postgres=# select nextval('myseq5');
 nextval
---------
       1
(1 row)

postgres=# select * from myseq5;
 last_value | log_cnt | is_called
------------+---------+-----------
          1 |      32 | t
(1 row)

By default it shows the last_value as the start value for the
sequence. So this looks ok to me.

> 2)
> func.sgml:
> a) pg_sequence_state : Don't we need to give input arg as regclass
> like we give in  nextval,setval etc?
> b) Should 'lsn' be changed to 'page_lsn' as returned in output of
> pg_sequence_state()

Modified

>
> 3)
> read_seq_tuple() header says:
>  * lsn_ret will be set to the page LSN if the caller requested it.
>  * This allows the caller to determine which sequence changes are
>  * before/after the returned sequence state.
>
> How, using lsn which is page-lsn and not sequence value/change lsn,
> does the user interpret if sequence changes are before/after the
> returned sequence state? Can you please elaborate or amend the
> comment?

I have added this to pg_sequence_state function header in 003 patch as
the subscriber side changes are present here, I felt that is more apt
to mention. This is also added in sequencesync.c file header.

The attached v20240805_2 version patch has the changes for the same.

Regards,
Vignesh

On Fri, 2 Aug 2024 at 14:24, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Thu, Aug 1, 2024 at 9:26 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Mon, Jul 29, 2024 at 4:17 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > Thanks for reporting this, these issues are fixed in the attached
> > > v20240730_2 version patch.
> > >
>
> I was reviewing the design of patch003, and I have a query. Do we need
> to even start an apply worker and create replication slot when
> subscription created is for 'sequences only'? IIUC, currently logical
> replication apply worker is the one launching sequence-sync worker
> whenever needed. I think it should be the launcher doing this job and
> thus apply worker may even not be needed for current functionality of
> sequence sync? Going forward when we implement incremental sync of
> sequences, then we may have apply worker started but now it is not
> needed.

I believe the current method of having the apply worker initiate the
sequence sync worker is advantageous for several reasons:
a) Reduces Launcher Load: This approach prevents overloading the
launcher, which must handle various other subscription requests.
b) Facilitates Incremental Sync: It provides a more straightforward
path to extend support for incremental sequence synchronization.
c) Reuses Existing Code: It leverages the existing tablesync worker
code for starting the tablesync process, avoiding the need to
duplicate code in the launcher.
d) Simplified Code Maintenance: Centralizing sequence synchronization
logic within the apply worker can simplify code maintenance and
updates, as changes will only need to be made in one place rather than
across multiple components.
e) Better Monitoring and Debugging: With sequence synchronization
being handled by the apply worker, you can more effectively monitor
and debug synchronization processes since all related operations are
managed by a single component.

Also, I noticed that even when a publication has no tables, we create
replication slot and start apply worker.

Regards,
Vignesh

On Fri, 2 Aug 2024 at 14:33, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Fri, Aug 2, 2024 at 2:24 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Thu, Aug 1, 2024 at 9:26 AM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > On Mon, Jul 29, 2024 at 4:17 PM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > Thanks for reporting this, these issues are fixed in the attached
> > > > v20240730_2 version patch.
> > > >
> >
> > I was reviewing the design of patch003, and I have a query. Do we need
> > to even start an apply worker and create replication slot when
> > subscription created is for 'sequences only'? IIUC, currently logical
> > replication apply worker is the one launching sequence-sync worker
> > whenever needed. I think it should be the launcher doing this job and
> > thus apply worker may even not be needed for current functionality of
> > sequence sync? Going forward when we implement incremental sync of
> > sequences, then we may have apply worker started but now it is not
> > needed.
> >
>
> Also, can we please mention the state change and 'who does what' atop
> sequencesync.c file similar to what we have atop tablesync.c file
> otherwise it is difficult to figure out the flow.

I have added this in sequencesync.c file, the changes for the same are
available at v20240805_2 version patch at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm1kk1MHGk3BU_XTxay%3DdR6sMHnm4TT5cmVz2f_JXkWENQ%40mail.gmail.com

Regards,
Vignesh

On Mon, Aug 5, 2024 at 2:36 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 2 Aug 2024 at 14:24, shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Thu, Aug 1, 2024 at 9:26 AM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > On Mon, Jul 29, 2024 at 4:17 PM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > Thanks for reporting this, these issues are fixed in the attached
> > > > v20240730_2 version patch.
> > > >
> >
> > I was reviewing the design of patch003, and I have a query. Do we need
> > to even start an apply worker and create replication slot when
> > subscription created is for 'sequences only'? IIUC, currently logical
> > replication apply worker is the one launching sequence-sync worker
> > whenever needed. I think it should be the launcher doing this job and
> > thus apply worker may even not be needed for current functionality of
> > sequence sync?
>

But that would lead to maintaining all sequence-sync of each
subscription by launcher. Say there are 100 sequences per subscription
and some of them from each subscription are failing due to some
reasons then the launcher will be responsible for ensuring all the
sequences are synced. I think it would be better to handle
per-subscription work by the apply worker.

>
> Going forward when we implement incremental sync of
> > sequences, then we may have apply worker started but now it is not
> > needed.
>
> I believe the current method of having the apply worker initiate the
> sequence sync worker is advantageous for several reasons:
> a) Reduces Launcher Load: This approach prevents overloading the
> launcher, which must handle various other subscription requests.
> b) Facilitates Incremental Sync: It provides a more straightforward
> path to extend support for incremental sequence synchronization.
> c) Reuses Existing Code: It leverages the existing tablesync worker
> code for starting the tablesync process, avoiding the need to
> duplicate code in the launcher.
> d) Simplified Code Maintenance: Centralizing sequence synchronization
> logic within the apply worker can simplify code maintenance and
> updates, as changes will only need to be made in one place rather than
> across multiple components.
> e) Better Monitoring and Debugging: With sequence synchronization
> being handled by the apply worker, you can more effectively monitor
> and debug synchronization processes since all related operations are
> managed by a single component.
>
> Also, I noticed that even when a publication has no tables, we create
> replication slot and start apply worker.
>

As far as I understand slots and origins are primarily required for
incremental sync. Would it be used only for sequence-sync cases? If
not then we can avoid creating those. I agree that it would add some
complexity to the code with sequence-specific checks, so we can create
a top-up patch for this if required and evaluate its complexity versus
the benefit it produces.

--
With Regards,
Amit Kapila.

On Mon, Aug 5, 2024 at 11:04 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 31 Jul 2024 at 14:39, shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Mon, Jun 10, 2024 at 5:00 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Mon, 10 Jun 2024 at 12:24, Amul Sul <sulamul@gmail.com> wrote:
> > > >
> > > >
> > > >
> > > > On Sat, Jun 8, 2024 at 6:43 PM vignesh C <vignesh21@gmail.com> wrote:
> > > >>
> > > >> On Wed, 5 Jun 2024 at 14:11, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >> [...]
> > > >> A new catalog table, pg_subscription_seq, has been introduced for
> > > >> mapping subscriptions to sequences. Additionally, the sequence LSN
> > > >> (Log Sequence Number) is stored, facilitating determination of
> > > >> sequence changes occurring before or after the returned sequence
> > > >> state.
> > > >
> > > >
> > > > Can't it be done using pg_depend? It seems a bit excessive unless I'm missing
> > > > something.
> > >
> > > We'll require the lsn because the sequence LSN informs the user that
> > > it has been synchronized up to the LSN in pg_subscription_seq. Since
> > > we are not supporting incremental sync, the user will be able to
> > > identify if he should run refresh sequences or not by checking the lsn
> > > of the pg_subscription_seq and the lsn of the sequence(using
> > > pg_sequence_state added) in the publisher.
> >
> > How the user will know from seq's lsn that he needs to run refresh.
> > lsn indicates page_lsn and thus the sequence might advance on pub
> > without changing lsn and thus lsn may look the same on subscriber even
> > though a sequence-refresh is needed. Am I missing something here?
>
> When a sequence is synchronized to the subscriber, the page LSN of the
> sequence from the publisher is also retrieved and stored in
> pg_subscriber_rel as shown below:
> --- Publisher page lsn
> publisher=# select pg_sequence_state('seq1');
>  pg_sequence_state
> --------------------
>  (0/1510E38,65,1,t)
> (1 row)
>
> --- Subscriber stores the publisher's page lsn for the sequence
> subscriber=# select * from pg_subscription_rel where srrelid = 16384;
>  srsubid | srrelid | srsubstate | srsublsn
> ---------+---------+------------+-----------
>    16389 |   16384 | r          | 0/1510E38
> (1 row)
>
> If changes are made to the sequence, such as performing many nextvals,
> the page LSN will be updated. Currently the sequence values are
> prefetched for SEQ_LOG_VALS 32, so the lsn will not get updated for
> the prefetched values, once the prefetched values are consumed the lsn
> will get updated.
> For example:
> --- Updated LSN on the publisher (old lsn - 0/1510E38, new lsn - 0/1558CA8)
> publisher=# select pg_sequence_state('seq1');
>   pg_sequence_state
> ----------------------
>  (0/1558CA8,143,22,t)
> (1 row)
>
> The user can then compare this updated value with the sequence's LSN
> in pg_subscription_rel to determine when to re-synchronize the
> sequence.

Thanks for the details. But I was referring to the case where we are
in between pre-fetched values on publisher (say at 25th value), while
on subscriber we are slightly behind (say at 15th value), but page-lsn
will be the same on both. Since the subscriber is behind, a
sequence-refresh is needed on sub, but by looking at lsn (which is
same), one can not say that for sure.  Let me know if I have
misunderstood it.

thanks
Shveta

On Mon, Aug 5, 2024 at 5:28 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Aug 5, 2024 at 2:36 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Fri, 2 Aug 2024 at 14:24, shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > On Thu, Aug 1, 2024 at 9:26 AM shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > On Mon, Jul 29, 2024 at 4:17 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > >
> > > > > Thanks for reporting this, these issues are fixed in the attached
> > > > > v20240730_2 version patch.
> > > > >
> > >
> > > I was reviewing the design of patch003, and I have a query. Do we need
> > > to even start an apply worker and create replication slot when
> > > subscription created is for 'sequences only'? IIUC, currently logical
> > > replication apply worker is the one launching sequence-sync worker
> > > whenever needed. I think it should be the launcher doing this job and
> > > thus apply worker may even not be needed for current functionality of
> > > sequence sync?
> >
>
> But that would lead to maintaining all sequence-sync of each
> subscription by launcher. Say there are 100 sequences per subscription
> and some of them from each subscription are failing due to some
> reasons then the launcher will be responsible for ensuring all the
> sequences are synced. I think it would be better to handle
> per-subscription work by the apply worker.

I thought we can give that task to sequence-sync worker. Once sequence
sync worker is started by launcher, it keeps on syncing until all the
sequences are synced (even failed ones) and then exits only after all
are synced; instead of apply worker starting it multiple times for
failed sequences. Launcher to start sequence sync worker when signaled
by 'alter-sub refresh seq'.
But after going through details given by Vignesh in [1], I also see
the benefits of using apply worker for this task. Since apply worker
is already looping and doing that for table-sync, we can reuse the
same code for sequence sync and maintenance will be easy. So looks
okay if we go with existing apply worker design.

[1]: https://www.postgresql.org/message-id/CALDaNm1KO8f3Fj%2BRHHXM%3DUSGwOcW242M1jHee%3DX_chn2ToiCpw%40mail.gmail.com

>
> >
> > Going forward when we implement incremental sync of
> > > sequences, then we may have apply worker started but now it is not
> > > needed.
> >
> > I believe the current method of having the apply worker initiate the
> > sequence sync worker is advantageous for several reasons:
> > a) Reduces Launcher Load: This approach prevents overloading the
> > launcher, which must handle various other subscription requests.
> > b) Facilitates Incremental Sync: It provides a more straightforward
> > path to extend support for incremental sequence synchronization.
> > c) Reuses Existing Code: It leverages the existing tablesync worker
> > code for starting the tablesync process, avoiding the need to
> > duplicate code in the launcher.
> > d) Simplified Code Maintenance: Centralizing sequence synchronization
> > logic within the apply worker can simplify code maintenance and
> > updates, as changes will only need to be made in one place rather than
> > across multiple components.
> > e) Better Monitoring and Debugging: With sequence synchronization
> > being handled by the apply worker, you can more effectively monitor
> > and debug synchronization processes since all related operations are
> > managed by a single component.
> >
> > Also, I noticed that even when a publication has no tables, we create
> > replication slot and start apply worker.
> >
>
> As far as I understand slots and origins are primarily required for
> incremental sync. Would it be used only for sequence-sync cases? If
> not then we can avoid creating those. I agree that it would add some
> complexity to the code with sequence-specific checks, so we can create
> a top-up patch for this if required and evaluate its complexity versus
> the benefit it produces.
>
> --
> With Regards,
> Amit Kapila.

On Tue, Aug 6, 2024 at 8:49 AM shveta malik <shveta.malik@gmail.com> wrote:
>

Do we need some kind of coordination between table sync and sequence
sync for internally generated sequences? Lets say we have an identity
column with a 'GENERATED ALWAYS' sequence. When the sequence is synced
to subscriber, subscriber can also do an insert to table (extra one)
incrementing the sequence and then when publisher performs an insert,
apply worker will blindly copy that row to sub's table making identity
column's duplicate entries.

CREATE TABLE color ( color_id INT GENERATED ALWAYS AS
IDENTITY,color_name VARCHAR NOT NULL);

Pub: insert into color(color_name) values('red');

Sub: perform sequence refresh and check 'r' state is reached, then do insert:
insert into color(color_name) values('yellow');

Pub: insert into color(color_name) values('blue');

After above, data on Pub: (1, 'red') ;(2, 'blue'),

After above, data on Sub: (1, 'red') ;(2, 'yellow'); (2, 'blue'),

Identity column has duplicate values. Should the apply worker error
out while inserting such a  row to the table? Or it is not in the
scope of this project?

thanks
Shveta

On Tue, Aug 6, 2024 at 8:49 AM shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Aug 5, 2024 at 5:28 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Aug 5, 2024 at 2:36 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Fri, 2 Aug 2024 at 14:24, shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > On Thu, Aug 1, 2024 at 9:26 AM shveta malik <shveta.malik@gmail.com> wrote:
> > > > >
> > > > > On Mon, Jul 29, 2024 at 4:17 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > > >
> > > > > > Thanks for reporting this, these issues are fixed in the attached
> > > > > > v20240730_2 version patch.
> > > > > >
> > > >
> > > > I was reviewing the design of patch003, and I have a query. Do we need
> > > > to even start an apply worker and create replication slot when
> > > > subscription created is for 'sequences only'? IIUC, currently logical
> > > > replication apply worker is the one launching sequence-sync worker
> > > > whenever needed. I think it should be the launcher doing this job and
> > > > thus apply worker may even not be needed for current functionality of
> > > > sequence sync?
> > >
> >
> > But that would lead to maintaining all sequence-sync of each
> > subscription by launcher. Say there are 100 sequences per subscription
> > and some of them from each subscription are failing due to some
> > reasons then the launcher will be responsible for ensuring all the
> > sequences are synced. I think it would be better to handle
> > per-subscription work by the apply worker.
>
> I thought we can give that task to sequence-sync worker. Once sequence
> sync worker is started by launcher, it keeps on syncing until all the
> sequences are synced (even failed ones) and then exits only after all
> are synced; instead of apply worker starting it multiple times for
> failed sequences. Launcher to start sequence sync worker when signaled
> by 'alter-sub refresh seq'.
> But after going through details given by Vignesh in [1], I also see
> the benefits of using apply worker for this task. Since apply worker
> is already looping and doing that for table-sync, we can reuse the
> same code for sequence sync and maintenance will be easy. So looks
> okay if we go with existing apply worker design.
>

Fair enough. However, I was wondering whether apply_worker should exit
after syncing all sequences for a sequence-only subscription or should
it be there for future commands that can refresh the subscription and
add additional tables or sequences?

--
With Regards,
Amit Kapila.

On Tue, Aug 6, 2024 at 9:54 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Aug 6, 2024 at 8:49 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > > > > I was reviewing the design of patch003, and I have a query. Do we need
> > > > > to even start an apply worker and create replication slot when
> > > > > subscription created is for 'sequences only'? IIUC, currently logical
> > > > > replication apply worker is the one launching sequence-sync worker
> > > > > whenever needed. I think it should be the launcher doing this job and
> > > > > thus apply worker may even not be needed for current functionality of
> > > > > sequence sync?
> > > >
> > >
> > > But that would lead to maintaining all sequence-sync of each
> > > subscription by launcher. Say there are 100 sequences per subscription
> > > and some of them from each subscription are failing due to some
> > > reasons then the launcher will be responsible for ensuring all the
> > > sequences are synced. I think it would be better to handle
> > > per-subscription work by the apply worker.
> >
> > I thought we can give that task to sequence-sync worker. Once sequence
> > sync worker is started by launcher, it keeps on syncing until all the
> > sequences are synced (even failed ones) and then exits only after all
> > are synced; instead of apply worker starting it multiple times for
> > failed sequences. Launcher to start sequence sync worker when signaled
> > by 'alter-sub refresh seq'.
> > But after going through details given by Vignesh in [1], I also see
> > the benefits of using apply worker for this task. Since apply worker
> > is already looping and doing that for table-sync, we can reuse the
> > same code for sequence sync and maintenance will be easy. So looks
> > okay if we go with existing apply worker design.
> >
>
> Fair enough. However, I was wondering whether apply_worker should exit
> after syncing all sequences for a sequence-only subscription

If apply worker exits, then on next sequence-refresh, we need a way to
wake-up launcher to start apply worker which then will start
table-sync worker. Instead, won't it be better if the launcher starts
table-sync worker directly without the need of apply worker being
present (which I stated earlier).

>  or should
> it be there for future commands that can refresh the subscription and
> add additional tables or sequences?

If we stick with apply worker starting table sync worker when needed
by continuously checking seq-sync states ('i'/'r'), then IMO, it is
better that apply-worker stays. But if we want  apply-worker to exit
and start only when needed, then why not to start sequence-sync worker
directly for seq-only subscriptions?

thanks
Shveta

On Tue, 6 Aug 2024 at 10:24, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Tue, Aug 6, 2024 at 9:54 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Aug 6, 2024 at 8:49 AM shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > > > > I was reviewing the design of patch003, and I have a query. Do we need
> > > > > > to even start an apply worker and create replication slot when
> > > > > > subscription created is for 'sequences only'? IIUC, currently logical
> > > > > > replication apply worker is the one launching sequence-sync worker
> > > > > > whenever needed. I think it should be the launcher doing this job and
> > > > > > thus apply worker may even not be needed for current functionality of
> > > > > > sequence sync?
> > > > >
> > > >
> > > > But that would lead to maintaining all sequence-sync of each
> > > > subscription by launcher. Say there are 100 sequences per subscription
> > > > and some of them from each subscription are failing due to some
> > > > reasons then the launcher will be responsible for ensuring all the
> > > > sequences are synced. I think it would be better to handle
> > > > per-subscription work by the apply worker.
> > >
> > > I thought we can give that task to sequence-sync worker. Once sequence
> > > sync worker is started by launcher, it keeps on syncing until all the
> > > sequences are synced (even failed ones) and then exits only after all
> > > are synced; instead of apply worker starting it multiple times for
> > > failed sequences. Launcher to start sequence sync worker when signaled
> > > by 'alter-sub refresh seq'.
> > > But after going through details given by Vignesh in [1], I also see
> > > the benefits of using apply worker for this task. Since apply worker
> > > is already looping and doing that for table-sync, we can reuse the
> > > same code for sequence sync and maintenance will be easy. So looks
> > > okay if we go with existing apply worker design.
> > >
> >
> > Fair enough. However, I was wondering whether apply_worker should exit
> > after syncing all sequences for a sequence-only subscription
>
> If apply worker exits, then on next sequence-refresh, we need a way to
> wake-up launcher to start apply worker which then will start
> table-sync worker. Instead, won't it be better if the launcher starts
> table-sync worker directly without the need of apply worker being
> present (which I stated earlier).

I favour the current design because it ensures the system remains
extendable for future incremental sequence synchronization. If the
launcher were responsible for starting the sequence sync worker, it
would add extra load that could hinder its ability to service other
subscriptions and complicate the design for supporting incremental
sync of sequences. Additionally, this approach offers the other
benefits mentioned in [1].

> >  or should
> > it be there for future commands that can refresh the subscription and
> > add additional tables or sequences?
>
> If we stick with apply worker starting table sync worker when needed
> by continuously checking seq-sync states ('i'/'r'), then IMO, it is
> better that apply-worker stays. But if we want  apply-worker to exit
> and start only when needed, then why not to start sequence-sync worker
> directly for seq-only subscriptions?

There is a risk that sequence synchronization might fail if the
sequence value from the publisher falls outside the defined minvalue
or maxvalue range. The apply worker must be active to determine
whether to initiate the sequence sync worker after the
wal_retrieve_retry_interval period. Typically, publications consisting
solely of sequences are uncommon. However, if a user wishes to use
such publications, they can disable the subscription if necessary and
re-enable it when a sequence refresh is needed.

[1] - https://www.postgresql.org/message-id/CALDaNm1KO8f3Fj%2BRHHXM%3DUSGwOcW242M1jHee%3DX_chn2ToiCpw%40mail.gmail.com

Regards,
Vignesh

Here are some review comments for the patch v20240805_2-0003.

======
doc/src/sgml/catalogs.sgml

nitpick - removed the word "either"

======
doc/src/sgml/ref/alter_subscription.sgml

I felt the discussions about "how to handle warnings" are a bit scattered:
e.g.1 - ALTER SUBSCRIPTION REFRESH PUBLICATION copy data referred to
CREATE SUBSCRIPTION copy data.
e.g.2 - ALTER SUBSCRIPTION REFRESH explains what to do, but now the
explanation is in 2 places.
e.g.3 - CREATE SUBSCRIPTION copy data explains what to do (again), but
IMO it belongs better in the common "Notes" part

FYI, I've moved all the information to one place (in the CREATE
SUBSCRIPTION "Notes") and others refer to this central place. See the
attached nitpicks diff.

REFRESH PUBLICATION copy_data
nitpick - now refers to  CREATE SUBSCRIPTION "Notes". I also moved it
to be nearer to the other sequence stuff.

REFRESH PUBLICATION SEQUENCES:
nitpick - now refers to CREATE SUBSCRIPTION "Notes".

======
doc/src/sgml/ref/create_subscription.sgml

REFRESH PUBLICATION copy_data
nitpick - now refers to CREATE SUBSCRIPTION "Notes"

Notes:
nitpick - the explanation of, and what to do about sequence WARNINGS,
is moved to here

======
src/backend/commands/sequence.c

pg_sequence_state:
nitpick - I just moved the comment in pg_sequence_state() to below the
NOTE, which talks about "page LSN".

======
src/backend/catalog/pg_subscription.c

1. HasSubscriptionRelations

Should function 'HasSubscriptionRelations' be renamed to
'HasSubscriptionTables'?

~~~

GetSubscriptionRelations:
nitpick - tweak some "skip" comments.

======
src/backend/commands/subscriptioncmds.c

2. CreateSubscription

  tables = fetch_table_list(wrconn, publications);
- foreach(lc, tables)
+ foreach_ptr(RangeVar, rv, tables)
+ {
+ Oid relid;
+
+ relid = RangeVarGetRelid(rv, AccessShareLock, false);
+
+ /* Check for supported relkind. */
+ CheckSubscriptionRelkind(get_rel_relkind(relid),
+ rv->schemaname, rv->relname);
+
+ AddSubscriptionRelState(subid, relid, table_state,
+ InvalidXLogRecPtr, true);
+ }
+
+ /* Add the sequences in init state */
+ sequences = fetch_sequence_list(wrconn, publications);
+ foreach_ptr(RangeVar, rv, sequences)

These 2 loops (first for tables and then for sequences) seem to be
executing the same code. If you wanted you could combine the lists
up-front, and then have one code loop instead of 2. It would mean less
code. OTOH, maybe the current code is more readable? I am not sure
what is best, so just bringing this to your attention.

~~~

AlterSubscription_refresh:
nitpick = typo /indicating tha/indicating that/

~~~

3. fetch_sequence_list

+ appendStringInfoString(&cmd, "SELECT DISTINCT n.nspname, c.relname,
s.seqtypid, s.seqmin, s.seqmax, s.seqstart, s.seqincrement,
s.seqcycle"
+    " FROM pg_publication p, LATERAL
pg_get_publication_sequences(p.pubname::text) gps(relid),"
+    " pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace JOIN
pg_sequence s ON c.oid = s.seqrelid"
+    " WHERE c.oid = gps.relid AND p.pubname IN (");
+ get_publications_str(publications, &cmd, true);
+ appendStringInfoChar(&cmd, ')');

Please wrap this better to make the SQL more readable.

~~

4.
+ if (seqform->seqtypid != seqtypid || seqform->seqmin != seqmin ||
+ seqform->seqmax != seqmax || seqform->seqstart != seqstart ||
+ seqform->seqincrement != seqincrement ||
+ seqform->seqcycle != seqcycle)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("Sequence option in remote and local is not same for \"%s.%s\"",
+    get_namespace_name(get_rel_namespace(relid)), get_rel_name(relid)),
+ errhint("Alter/Re-create the sequence using the same options as in remote."));

4a.
Are these really known as "options"? Or should they be called
"sequence parameters", or something else, like "sequence attributes"?

4a.
Is there a way to give more helpful information by identifying what
was different in the log? OTOH, maybe it would become too messy if
there were multiple differences...

======
src/backend/replication/logical/launcher.c

5. logicalrep_sync_worker_count

- if (isTablesyncWorker(w) && w->subid == subid)
+ if ((isTableSyncWorker(w) || isSequenceSyncWorker(w)) &&
+ w->subid == subid)

You could micro-optimize this -- it may be more efficient to write the
condition the other way around.

SUGGESTION
if (w->subid == subid && (isTableSyncWorker(w) || isSequenceSyncWorker(w)))

======
.../replication/logical/sequencesync.c

File header comment:
nitpick - there seems a large cut/paste mistake (the first 2
paragraphs are almost the same).
nitpick - reworded with the help of Chat-GPT for slightly better
clarity. Also fixed a couple of typos.
nitpick - it mentioned MAX_SEQUENCES_SYNC_PER_BATCH several times so I
changed the wording of one of them

~~~

fetch_remote_sequence_data:
nitpick - all other params have the same name as sequence members, so
change the parameter name /lsn/page_lsn/

~

copy_sequence:
nitpick - rename var /seq_lsn/seq_page_lsn/

======
src/backend/replication/logical/tablesync.c

6. process_syncing_sequences_for_apply

+ * If a sequencesync worker is running already, there is no need to start a new
+ * one; the existing sequencesync worker will synchronize all the sequences. If
+ * there are still any sequences to be synced after the sequencesync worker
+ * exited, then a new sequencesync worker can be started in the next iteration.
+ * To prevent starting the sequencesync worker at a high frequency after a
+ * failure, we store its last failure time. We start the sync worker for the
+ * same relation after waiting at least wal_retrieve_retry_interval.

Why is it talking about "We start the sync worker for the same
relation ...". The sequencesync_failuretime is per sync worker, not
per relation. And, I don't see any 'same relation' check in the code.

======
src/include/catalog/pg_subscription_rel.h

GetSubscriptionRelations:
nitpick - changed parameter name /all_relations/all_states/

======
src/test/subscription/t/034_sequences.pl

nitpick - add some ########## comments to highlight the main test
parts to make it easier to read.
nitpick - fix typo /syned/synced/

7. More test cases?
IIUC you can also get a sequence mismatch warning during "ALTER ...
REFRESH PUBLICATION", and "CREATE SUBSCRIPTION". So, should those be
tested also?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Mon, 5 Aug 2024 at 18:05, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Aug 5, 2024 at 11:04 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, 31 Jul 2024 at 14:39, shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > On Mon, Jun 10, 2024 at 5:00 PM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > On Mon, 10 Jun 2024 at 12:24, Amul Sul <sulamul@gmail.com> wrote:
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Jun 8, 2024 at 6:43 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > >>
> > > > >> On Wed, 5 Jun 2024 at 14:11, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >> [...]
> > > > >> A new catalog table, pg_subscription_seq, has been introduced for
> > > > >> mapping subscriptions to sequences. Additionally, the sequence LSN
> > > > >> (Log Sequence Number) is stored, facilitating determination of
> > > > >> sequence changes occurring before or after the returned sequence
> > > > >> state.
> > > > >
> > > > >
> > > > > Can't it be done using pg_depend? It seems a bit excessive unless I'm missing
> > > > > something.
> > > >
> > > > We'll require the lsn because the sequence LSN informs the user that
> > > > it has been synchronized up to the LSN in pg_subscription_seq. Since
> > > > we are not supporting incremental sync, the user will be able to
> > > > identify if he should run refresh sequences or not by checking the lsn
> > > > of the pg_subscription_seq and the lsn of the sequence(using
> > > > pg_sequence_state added) in the publisher.
> > >
> > > How the user will know from seq's lsn that he needs to run refresh.
> > > lsn indicates page_lsn and thus the sequence might advance on pub
> > > without changing lsn and thus lsn may look the same on subscriber even
> > > though a sequence-refresh is needed. Am I missing something here?
> >
> > When a sequence is synchronized to the subscriber, the page LSN of the
> > sequence from the publisher is also retrieved and stored in
> > pg_subscriber_rel as shown below:
> > --- Publisher page lsn
> > publisher=# select pg_sequence_state('seq1');
> >  pg_sequence_state
> > --------------------
> >  (0/1510E38,65,1,t)
> > (1 row)
> >
> > --- Subscriber stores the publisher's page lsn for the sequence
> > subscriber=# select * from pg_subscription_rel where srrelid = 16384;
> >  srsubid | srrelid | srsubstate | srsublsn
> > ---------+---------+------------+-----------
> >    16389 |   16384 | r          | 0/1510E38
> > (1 row)
> >
> > If changes are made to the sequence, such as performing many nextvals,
> > the page LSN will be updated. Currently the sequence values are
> > prefetched for SEQ_LOG_VALS 32, so the lsn will not get updated for
> > the prefetched values, once the prefetched values are consumed the lsn
> > will get updated.
> > For example:
> > --- Updated LSN on the publisher (old lsn - 0/1510E38, new lsn - 0/1558CA8)
> > publisher=# select pg_sequence_state('seq1');
> >   pg_sequence_state
> > ----------------------
> >  (0/1558CA8,143,22,t)
> > (1 row)
> >
> > The user can then compare this updated value with the sequence's LSN
> > in pg_subscription_rel to determine when to re-synchronize the
> > sequence.
>
> Thanks for the details. But I was referring to the case where we are
> in between pre-fetched values on publisher (say at 25th value), while
> on subscriber we are slightly behind (say at 15th value), but page-lsn
> will be the same on both. Since the subscriber is behind, a
> sequence-refresh is needed on sub, but by looking at lsn (which is
> same), one can not say that for sure.  Let me know if I have
> misunderstood it.

Yes, at present, if the value is within the pre-fetched range, we
cannot distinguish it solely using the page_lsn. However, the
pg_sequence_state function also provides last_value and log_cnt, which
can be used to handle these specific cases.

Regards,
Vignesh

On Tue, Aug 6, 2024 at 5:13 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, 5 Aug 2024 at 18:05, shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Mon, Aug 5, 2024 at 11:04 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Wed, 31 Jul 2024 at 14:39, shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > On Mon, Jun 10, 2024 at 5:00 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > >
> > > > > On Mon, 10 Jun 2024 at 12:24, Amul Sul <sulamul@gmail.com> wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Jun 8, 2024 at 6:43 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > > >>
> > > > > >> On Wed, 5 Jun 2024 at 14:11, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > >> [...]
> > > > > >> A new catalog table, pg_subscription_seq, has been introduced for
> > > > > >> mapping subscriptions to sequences. Additionally, the sequence LSN
> > > > > >> (Log Sequence Number) is stored, facilitating determination of
> > > > > >> sequence changes occurring before or after the returned sequence
> > > > > >> state.
> > > > > >
> > > > > >
> > > > > > Can't it be done using pg_depend? It seems a bit excessive unless I'm missing
> > > > > > something.
> > > > >
> > > > > We'll require the lsn because the sequence LSN informs the user that
> > > > > it has been synchronized up to the LSN in pg_subscription_seq. Since
> > > > > we are not supporting incremental sync, the user will be able to
> > > > > identify if he should run refresh sequences or not by checking the lsn
> > > > > of the pg_subscription_seq and the lsn of the sequence(using
> > > > > pg_sequence_state added) in the publisher.
> > > >
> > > > How the user will know from seq's lsn that he needs to run refresh.
> > > > lsn indicates page_lsn and thus the sequence might advance on pub
> > > > without changing lsn and thus lsn may look the same on subscriber even
> > > > though a sequence-refresh is needed. Am I missing something here?
> > >
> > > When a sequence is synchronized to the subscriber, the page LSN of the
> > > sequence from the publisher is also retrieved and stored in
> > > pg_subscriber_rel as shown below:
> > > --- Publisher page lsn
> > > publisher=# select pg_sequence_state('seq1');
> > >  pg_sequence_state
> > > --------------------
> > >  (0/1510E38,65,1,t)
> > > (1 row)
> > >
> > > --- Subscriber stores the publisher's page lsn for the sequence
> > > subscriber=# select * from pg_subscription_rel where srrelid = 16384;
> > >  srsubid | srrelid | srsubstate | srsublsn
> > > ---------+---------+------------+-----------
> > >    16389 |   16384 | r          | 0/1510E38
> > > (1 row)
> > >
> > > If changes are made to the sequence, such as performing many nextvals,
> > > the page LSN will be updated. Currently the sequence values are
> > > prefetched for SEQ_LOG_VALS 32, so the lsn will not get updated for
> > > the prefetched values, once the prefetched values are consumed the lsn
> > > will get updated.
> > > For example:
> > > --- Updated LSN on the publisher (old lsn - 0/1510E38, new lsn - 0/1558CA8)
> > > publisher=# select pg_sequence_state('seq1');
> > >   pg_sequence_state
> > > ----------------------
> > >  (0/1558CA8,143,22,t)
> > > (1 row)
> > >
> > > The user can then compare this updated value with the sequence's LSN
> > > in pg_subscription_rel to determine when to re-synchronize the
> > > sequence.
> >
> > Thanks for the details. But I was referring to the case where we are
> > in between pre-fetched values on publisher (say at 25th value), while
> > on subscriber we are slightly behind (say at 15th value), but page-lsn
> > will be the same on both. Since the subscriber is behind, a
> > sequence-refresh is needed on sub, but by looking at lsn (which is
> > same), one can not say that for sure.  Let me know if I have
> > misunderstood it.
>
> Yes, at present, if the value is within the pre-fetched range, we
> cannot distinguish it solely using the page_lsn.
>

This makes sense to me.

>
> However, the
> pg_sequence_state function also provides last_value and log_cnt, which
> can be used to handle these specific cases.
>

BTW, can we document all these steps for users to know when to refresh
the sequences, if not already documented?

--
With Regards,
Amit Kapila.

Hi Vignesh,

This is mostly a repeat of my previous mail from a while ago [1] but
includes some corrections, answers, and more examples. I'm going to
try to persuade one last time because the current patch is becoming
stable, so I wanted to revisit this syntax proposal before it gets too
late to change anything.

If there is some problem with the proposed idea please let me know
because I can see only the advantages and no disadvantages of doing it
this way.

~~~

The current patchset offers two forms of subscription refresh:
1. ALTER SUBSCRIPTION name REFRESH PUBLICATION [ WITH ( refresh_option
[= value] [, ... ] ) ]
2. ALTER SUBSCRIPTION name REFRESH PUBLICATION SEQUENCES

Since 'copy_data' is the only supported refresh_option, really it is more like:
1. ALTER SUBSCRIPTION name REFRESH PUBLICATION [ WITH ( copy_data [=
true|false] ) ]
2. ALTER SUBSCRIPTION name REFRESH PUBLICATION SEQUENCES

~~~

I proposed previously that instead of having 2 commands for refreshing
subscriptions we should have a single refresh command:

ALTER SUBSCRIPTION name REFRESH PUBLICATION [TABLES|SEQUENCES] [ WITH
( copy_data [= true|false] ) ]

Why?

- IMO it is less confusing than having 2 commands that both refresh
sequences in slightly different ways

- It is more flexible because apart from refreshing everything, a user
can choose to refresh only tables or only sequences if desired; IMO
more flexibility is always good.

- There is no loss of functionality from the current implementation
AFAICT. You can still say "ALTER SUBSCRIPTION sub REFRESH PUBLICATION
SEQUENCES" exactly the same as the patchset allows.

- The implementation code will become simpler. For example, the
current implementation of AlterSubscription_refresh(...) includes the
(hacky?) 'resync_all_sequences' parameter and has an overcomplicated
relationship with other parameters as demonstrated by the assertions
below. IMO using the prosed syntax means this coding will become not
only simpler, but shorter too.
+ /* resync_all_sequences cannot be specified with refresh_tables */
+ Assert(!(resync_all_sequences && refresh_tables));
+
+ /* resync_all_sequences cannot be specified with copy_data as false */
+ Assert(!(resync_all_sequences && !copy_data));

~~~

So, to continue this proposal, let the meaning of 'copy_data' for
SEQUENCES be as follows:

- when copy_data == false: it means don't copy data (i.e. don't
synchronize anything). Add/remove sequences from pg_subscriber_rel as
needed.

- when copy_data == true: it means to copy data (i.e. synchronize) for
all sequences. Add/remove sequences from pg_subscriber_rel as needed)


~~~

EXAMPLES using the proposed syntax:

Refreshing TABLES only...

ex1.
ALTER SUBSCRIPTION sub REFRESH PUBLICATION TABLES WITH (copy_data = false)
- same as PG17 functionality for "ALTER SUBSCRIPTION sub REFRESH
PUBLICATION WITH (copy_data = false)"

ex2.
ALTER SUBSCRIPTION sub REFRESH PUBLICATION TABLES WITH (copy_data = true)
- same as PG17 functionality for "ALTER SUBSCRIPTION sub REFRESH
PUBLICATION WITH (copy_data = true)"

ex3. (using default copy_data)
ALTER SUBSCRIPTION sub REFRESH PUBLICATION TABLES
- same as ex2.

~

Refreshing SEQUENCES only...

ex4.
ALTER SUBSCRIPTION sub REFRESH PUBLICATION SEQUENCES WITH (copy data = false)
- this adds/removes only sequences to pg_subscription_rel but doesn't
update the sequence values

ex5.
ALTER SUBSCRIPTION sub REFRESH PUBLICATION SEQUENCES WITH (copy data = true)
- this adds/removes only sequences to pg_subscription_rel and also
updates (synchronizes) all sequence values.
- same functionality as "ALTER SUBSCRIPTION sub REFRESH PUBLICATION
SEQUENCES" in your current patchset

ex6. (using default copy_data)
ALTER SUBSCRIPTION sub REFRESH PUBLICATION SEQUENCES
- same as ex5.
- note, that this command has the same syntax and functionality as the
current patchset

~~~

When no object_type is specified it has intuitive meaning to refresh
both TABLES and SEQUENCES...

ex7.
ALTER SUBSCRIPTION sub REFRESH PUBLICATION WITH (copy_data = false)
- For tables, it is the same as the PG17 functionality
- For sequences it includes the same behaviour of ex4.

ex8.
ALTER SUBSCRIPTION sub REFRESH PUBLICATION WITH (copy_data = true)
- For tables, it is the same as the PG17 functionality
- For sequences it includes the same behaviour of ex5.
- There is one subtle difference from the current patchset because
this proposal will synchronize *all* sequences instead of only new
ones. But, this is a good thing. The current documentation is
complicated by having to explain the differences between REFRESH
PUBLICATION and REFRESH PUBLICATION SEQUENCES. The current patchset
also raises questions like how the user chooses whether to use
"REFRESH PUBLICATION SEQUENCES" versus "REFRESH PUBLICATION WITH
(copy_data=true)". OTHO, the proposed syntax eliminates ambiguity.

ex9. (using default copy_data)
ALTER SUBSCRIPTION sub REFRESH PUBLICATION
- same as ex8

======
[1] https://www.postgresql.org/message-id/CAHut%2BPuFH1OCj-P1UKoRQE2X4-0zMG%2BN1V7jdn%3DtOQV4RNbAbw%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

On Mon, Aug 5, 2024 at 10:26 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 1 Aug 2024 at 04:25, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Vignesh,
> >
> > I noticed that when replicating sequences (using the latest patches
> > 0730_2*)  the subscriber-side checks the *existence* of the sequence,
> > but apparently it is not checking other sequence attributes.
> >
> > For example, consider:
> >
> > Publisher: "CREATE SEQUENCE s1 START 1 INCREMENT 2;" should be a
> > sequence of only odd numbers.
> > Subscriber: "CREATE SEQUENCE s1 START 2 INCREMENT 2;" should be a
> > sequence of only even numbers.
> >
> > Because the names match, currently the patch allows replication of the
> > s1 sequence. I think that might lead to unexpected results on the
> > subscriber. IMO it might be safer to report ERROR unless the sequences
> > match properly (i.e. not just a name check).
> >
> > Below is a demonstration the problem:
> >
> > ==========
> > Publisher:
> > ==========
> >
> > (publisher sequence is odd numbers)
> >
> > test_pub=# create sequence s1 start 1 increment 2;
> > CREATE SEQUENCE
> > test_pub=# select * from nextval('s1');
> >  nextval
> > ---------
> >        1
> > (1 row)
> >
> > test_pub=# select * from nextval('s1');
> >  nextval
> > ---------
> >        3
> > (1 row)
> >
> > test_pub=# select * from nextval('s1');
> >  nextval
> > ---------
> >        5
> > (1 row)
> >
> > test_pub=# CREATE PUBLICATION pub1 FOR ALL SEQUENCES;
> > CREATE PUBLICATION
> > test_pub=#
> >
> > ==========
> > Subscriber:
> > ==========
> >
> > (subscriber sequence is even numbers)
> >
> > test_sub=# create sequence s1 start 2 increment 2;
> > CREATE SEQUENCE
> > test_sub=# SELECT * FROM nextval('s1');
> >  nextval
> > ---------
> >        2
> > (1 row)
> >
> > test_sub=# SELECT * FROM nextval('s1');
> >  nextval
> > ---------
> >        4
> > (1 row)
> >
> > test_sub=# SELECT * FROM nextval('s1');
> >  nextval
> > ---------
> >        6
> > (1 row)
> >
> > test_sub=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=test_pub'
> > PUBLICATION pub1;
> > 2024-08-01 08:43:04.198 AEST [24325] WARNING:  subscriptions created
> > by regression test cases should have names starting with "regress_"
> > WARNING:  subscriptions created by regression test cases should have
> > names starting with "regress_"
> > NOTICE:  created replication slot "sub1" on publisher
> > CREATE SUBSCRIPTION
> > test_sub=# 2024-08-01 08:43:04.294 AEST [26240] LOG:  logical
> > replication apply worker for subscription "sub1" has started
> > 2024-08-01 08:43:04.309 AEST [26244] LOG:  logical replication
> > sequence synchronization worker for subscription "sub1" has started
> > 2024-08-01 08:43:04.323 AEST [26244] LOG:  logical replication
> > synchronization for subscription "sub1", sequence "s1" has finished
> > 2024-08-01 08:43:04.323 AEST [26244] LOG:  logical replication
> > sequence synchronization worker for subscription "sub1" has finished
> >
> > (after the CREATE SUBSCRIPTION we are getting replicated odd values
> > from the publisher, even though the subscriber side sequence was
> > supposed to be even numbers)
> >
> > test_sub=# SELECT * FROM nextval('s1');
> >  nextval
> > ---------
> >        7
> > (1 row)
> >
> > test_sub=# SELECT * FROM nextval('s1');
> >  nextval
> > ---------
> >        9
> > (1 row)
> >
> > test_sub=# SELECT * FROM nextval('s1');
> >  nextval
> > ---------
> >       11
> > (1 row)
> >
> > (Looking at the description you would expect odd values for this
> > sequence to be impossible)

I see that for such even sequences, user can still do 'setval'  to a
odd number and then nextval will keep on returning odd value.

postgres=# SELECT nextval('s1');
       6

postgres=SELECT setval('s1', 43);
     43

postgres=# SELECT nextval('s1');
     45

> > test_sub=# \dS+ s1
> >                              Sequence "public.s1"
> >   Type  | Start | Minimum |       Maximum       | Increment | Cycles? | Cache
> > --------+-------+---------+---------------------+-----------+---------+-------
> >  bigint |     2 |       1 | 9223372036854775807 |         2 | no      |     1
>
> Even if we check the sequence definition during the CREATE
> SUBSCRIPTION/ALTER SUBSCRIPTION ... REFRESH PUBLICATION or ALTER
> SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES commands, there's still
> a chance that the sequence definition might change after the command
> has been executed. Currently, there's no mechanism to lock a sequence,
> and we also permit replication of table data even if the table
> structures differ, such as mismatched data types like int and
> smallint. I have modified it to log a warning to inform users that the
> sequence options on the publisher and subscriber are not the same and
> advise them to ensure that the sequence definitions are consistent
> between both.
> The v20240805 version patch attached at [1] has the changes for the same.
> [1] - https://www.postgresql.org/message-id/CALDaNm1Y_ot-jFRfmtwDuwmFrgSSYHjVuy28RspSopTtwzXy8w%40mail.gmail.com

The behavior for applying is no different from setval. Having said
that, I agree that sequence definition can change even after the
subscription creation, but earlier we were not syncing sequences and
thus the value of a particular sequence was going to remain in the
range/pattern defined by its attributes unless user sets it manually
using setval. But now, it is being changed in the background without
user's knowledge.
The table case is different. In case of table replication, if we have
CHECK constraint or say primary-key etc, then the value which violates
these constraints will never be  inserted to a table even during
replication on sub. For sequences, parameters (MIN,MAX, START,
INCREMENT) can be considered similar to check-constraints, the only
difference is during apply, we are still overriding these and copying
pub's value. May be such inconsistencies detection can be targeted
later in next project. But for the time being, it will be good to add
a 'caveat' section in doc mentioning all such cases. The scope of this
project should be clearly documented.

thanks
Shveta

On Tue, 6 Aug 2024 at 14:38, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Here are some review comments for the patch v20240805_2-0003.
>
> 4a.
> Is there a way to give more helpful information by identifying what
> was different in the log? OTOH, maybe it would become too messy if
> there were multiple differences...

I had considered this while implementing and did not implement it to
print each of the parameters because it will be too messy. I felt
existing is better.

>
> 7. More test cases?
> IIUC you can also get a sequence mismatch warning during "ALTER ...
> REFRESH PUBLICATION", and "CREATE SUBSCRIPTION". So, should those be
> tested also?

Since it won't add any extra coverage, I feel no need to add this test.

The remaining comments have been addressed, and the changes are
included in the attached v20240807 version patch.

Regards,
Vignesh

On Tue, 6 Aug 2024 at 09:28, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Tue, Aug 6, 2024 at 8:49 AM shveta malik <shveta.malik@gmail.com> wrote:
> >
>
> Do we need some kind of coordination between table sync and sequence
> sync for internally generated sequences? Lets say we have an identity
> column with a 'GENERATED ALWAYS' sequence. When the sequence is synced
> to subscriber, subscriber can also do an insert to table (extra one)
> incrementing the sequence and then when publisher performs an insert,
> apply worker will blindly copy that row to sub's table making identity
> column's duplicate entries.
>
> CREATE TABLE color ( color_id INT GENERATED ALWAYS AS
> IDENTITY,color_name VARCHAR NOT NULL);
>
> Pub: insert into color(color_name) values('red');
>
> Sub: perform sequence refresh and check 'r' state is reached, then do insert:
> insert into color(color_name) values('yellow');
>
> Pub: insert into color(color_name) values('blue');
>
> After above, data on Pub: (1, 'red') ;(2, 'blue'),
>
> After above, data on Sub: (1, 'red') ;(2, 'yellow'); (2, 'blue'),
>
> Identity column has duplicate values. Should the apply worker error
> out while inserting such a  row to the table? Or it is not in the
> scope of this project?

This behavior is documented at [1]:
Sequence data is not replicated. The data in serial or identity
columns backed by sequences will of course be replicated as part of
the table, but the sequence itself would still show the start value on
the subscriber.

This behavior is because of the above logical replication restriction.
So the behavior looks ok to me and I feel this is not part of the
scope of this project. I have updated this documentation section here
to mention sequences can be updated using ALTER SEQUENCE ... REFRESH
PUBLICATION SEQUENCES at v20240807 version patch attached at [2].

[1] - https://www.postgresql.org/docs/devel/logical-replication-restrictions.html
[2] - https://www.postgresql.org/message-id/CALDaNm01Z6Oo9osGMFTOoyTR1kVoyh1rEvZ%2B6uJn-ZymV%3D0dbQ%40mail.gmail.com

Regards,
Vignesh

On Mon, 5 Aug 2024 at 17:28, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Aug 5, 2024 at 2:36 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Fri, 2 Aug 2024 at 14:24, shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > On Thu, Aug 1, 2024 at 9:26 AM shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > On Mon, Jul 29, 2024 at 4:17 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > >
> > > > > Thanks for reporting this, these issues are fixed in the attached
> > > > > v20240730_2 version patch.
> > > > >
> > >
> > > I was reviewing the design of patch003, and I have a query. Do we need
> > > to even start an apply worker and create replication slot when
> > > subscription created is for 'sequences only'? IIUC, currently logical
> > > replication apply worker is the one launching sequence-sync worker
> > > whenever needed. I think it should be the launcher doing this job and
> > > thus apply worker may even not be needed for current functionality of
> > > sequence sync?
> >
>
> But that would lead to maintaining all sequence-sync of each
> subscription by launcher. Say there are 100 sequences per subscription
> and some of them from each subscription are failing due to some
> reasons then the launcher will be responsible for ensuring all the
> sequences are synced. I think it would be better to handle
> per-subscription work by the apply worker.
>
> >
> > Going forward when we implement incremental sync of
> > > sequences, then we may have apply worker started but now it is not
> > > needed.
> >
> > I believe the current method of having the apply worker initiate the
> > sequence sync worker is advantageous for several reasons:
> > a) Reduces Launcher Load: This approach prevents overloading the
> > launcher, which must handle various other subscription requests.
> > b) Facilitates Incremental Sync: It provides a more straightforward
> > path to extend support for incremental sequence synchronization.
> > c) Reuses Existing Code: It leverages the existing tablesync worker
> > code for starting the tablesync process, avoiding the need to
> > duplicate code in the launcher.
> > d) Simplified Code Maintenance: Centralizing sequence synchronization
> > logic within the apply worker can simplify code maintenance and
> > updates, as changes will only need to be made in one place rather than
> > across multiple components.
> > e) Better Monitoring and Debugging: With sequence synchronization
> > being handled by the apply worker, you can more effectively monitor
> > and debug synchronization processes since all related operations are
> > managed by a single component.
> >
> > Also, I noticed that even when a publication has no tables, we create
> > replication slot and start apply worker.
> >
>
> As far as I understand slots and origins are primarily required for
> incremental sync. Would it be used only for sequence-sync cases? If
> not then we can avoid creating those. I agree that it would add some
> complexity to the code with sequence-specific checks, so we can create
> a top-up patch for this if required and evaluate its complexity versus
> the benefit it produces.

I have added a XXX todo comments in the v20240807 version patch
attached at [1]. I will handle this as a separate patch once the
current patch is stable.
[1] - https://www.postgresql.org/message-id/CALDaNm01Z6Oo9osGMFTOoyTR1kVoyh1rEvZ%2B6uJn-ZymV%3D0dbQ%40mail.gmail.com

Regards,
Vignesh

On Wed, 7 Aug 2024 at 08:09, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Aug 6, 2024 at 5:13 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Mon, 5 Aug 2024 at 18:05, shveta malik <shveta.malik@gmail.com> wrote:
> > >
> > > On Mon, Aug 5, 2024 at 11:04 AM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > On Wed, 31 Jul 2024 at 14:39, shveta malik <shveta.malik@gmail.com> wrote:
> > > > >
> > > > > On Mon, Jun 10, 2024 at 5:00 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > > >
> > > > > > On Mon, 10 Jun 2024 at 12:24, Amul Sul <sulamul@gmail.com> wrote:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Jun 8, 2024 at 6:43 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > > > >>
> > > > > > >> On Wed, 5 Jun 2024 at 14:11, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > > >> [...]
> > > > > > >> A new catalog table, pg_subscription_seq, has been introduced for
> > > > > > >> mapping subscriptions to sequences. Additionally, the sequence LSN
> > > > > > >> (Log Sequence Number) is stored, facilitating determination of
> > > > > > >> sequence changes occurring before or after the returned sequence
> > > > > > >> state.
> > > > > > >
> > > > > > >
> > > > > > > Can't it be done using pg_depend? It seems a bit excessive unless I'm missing
> > > > > > > something.
> > > > > >
> > > > > > We'll require the lsn because the sequence LSN informs the user that
> > > > > > it has been synchronized up to the LSN in pg_subscription_seq. Since
> > > > > > we are not supporting incremental sync, the user will be able to
> > > > > > identify if he should run refresh sequences or not by checking the lsn
> > > > > > of the pg_subscription_seq and the lsn of the sequence(using
> > > > > > pg_sequence_state added) in the publisher.
> > > > >
> > > > > How the user will know from seq's lsn that he needs to run refresh.
> > > > > lsn indicates page_lsn and thus the sequence might advance on pub
> > > > > without changing lsn and thus lsn may look the same on subscriber even
> > > > > though a sequence-refresh is needed. Am I missing something here?
> > > >
> > > > When a sequence is synchronized to the subscriber, the page LSN of the
> > > > sequence from the publisher is also retrieved and stored in
> > > > pg_subscriber_rel as shown below:
> > > > --- Publisher page lsn
> > > > publisher=# select pg_sequence_state('seq1');
> > > >  pg_sequence_state
> > > > --------------------
> > > >  (0/1510E38,65,1,t)
> > > > (1 row)
> > > >
> > > > --- Subscriber stores the publisher's page lsn for the sequence
> > > > subscriber=# select * from pg_subscription_rel where srrelid = 16384;
> > > >  srsubid | srrelid | srsubstate | srsublsn
> > > > ---------+---------+------------+-----------
> > > >    16389 |   16384 | r          | 0/1510E38
> > > > (1 row)
> > > >
> > > > If changes are made to the sequence, such as performing many nextvals,
> > > > the page LSN will be updated. Currently the sequence values are
> > > > prefetched for SEQ_LOG_VALS 32, so the lsn will not get updated for
> > > > the prefetched values, once the prefetched values are consumed the lsn
> > > > will get updated.
> > > > For example:
> > > > --- Updated LSN on the publisher (old lsn - 0/1510E38, new lsn - 0/1558CA8)
> > > > publisher=# select pg_sequence_state('seq1');
> > > >   pg_sequence_state
> > > > ----------------------
> > > >  (0/1558CA8,143,22,t)
> > > > (1 row)
> > > >
> > > > The user can then compare this updated value with the sequence's LSN
> > > > in pg_subscription_rel to determine when to re-synchronize the
> > > > sequence.
> > >
> > > Thanks for the details. But I was referring to the case where we are
> > > in between pre-fetched values on publisher (say at 25th value), while
> > > on subscriber we are slightly behind (say at 15th value), but page-lsn
> > > will be the same on both. Since the subscriber is behind, a
> > > sequence-refresh is needed on sub, but by looking at lsn (which is
> > > same), one can not say that for sure.  Let me know if I have
> > > misunderstood it.
> >
> > Yes, at present, if the value is within the pre-fetched range, we
> > cannot distinguish it solely using the page_lsn.
> >
>
> This makes sense to me.
>
> >
> > However, the
> > pg_sequence_state function also provides last_value and log_cnt, which
> > can be used to handle these specific cases.
> >
>
> BTW, can we document all these steps for users to know when to refresh
> the sequences, if not already documented?

This has been documented in the v20240807 version attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm01Z6Oo9osGMFTOoyTR1kVoyh1rEvZ%2B6uJn-ZymV%3D0dbQ%40mail.gmail.com

Regards,
Vignesh

Hi Vignesh, Here are my v20240807-0003 review comments.

======
1. GENERAL DOCS.

IMO the replication of SEQUENCES is a big enough topic that it
deserves to have its own section in the docs chapter 31 [1].

Some of the create/alter subscription docs content would stay where it
is in, but a new chapter would just tie everything together better. It
could also serve as a better place to describe the other sequence
replication content like:
(a) getting a WARNING for mismatched sequences and how to handle it.
(b) how can the user know when a subscription refresh is required to
(re-)synchronise sequences
(c) pub/sub examples

======
doc/src/sgml/logical-replication.sgml

2. Restrictions

Sequence data is not replicated. The data in serial or identity
columns backed by sequences will of course be replicated as part of
the table, but the sequence itself would still show the start value on
the subscriber. If the subscriber is used as a read-only database,
then this should typically not be a problem. If, however, some kind of
switchover or failover to the subscriber database is intended, then
the sequences would need to be updated to the latest values, either by
executing ALTER SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES or by
copying the current data from the publisher (perhaps using pg_dump) or
by determining a sufficiently high value from the tables themselves.

~

2a.
The paragraph starts by saying "Sequence data is not replicated.". It
seems wrong now. Doesn't that need rewording or removing?

~

2b.
Should the info "If, however, some kind of switchover or failover..."
be mentioned in the "Logical Replication Failover" section [2],
instead of here?

======
doc/src/sgml/ref/alter_subscription.sgml

3.
Sequence values may occasionally become out of sync due to updates in
the publisher. To verify this, compare the
pg_subscription_rel.srsublsn on the subscriber with the page_lsn
obtained from the pg_sequence_state for the sequence on the publisher.
If the sequence is still using prefetched values, the page_lsn will
not be updated. In such cases, you will need to directly compare the
sequences and execute REFRESH PUBLICATION SEQUENCES if required.

~

3a.
This whole paragraph may be better put in the new chapter that was
suggested earlier in review comment #1.

~

3b.
Is it only "Occasionally"? I expected subscriber-side sequences could
become stale quite often.

~

3c.
Is this advice very useful? It's saying if the LSN is different then
the sequence is out of date, but if the LSN is not different then you
cannot tell. Why not ignore LSN altogether and just advise the user to
directly compare the sequences in the first place?

======

Also, there are more minor suggestions in the attached nitpicks diff.

======
[1] https://www.postgresql.org/docs/current/logical-replication.html
[2] file:///usr/local/pg_oss/share/doc/postgresql/html/logical-replication-failover.html

Kind Regards,
Peter Smith.
Fujitsu Australia

On Wed, Aug 7, 2024 at 10:12 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> This is mostly a repeat of my previous mail from a while ago [1] but
> includes some corrections, answers, and more examples. I'm going to
> try to persuade one last time because the current patch is becoming
> stable, so I wanted to revisit this syntax proposal before it gets too
> late to change anything.
>
> If there is some problem with the proposed idea please let me know
> because I can see only the advantages and no disadvantages of doing it
> this way.
>
> ~~~
>
> The current patchset offers two forms of subscription refresh:
> 1. ALTER SUBSCRIPTION name REFRESH PUBLICATION [ WITH ( refresh_option
> [= value] [, ... ] ) ]
> 2. ALTER SUBSCRIPTION name REFRESH PUBLICATION SEQUENCES
>
> Since 'copy_data' is the only supported refresh_option, really it is more like:
> 1. ALTER SUBSCRIPTION name REFRESH PUBLICATION [ WITH ( copy_data [=
> true|false] ) ]
> 2. ALTER SUBSCRIPTION name REFRESH PUBLICATION SEQUENCES
>
> ~~~
>
> I proposed previously that instead of having 2 commands for refreshing
> subscriptions we should have a single refresh command:
>
> ALTER SUBSCRIPTION name REFRESH PUBLICATION [TABLES|SEQUENCES] [ WITH
> ( copy_data [= true|false] ) ]
>
> Why?
>
> - IMO it is less confusing than having 2 commands that both refresh
> sequences in slightly different ways
>
> - It is more flexible because apart from refreshing everything, a user
> can choose to refresh only tables or only sequences if desired; IMO
> more flexibility is always good.
>
> - There is no loss of functionality from the current implementation
> AFAICT. You can still say "ALTER SUBSCRIPTION sub REFRESH PUBLICATION
> SEQUENCES" exactly the same as the patchset allows.
>
> ~~~
>
> So, to continue this proposal, let the meaning of 'copy_data' for
> SEQUENCES be as follows:
>
> - when copy_data == false: it means don't copy data (i.e. don't
> synchronize anything). Add/remove sequences from pg_subscriber_rel as
> needed.
>
> - when copy_data == true: it means to copy data (i.e. synchronize) for
> all sequences. Add/remove sequences from pg_subscriber_rel as needed)
>

I find overloading the copy_data option more confusing than adding a
new variant for REFRESH. To make it clear, we can even think of
extending the command as ALTER SUBSCRIPTION name REFRESH PUBLICATION
ALL SEQUENCES or something like that.  I don't know where there is a
need or not but one can imagine extending it as ALTER SUBSCRIPTION
name REFRESH PUBLICATION SEQUENCES [<seq_name_1>, <seq_name_2>, ..].
This will allow to selectively refresh the sequences.

--
With Regards,
Amit Kapila.

On Thu, Aug 8, 2024 at 1:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Aug 7, 2024 at 10:12 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > This is mostly a repeat of my previous mail from a while ago [1] but
> > includes some corrections, answers, and more examples. I'm going to
> > try to persuade one last time because the current patch is becoming
> > stable, so I wanted to revisit this syntax proposal before it gets too
> > late to change anything.
> >
> > If there is some problem with the proposed idea please let me know
> > because I can see only the advantages and no disadvantages of doing it
> > this way.
> >
> > ~~~
> >
> > The current patchset offers two forms of subscription refresh:
> > 1. ALTER SUBSCRIPTION name REFRESH PUBLICATION [ WITH ( refresh_option
> > [= value] [, ... ] ) ]
> > 2. ALTER SUBSCRIPTION name REFRESH PUBLICATION SEQUENCES
> >
> > Since 'copy_data' is the only supported refresh_option, really it is more like:
> > 1. ALTER SUBSCRIPTION name REFRESH PUBLICATION [ WITH ( copy_data [=
> > true|false] ) ]
> > 2. ALTER SUBSCRIPTION name REFRESH PUBLICATION SEQUENCES
> >
> > ~~~
> >
> > I proposed previously that instead of having 2 commands for refreshing
> > subscriptions we should have a single refresh command:
> >
> > ALTER SUBSCRIPTION name REFRESH PUBLICATION [TABLES|SEQUENCES] [ WITH
> > ( copy_data [= true|false] ) ]
> >
> > Why?
> >
> > - IMO it is less confusing than having 2 commands that both refresh
> > sequences in slightly different ways
> >
> > - It is more flexible because apart from refreshing everything, a user
> > can choose to refresh only tables or only sequences if desired; IMO
> > more flexibility is always good.
> >
> > - There is no loss of functionality from the current implementation
> > AFAICT. You can still say "ALTER SUBSCRIPTION sub REFRESH PUBLICATION
> > SEQUENCES" exactly the same as the patchset allows.
> >
> > ~~~
> >
> > So, to continue this proposal, let the meaning of 'copy_data' for
> > SEQUENCES be as follows:
> >
> > - when copy_data == false: it means don't copy data (i.e. don't
> > synchronize anything). Add/remove sequences from pg_subscriber_rel as
> > needed.
> >
> > - when copy_data == true: it means to copy data (i.e. synchronize) for
> > all sequences. Add/remove sequences from pg_subscriber_rel as needed)
> >
>
> I find overloading the copy_data option more confusing than adding a
> new variant for REFRESH. To make it clear, we can even think of
> extending the command as ALTER SUBSCRIPTION name REFRESH PUBLICATION
> ALL SEQUENCES or something like that.  I don't know where there is a
> need or not but one can imagine extending it as ALTER SUBSCRIPTION
> name REFRESH PUBLICATION SEQUENCES [<seq_name_1>, <seq_name_2>, ..].
> This will allow to selectively refresh the sequences.
>

But, I haven't invented a new overloading for "copy_data" option
(meaning "synchronize") for sequences. The current patchset already
interprets copy_data exactly this way.

For example, below are patch 0003 results:

ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION WITH (copy_data=false)
- this will add/remove new sequences in pg_subscription_rel, but it
will *not* synchronize the new sequence

ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION WITH (copy_data=true)
- this will add/remove new sequences in pg_subscription_rel, and it
*will* synchronize the new sequence

~

I only proposed that copy_data should apply to *all* sequences, not
just new ones.

======
Kind Regards.
Peter Smith.
Fujitsu Australia.

On Wed, Aug 7, 2024 at 1:45 PM vignesh C <vignesh21@gmail.com> wrote:
>
>
> The remaining comments have been addressed, and the changes are
> included in the attached v20240807 version patch.

Thanks for addressing the comment. Please find few comments for v20240807 :

patch002:
1)
create_publication.sgml:

--I think it will be good to add another example for both tables and sequences:
CREATE PUBLICATION all_sequences FOR ALL TABLES, SEQUENCES;
I was trying FOR ALL TABLES, FOR ALL SEQUENCES; but I think it is not
the correct way, so good to have the correct way mentioned in one
example.

patch003:
2)

 * The page_lsn allows the user to determine if the sequence has been updated
 * since the last synchronization with the subscriber. This is done by
 * comparing the current page_lsn with the value stored in pg_subscription_rel
 * from the last synchronization.
 */
Datum
pg_sequence_state(PG_FUNCTION_ARGS)

--This information is still incomplete. Maybe we should mention the
other attribute name as well which helps to determine this.

3)
Shall process_syncing_sequences_for_apply() be moved to sequencesync.c

4)
Would it be better to give a single warning for all unequal sequences
(comma separated list of sequenec names?)

postgres=# create subscription sub1 connection '....' publication pub1;
WARNING:  Sequence parameter in remote and local is not same for "public.myseq2"
HINT:  Alter/Re-create the sequence using the same parameter as in remote.
WARNING:  Sequence parameter in remote and local is not same for "public.myseq0"
HINT:  Alter/Re-create the sequence using the same parameter as in remote.
WARNING:  Sequence parameter in remote and local is not same for "public.myseq4"
HINT:  Alter/Re-create the sequence using the same parameter as in remote.


5)
IIUC, sequencesync_failure_time is changed by multiple processes.
Seq-sync worker sets it before exiting on failure, while apply worker
resets it. Also, the applied worker reads it at a few places. Shall it
be accessed using LogicalRepWorkerLock?

6)
process_syncing_sequences_for_apply():

--I feel MyLogicalRepWorker->sequencesync_failure_time should be reset
to 0 after we are sure that logicalrep_worker_launch() has launched
the worker without any error. But not sure what could be the clean way
to do it? If we move it after logicalrep_worker_launch() call, there
are chances that seq-sync worker has started and failed already and
has set this failure time which will then be mistakenly reset by apply
worker. Also moving it inside logicalrep_worker_launch() does not seem
a good way.

7)
sequencesync.c
PostgreSQL logical replication: initial sequence synchronization

--Since it is called by REFRESH also. So shall we remove 'initial'?

8)
/*
* Process any tables that are being synchronized in parallel and
* any newly added relations.
*/
process_syncing_relations(last_received);

--I did not understand the comment very well. Why are we using 2
separate words 'tables' and 'relations'? I feel we should have
mentioned sequences too in the comment.


9)
logical-replication.sgml: Sequence data is not replicated.

--I feel we should rephrase this line now to indicate that it could be
replicated by the new options.

thanks
Shveta

On Thu, Aug 8, 2024 at 11:09 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> But, I haven't invented a new overloading for "copy_data" option
> (meaning "synchronize") for sequences. The current patchset already
> interprets copy_data exactly this way.
>
> For example, below are patch 0003 results:
>
> ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION WITH (copy_data=false)
> - this will add/remove new sequences in pg_subscription_rel, but it
> will *not* synchronize the new sequence
>
> ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION WITH (copy_data=true)
> - this will add/remove new sequences in pg_subscription_rel, and it
> *will* synchronize the new sequence
>
> ~
>
> I only proposed that copy_data should apply to *all* sequences, not
> just new ones.
>

I don't like this difference because for tables, it would *not*
consider syncing already the existing tables whereas for sequences it
would consider syncing existing ones. We previously discussed adding a
new option like copy_all_sequences instead of adding a new variant of
command but that has its own set of problems, so we agreed to proceed
with a new variant. See [1] ( ...Good point. And I understood that the
REFRESH PUBLICATION SEQUENCES command would be helpful when users want
to synchronize sequences between two nodes before upgrading.).

Having said that, if others also prefer to use copy_data for this
purpose with a different meaning of this option w.r.t tables and
sequences then we can still consider it.

[1] - https://www.postgresql.org/message-id/CAD21AoAAszSeHNRha4HND8b9XyzNrx6jbA7t3Mbe%2BfH4hNRj9A%40mail.gmail.com

--
With Regards,
Amit Kapila.

On Thu, 8 Aug 2024 at 08:30, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh, Here are my v20240807-0003 review comments.
>
> 2a.
> The paragraph starts by saying "Sequence data is not replicated.". It
> seems wrong now. Doesn't that need rewording or removing?

Changed it to incremental sequence changes.

> ~
>
> 2b.
> Should the info "If, however, some kind of switchover or failover..."
> be mentioned in the "Logical Replication Failover" section [2],
> instead of here?

I think mentioning this here is appropriate. The other section focuses
more on how logical replication can proceed with a new primary. Once
the logical replication setup is complete, sequences can be refreshed
at any time.

Rest of the comments  are fixed,  the attached v20240808 version patch
has the changes for the same.

Regards,
Vignesh

On Wed, 7 Aug 2024 at 10:27, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Mon, Aug 5, 2024 at 10:26 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Thu, 1 Aug 2024 at 04:25, Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Vignesh,
> > >
> > > I noticed that when replicating sequences (using the latest patches
> > > 0730_2*)  the subscriber-side checks the *existence* of the sequence,
> > > but apparently it is not checking other sequence attributes.
> > >
> > > For example, consider:
> > >
> > > Publisher: "CREATE SEQUENCE s1 START 1 INCREMENT 2;" should be a
> > > sequence of only odd numbers.
> > > Subscriber: "CREATE SEQUENCE s1 START 2 INCREMENT 2;" should be a
> > > sequence of only even numbers.
> > >
> > > Because the names match, currently the patch allows replication of the
> > > s1 sequence. I think that might lead to unexpected results on the
> > > subscriber. IMO it might be safer to report ERROR unless the sequences
> > > match properly (i.e. not just a name check).
> > >
> > > Below is a demonstration the problem:
> > >
> > > ==========
> > > Publisher:
> > > ==========
> > >
> > > (publisher sequence is odd numbers)
> > >
> > > test_pub=# create sequence s1 start 1 increment 2;
> > > CREATE SEQUENCE
> > > test_pub=# select * from nextval('s1');
> > >  nextval
> > > ---------
> > >        1
> > > (1 row)
> > >
> > > test_pub=# select * from nextval('s1');
> > >  nextval
> > > ---------
> > >        3
> > > (1 row)
> > >
> > > test_pub=# select * from nextval('s1');
> > >  nextval
> > > ---------
> > >        5
> > > (1 row)
> > >
> > > test_pub=# CREATE PUBLICATION pub1 FOR ALL SEQUENCES;
> > > CREATE PUBLICATION
> > > test_pub=#
> > >
> > > ==========
> > > Subscriber:
> > > ==========
> > >
> > > (subscriber sequence is even numbers)
> > >
> > > test_sub=# create sequence s1 start 2 increment 2;
> > > CREATE SEQUENCE
> > > test_sub=# SELECT * FROM nextval('s1');
> > >  nextval
> > > ---------
> > >        2
> > > (1 row)
> > >
> > > test_sub=# SELECT * FROM nextval('s1');
> > >  nextval
> > > ---------
> > >        4
> > > (1 row)
> > >
> > > test_sub=# SELECT * FROM nextval('s1');
> > >  nextval
> > > ---------
> > >        6
> > > (1 row)
> > >
> > > test_sub=# CREATE SUBSCRIPTION sub1 CONNECTION 'dbname=test_pub'
> > > PUBLICATION pub1;
> > > 2024-08-01 08:43:04.198 AEST [24325] WARNING:  subscriptions created
> > > by regression test cases should have names starting with "regress_"
> > > WARNING:  subscriptions created by regression test cases should have
> > > names starting with "regress_"
> > > NOTICE:  created replication slot "sub1" on publisher
> > > CREATE SUBSCRIPTION
> > > test_sub=# 2024-08-01 08:43:04.294 AEST [26240] LOG:  logical
> > > replication apply worker for subscription "sub1" has started
> > > 2024-08-01 08:43:04.309 AEST [26244] LOG:  logical replication
> > > sequence synchronization worker for subscription "sub1" has started
> > > 2024-08-01 08:43:04.323 AEST [26244] LOG:  logical replication
> > > synchronization for subscription "sub1", sequence "s1" has finished
> > > 2024-08-01 08:43:04.323 AEST [26244] LOG:  logical replication
> > > sequence synchronization worker for subscription "sub1" has finished
> > >
> > > (after the CREATE SUBSCRIPTION we are getting replicated odd values
> > > from the publisher, even though the subscriber side sequence was
> > > supposed to be even numbers)
> > >
> > > test_sub=# SELECT * FROM nextval('s1');
> > >  nextval
> > > ---------
> > >        7
> > > (1 row)
> > >
> > > test_sub=# SELECT * FROM nextval('s1');
> > >  nextval
> > > ---------
> > >        9
> > > (1 row)
> > >
> > > test_sub=# SELECT * FROM nextval('s1');
> > >  nextval
> > > ---------
> > >       11
> > > (1 row)
> > >
> > > (Looking at the description you would expect odd values for this
> > > sequence to be impossible)
>
> I see that for such even sequences, user can still do 'setval'  to a
> odd number and then nextval will keep on returning odd value.
>
> postgres=# SELECT nextval('s1');
>        6
>
> postgres=SELECT setval('s1', 43);
>      43
>
> postgres=# SELECT nextval('s1');
>      45
>
> > > test_sub=# \dS+ s1
> > >                              Sequence "public.s1"
> > >   Type  | Start | Minimum |       Maximum       | Increment | Cycles? | Cache
> > > --------+-------+---------+---------------------+-----------+---------+-------
> > >  bigint |     2 |       1 | 9223372036854775807 |         2 | no      |     1
> >
> > Even if we check the sequence definition during the CREATE
> > SUBSCRIPTION/ALTER SUBSCRIPTION ... REFRESH PUBLICATION or ALTER
> > SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES commands, there's still
> > a chance that the sequence definition might change after the command
> > has been executed. Currently, there's no mechanism to lock a sequence,
> > and we also permit replication of table data even if the table
> > structures differ, such as mismatched data types like int and
> > smallint. I have modified it to log a warning to inform users that the
> > sequence options on the publisher and subscriber are not the same and
> > advise them to ensure that the sequence definitions are consistent
> > between both.
> > The v20240805 version patch attached at [1] has the changes for the same.
> > [1] - https://www.postgresql.org/message-id/CALDaNm1Y_ot-jFRfmtwDuwmFrgSSYHjVuy28RspSopTtwzXy8w%40mail.gmail.com
>
> The behavior for applying is no different from setval. Having said
> that, I agree that sequence definition can change even after the
> subscription creation, but earlier we were not syncing sequences and
> thus the value of a particular sequence was going to remain in the
> range/pattern defined by its attributes unless user sets it manually
> using setval. But now, it is being changed in the background without
> user's knowledge.
> The table case is different. In case of table replication, if we have
> CHECK constraint or say primary-key etc, then the value which violates
> these constraints will never be  inserted to a table even during
> replication on sub. For sequences, parameters (MIN,MAX, START,
> INCREMENT) can be considered similar to check-constraints, the only
> difference is during apply, we are still overriding these and copying
> pub's value. May be such inconsistencies detection can be targeted
> later in next project. But for the time being, it will be good to add
> a 'caveat' section in doc mentioning all such cases. The scope of this
> project should be clearly documented.

I have added a Caveats section and mentioned it.
The changes for the same are available at v20240808 version attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm1QQK_Pgx35LrJGuRxBzzYSO8rm1YGJF4w8hYc3Gm%2B5NQ%40mail.gmail.com

Regards,
Vignesh

Hi Vignesh, I reviewed the latest v20240808-0003 patch.

Attached are my minor change suggestions.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Wed, Aug 7, 2024 at 2:00 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 7 Aug 2024 at 08:09, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Aug 6, 2024 at 5:13 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Mon, 5 Aug 2024 at 18:05, shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > On Mon, Aug 5, 2024 at 11:04 AM vignesh C <vignesh21@gmail.com> wrote:
> > > > >
> > > > > On Wed, 31 Jul 2024 at 14:39, shveta malik <shveta.malik@gmail.com> wrote:
> > > > > >
> > > > > > On Mon, Jun 10, 2024 at 5:00 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > > > >
> > > > > > > On Mon, 10 Jun 2024 at 12:24, Amul Sul <sulamul@gmail.com> wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Jun 8, 2024 at 6:43 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > > > > >>
> > > > > > > >> On Wed, 5 Jun 2024 at 14:11, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > > > >> [...]
> > > > > > > >> A new catalog table, pg_subscription_seq, has been introduced for
> > > > > > > >> mapping subscriptions to sequences. Additionally, the sequence LSN
> > > > > > > >> (Log Sequence Number) is stored, facilitating determination of
> > > > > > > >> sequence changes occurring before or after the returned sequence
> > > > > > > >> state.
> > > > > > > >
> > > > > > > >
> > > > > > > > Can't it be done using pg_depend? It seems a bit excessive unless I'm missing
> > > > > > > > something.
> > > > > > >
> > > > > > > We'll require the lsn because the sequence LSN informs the user that
> > > > > > > it has been synchronized up to the LSN in pg_subscription_seq. Since
> > > > > > > we are not supporting incremental sync, the user will be able to
> > > > > > > identify if he should run refresh sequences or not by checking the lsn
> > > > > > > of the pg_subscription_seq and the lsn of the sequence(using
> > > > > > > pg_sequence_state added) in the publisher.
> > > > > >
> > > > > > How the user will know from seq's lsn that he needs to run refresh.
> > > > > > lsn indicates page_lsn and thus the sequence might advance on pub
> > > > > > without changing lsn and thus lsn may look the same on subscriber even
> > > > > > though a sequence-refresh is needed. Am I missing something here?
> > > > >
> > > > > When a sequence is synchronized to the subscriber, the page LSN of the
> > > > > sequence from the publisher is also retrieved and stored in
> > > > > pg_subscriber_rel as shown below:
> > > > > --- Publisher page lsn
> > > > > publisher=# select pg_sequence_state('seq1');
> > > > >  pg_sequence_state
> > > > > --------------------
> > > > >  (0/1510E38,65,1,t)
> > > > > (1 row)
> > > > >
> > > > > --- Subscriber stores the publisher's page lsn for the sequence
> > > > > subscriber=# select * from pg_subscription_rel where srrelid = 16384;
> > > > >  srsubid | srrelid | srsubstate | srsublsn
> > > > > ---------+---------+------------+-----------
> > > > >    16389 |   16384 | r          | 0/1510E38
> > > > > (1 row)
> > > > >
> > > > > If changes are made to the sequence, such as performing many nextvals,
> > > > > the page LSN will be updated. Currently the sequence values are
> > > > > prefetched for SEQ_LOG_VALS 32, so the lsn will not get updated for
> > > > > the prefetched values, once the prefetched values are consumed the lsn
> > > > > will get updated.
> > > > > For example:
> > > > > --- Updated LSN on the publisher (old lsn - 0/1510E38, new lsn - 0/1558CA8)
> > > > > publisher=# select pg_sequence_state('seq1');
> > > > >   pg_sequence_state
> > > > > ----------------------
> > > > >  (0/1558CA8,143,22,t)
> > > > > (1 row)
> > > > >
> > > > > The user can then compare this updated value with the sequence's LSN
> > > > > in pg_subscription_rel to determine when to re-synchronize the
> > > > > sequence.
> > > >
> > > > Thanks for the details. But I was referring to the case where we are
> > > > in between pre-fetched values on publisher (say at 25th value), while
> > > > on subscriber we are slightly behind (say at 15th value), but page-lsn
> > > > will be the same on both. Since the subscriber is behind, a
> > > > sequence-refresh is needed on sub, but by looking at lsn (which is
> > > > same), one can not say that for sure.  Let me know if I have
> > > > misunderstood it.
> > >
> > > Yes, at present, if the value is within the pre-fetched range, we
> > > cannot distinguish it solely using the page_lsn.
> > >
> >
> > This makes sense to me.
> >
> > >
> > > However, the
> > > pg_sequence_state function also provides last_value and log_cnt, which
> > > can be used to handle these specific cases.
> > >
> >
> > BTW, can we document all these steps for users to know when to refresh
> > the sequences, if not already documented?
>
> This has been documented in the v20240807 version attached at [1].
> [1] - https://www.postgresql.org/message-id/CALDaNm01Z6Oo9osGMFTOoyTR1kVoyh1rEvZ%2B6uJn-ZymV%3D0dbQ%40mail.gmail.com
>

Vignesh, I looked at the patch dated 240808, but I could not  find
these steps. Are you referring to the section ' Examples:
Synchronizing Sequences Between Publisher and Subscriber' in doc
patch004? If not, please point me to the concerned section.

thanks
Shveta

Hi Vignesh, here are my review comments for the sequences docs patch
v20240808-0004.

======
doc/src/sgml/logical-replication.sgml

The new section content looked good.

Just some nitpicks including:
- renamed the section "Replicating Sequences"
- added missing mention about how to publish sequences
- rearranged the subscription commands into a more readable list
- some sect2 titles were very long; I shortened them.
- added <warning> markup for the sequence definition advice
- other minor rewording and typo fixes

~

1.
IMO the "Caveats" section can be removed.
- the advice to avoid changing the sequence definition is already
given earlier in the "Sequence Definition Mismatches" section
- the limitation of "incremental synchronization" is already stated in
the logical replication "Limitations" section
- (FYI, I removed it already in my nitpicks attachment)

======
doc/src/sgml/ref/alter_subscription.sgml

nitpick - I reversed the paragraphs to keep the references in a natural order.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Fri, Aug 9, 2024 at 1:52 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 8 Aug 2024 at 08:30, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Vignesh, Here are my v20240807-0003 review comments.
> >
> > 2a.
> > The paragraph starts by saying "Sequence data is not replicated.". It
> > seems wrong now. Doesn't that need rewording or removing?
>
> Changed it to incremental sequence changes.
>
> > ~
> >
> > 2b.
> > Should the info "If, however, some kind of switchover or failover..."
> > be mentioned in the "Logical Replication Failover" section [2],
> > instead of here?
>
> I think mentioning this here is appropriate. The other section focuses
> more on how logical replication can proceed with a new primary. Once
> the logical replication setup is complete, sequences can be refreshed
> at any time.
>
> Rest of the comments  are fixed,  the attached v20240808 version patch
> has the changes for the same.
>
> Regards,
> Vignesh

On Thu, 8 Aug 2024 at 12:21, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Wed, Aug 7, 2024 at 1:45 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> >
> > The remaining comments have been addressed, and the changes are
> > included in the attached v20240807 version patch.
>
> Thanks for addressing the comment. Please find few comments for v20240807 :
>
> patch003:
> 2)
>
>  * The page_lsn allows the user to determine if the sequence has been updated
>  * since the last synchronization with the subscriber. This is done by
>  * comparing the current page_lsn with the value stored in pg_subscription_rel
>  * from the last synchronization.
>  */
> Datum
> pg_sequence_state(PG_FUNCTION_ARGS)
>
> --This information is still incomplete. Maybe we should mention the
> other attribute name as well which helps to determine this.

I have removed this comment now as suggesting that users use
pg_sequence_state and sequence when page_lsn seems complex, the same
can be achieved by comparing the sequence values from a single
statement instead of a couple of statements. Peter had felt this would
be easier based on comment 3c at [1].

> 5)
> IIUC, sequencesync_failure_time is changed by multiple processes.
> Seq-sync worker sets it before exiting on failure, while apply worker
> resets it. Also, the applied worker reads it at a few places. Shall it
> be accessed using LogicalRepWorkerLock?

If sequenceApply worker is already running, apply worker will not
access sequencesync_failure_time. Only if sequence sync worker is not
running apply worker will access sequencesync_failure_time in the
below code. I feel no need to use LogicalRepWorkerLock in this case.

...
syncworker = logicalrep_worker_find(MyLogicalRepWorker->subid,
InvalidOid, WORKERTYPE_SEQUENCESYNC,
true);
if (syncworker)
{
/* Now safe to release the LWLock */
LWLockRelease(LogicalRepWorkerLock);
break;
}

/*
* Count running sync workers for this subscription, while we have the
* lock.
*/
nsyncworkers = logicalrep_sync_worker_count(MyLogicalRepWorker->subid);

/* Now safe to release the LWLock */
LWLockRelease(LogicalRepWorkerLock);

/*
* If there are free sync worker slot(s), start a new sequence sync
* worker, and break from the loop.
*/
if (nsyncworkers < max_sync_workers_per_subscription)
{
TimestampTz now = GetCurrentTimestamp();

if (!MyLogicalRepWorker->sequencesync_failure_time ||
TimestampDifferenceExceeds(MyLogicalRepWorker->sequencesync_failure_time,
   now, wal_retrieve_retry_interval))
{
MyLogicalRepWorker->sequencesync_failure_time = 0;

logicalrep_worker_launch(WORKERTYPE_SEQUENCESYNC,
MyLogicalRepWorker->dbid,
MySubscription->oid,
MySubscription->name,
MyLogicalRepWorker->userid,
InvalidOid,
DSM_HANDLE_INVALID);
break;
}
}
...

> 6)
> process_syncing_sequences_for_apply():
>
> --I feel MyLogicalRepWorker->sequencesync_failure_time should be reset
> to 0 after we are sure that logicalrep_worker_launch() has launched
> the worker without any error. But not sure what could be the clean way
> to do it? If we move it after logicalrep_worker_launch() call, there
> are chances that seq-sync worker has started and failed already and
> has set this failure time which will then be mistakenly reset by apply
> worker. Also moving it inside logicalrep_worker_launch() does not seem
> a good way.

I felt we can keep it in the existing way to keep it consistent with
table sync worker restart like in process_syncing_tables_for_apply.

The rest of the comments are fixed.  The rest of the comments are
fixed in the v20240809 version patch attached.

[1] - https://www.postgresql.org/message-id/CAHut%2BPvaq%3D0xsDWdVQ-kdjRa8Az%2BvgiMFTvT2E2nR3N-47TO8A%40mail.gmail.com

Regards,
Vignesh

On Fri, 9 Aug 2024 at 12:13, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Wed, Aug 7, 2024 at 2:00 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, 7 Aug 2024 at 08:09, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Tue, Aug 6, 2024 at 5:13 PM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > On Mon, 5 Aug 2024 at 18:05, shveta malik <shveta.malik@gmail.com> wrote:
> > > > >
> > > > > On Mon, Aug 5, 2024 at 11:04 AM vignesh C <vignesh21@gmail.com> wrote:
> > > > > >
> > > > > > On Wed, 31 Jul 2024 at 14:39, shveta malik <shveta.malik@gmail.com> wrote:
> > > > > > >
> > > > > > > On Mon, Jun 10, 2024 at 5:00 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > > > > >
> > > > > > > > On Mon, 10 Jun 2024 at 12:24, Amul Sul <sulamul@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sat, Jun 8, 2024 at 6:43 PM vignesh C <vignesh21@gmail.com> wrote:
> > > > > > > > >>
> > > > > > > > >> On Wed, 5 Jun 2024 at 14:11, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > > > > >> [...]
> > > > > > > > >> A new catalog table, pg_subscription_seq, has been introduced for
> > > > > > > > >> mapping subscriptions to sequences. Additionally, the sequence LSN
> > > > > > > > >> (Log Sequence Number) is stored, facilitating determination of
> > > > > > > > >> sequence changes occurring before or after the returned sequence
> > > > > > > > >> state.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Can't it be done using pg_depend? It seems a bit excessive unless I'm missing
> > > > > > > > > something.
> > > > > > > >
> > > > > > > > We'll require the lsn because the sequence LSN informs the user that
> > > > > > > > it has been synchronized up to the LSN in pg_subscription_seq. Since
> > > > > > > > we are not supporting incremental sync, the user will be able to
> > > > > > > > identify if he should run refresh sequences or not by checking the lsn
> > > > > > > > of the pg_subscription_seq and the lsn of the sequence(using
> > > > > > > > pg_sequence_state added) in the publisher.
> > > > > > >
> > > > > > > How the user will know from seq's lsn that he needs to run refresh.
> > > > > > > lsn indicates page_lsn and thus the sequence might advance on pub
> > > > > > > without changing lsn and thus lsn may look the same on subscriber even
> > > > > > > though a sequence-refresh is needed. Am I missing something here?
> > > > > >
> > > > > > When a sequence is synchronized to the subscriber, the page LSN of the
> > > > > > sequence from the publisher is also retrieved and stored in
> > > > > > pg_subscriber_rel as shown below:
> > > > > > --- Publisher page lsn
> > > > > > publisher=# select pg_sequence_state('seq1');
> > > > > >  pg_sequence_state
> > > > > > --------------------
> > > > > >  (0/1510E38,65,1,t)
> > > > > > (1 row)
> > > > > >
> > > > > > --- Subscriber stores the publisher's page lsn for the sequence
> > > > > > subscriber=# select * from pg_subscription_rel where srrelid = 16384;
> > > > > >  srsubid | srrelid | srsubstate | srsublsn
> > > > > > ---------+---------+------------+-----------
> > > > > >    16389 |   16384 | r          | 0/1510E38
> > > > > > (1 row)
> > > > > >
> > > > > > If changes are made to the sequence, such as performing many nextvals,
> > > > > > the page LSN will be updated. Currently the sequence values are
> > > > > > prefetched for SEQ_LOG_VALS 32, so the lsn will not get updated for
> > > > > > the prefetched values, once the prefetched values are consumed the lsn
> > > > > > will get updated.
> > > > > > For example:
> > > > > > --- Updated LSN on the publisher (old lsn - 0/1510E38, new lsn - 0/1558CA8)
> > > > > > publisher=# select pg_sequence_state('seq1');
> > > > > >   pg_sequence_state
> > > > > > ----------------------
> > > > > >  (0/1558CA8,143,22,t)
> > > > > > (1 row)
> > > > > >
> > > > > > The user can then compare this updated value with the sequence's LSN
> > > > > > in pg_subscription_rel to determine when to re-synchronize the
> > > > > > sequence.
> > > > >
> > > > > Thanks for the details. But I was referring to the case where we are
> > > > > in between pre-fetched values on publisher (say at 25th value), while
> > > > > on subscriber we are slightly behind (say at 15th value), but page-lsn
> > > > > will be the same on both. Since the subscriber is behind, a
> > > > > sequence-refresh is needed on sub, but by looking at lsn (which is
> > > > > same), one can not say that for sure.  Let me know if I have
> > > > > misunderstood it.
> > > >
> > > > Yes, at present, if the value is within the pre-fetched range, we
> > > > cannot distinguish it solely using the page_lsn.
> > > >
> > >
> > > This makes sense to me.
> > >
> > > >
> > > > However, the
> > > > pg_sequence_state function also provides last_value and log_cnt, which
> > > > can be used to handle these specific cases.
> > > >
> > >
> > > BTW, can we document all these steps for users to know when to refresh
> > > the sequences, if not already documented?
> >
> > This has been documented in the v20240807 version attached at [1].
> > [1] -
https://www.postgresql.org/message-id/CALDaNm01Z6Oo9osGMFTOoyTR1kVoyh1rEvZ%2B6uJn-ZymV%3D0dbQ%40mail.gmail.com
> >
>
> Vignesh, I looked at the patch dated 240808, but I could not  find
> these steps. Are you referring to the section ' Examples:
> Synchronizing Sequences Between Publisher and Subscriber' in doc
> patch004? If not, please point me to the concerned section.

I'm  referring to the "Refreshing Stale Sequences" part in the
v20240809 version patch attached at [1] which only mentions directly
comparing the sequence values.. I have removed the reference to
pg_sequence_state now as suggesting that users use pg_sequence_state
and sequence when page_lsn seems complex, the same can be achieved by
comparing the sequence values from a single statement instead of a
couple of statements. Peter had felt this would be easier based on
comment 3c at [1].

[1] - https://www.postgresql.org/message-id/CALDaNm0LJCtGoBCO6DFY-RDjR8vxapW3W1f7%3D-LSQx%3DXYjqU%3Dw%40mail.gmail.com

Regards,
Vignesh

On Fri, 9 Aug 2024 at 05:51, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh, I reviewed the latest v20240808-0003 patch.
>
> Attached are my minor change suggestions.

Thanks, these changes are merged in the v20240809 version posted at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm0LJCtGoBCO6DFY-RDjR8vxapW3W1f7%3D-LSQx%3DXYjqU%3Dw%40mail.gmail.com

Regards,
Vignesh

On Fri, 9 Aug 2024 at 12:40, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh, here are my review comments for the sequences docs patch
> v20240808-0004.
>
> ======
> doc/src/sgml/logical-replication.sgml
>
> The new section content looked good.
>
> Just some nitpicks including:
> - renamed the section "Replicating Sequences"
> - added missing mention about how to publish sequences
> - rearranged the subscription commands into a more readable list
> - some sect2 titles were very long; I shortened them.
> - added <warning> markup for the sequence definition advice
> - other minor rewording and typo fixes

I have retained the caveats section for now, I will think more and
remove it if required in the next version.

The rest of the comments are fixed in the v20240809 version patch
attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm0LJCtGoBCO6DFY-RDjR8vxapW3W1f7%3D-LSQx%3DXYjqU%3Dw%40mail.gmail.com

Regards,
Vignesh

Hi Vignesh,

v20240809-0001. No comments.
v20240809-0002. See below.
v20240809-0003. See below.
v20240809-0004. No comments.

//////////

Here are my review comments for patch v20240809-0002.

nit - Tweak wording in new docs example, because a publication only
publishes the sequences; it doesn't "synchronize" anything.

//////////

Here are my review comments for patch v20240809-0003.

fetch_sequence_list:
nit - move comment
nit - minor rewording for parameter WARNING message

======
.../replication/logical/sequencesync.c
src/backend/replication/logical/tablesync.c

1.
Currently the declaration 'sequence_states_not_ready' list seems
backwards. IMO it makes more sense for the declaration to be in
sequencesync.c, and the extern in the tablesync.c. (please also see
review comment #3 below which might affect this too).

~~~

2.
 static bool
-FetchTableStates(bool *started_tx)
+FetchTableStates(void)
 {
- static bool has_subrels = false;
-
- *started_tx = false;
+ static bool has_subtables = false;
+ bool started_tx = false;

Maybe give the explanation why 'has_subtables' is declared static here.

~~~

3.
I am not sure that it was an improvement to move the
process_syncing_sequences_for_apply() function into the
sequencesync.c. Calling the sequence code from the tablesync code
still looks strange. OTOH, I see why you don't want to leave it in
tablesync.c.

Perhaps it would be better to refactor/move all following functions
back to the (apply) worker.c instead:
- process_syncing_relations
- process_syncing_sequences_for_apply(void)
- process_syncing_tables_for_apply(void)

Actually, now that there are 2 kinds of 'sync' workers, maybe you
should introduce a new module (e.g. 'commonsync.c' or
'syncworker.c...), where you can put functions such as
process_syncing_relations() plus any other code common to both
tablesync and sequencesync. That might make more sense then having one
call to the other.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi Vignesh,

I noticed it is not currently possible (there is no syntax way to do
it) to ALTER an existing publication so that it will publish
SEQUENCES.

Isn't that a limitation? Why?

For example,. Why should users be prevented from changing a FOR ALL
TABLES publication into a FOR ALL TABLES, SEQUENCES one?

Similarly, there are other combinations not possible
DROP ALL SEQUENCES from a publication that is FOR ALL TABLES, SEQUENCES
DROP ALL TABLES from a publication that is FOR ALL TABLES, SEQUENCES
ADD ALL TABLES to a publication that is FOR ALL SEQUENCES
...

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi Vignesh,

I found that when 2 subscriptions are both subscribing to a
publication publishing sequences, an ERROR occurs on refresh.

======

Publisher:
----------

test_pub=# create publication pub1 for all sequences;

Subscriber:
-----------

test_sub=# create subscription sub1 connection 'dbname=test_pub'
publication pub1;

test_sub=# create subscription sub2 connection 'dbname=test_pub'
publication pub1;

test_sub=# alter subscription sub1 refresh publication sequences;
2024-08-12 15:04:04.947 AEST [7306] LOG:  sequence "public.seq1" of
subscription "sub1" set to INIT state
2024-08-12 15:04:04.947 AEST [7306] STATEMENT:  alter subscription
sub1 refresh publication sequences;
2024-08-12 15:04:04.947 AEST [7306] LOG:  sequence "public.seq1" of
subscription "sub1" set to INIT state
2024-08-12 15:04:04.947 AEST [7306] STATEMENT:  alter subscription
sub1 refresh publication sequences;
2024-08-12 15:04:04.947 AEST [7306] ERROR:  tuple already updated by self
2024-08-12 15:04:04.947 AEST [7306] STATEMENT:  alter subscription
sub1 refresh publication sequences;
ERROR:  tuple already updated by self

test_sub=# alter subscription sub2 refresh publication sequences;
2024-08-12 15:04:30.427 AEST [7306] LOG:  sequence "public.seq1" of
subscription "sub2" set to INIT state
2024-08-12 15:04:30.427 AEST [7306] STATEMENT:  alter subscription
sub2 refresh publication sequences;
2024-08-12 15:04:30.427 AEST [7306] LOG:  sequence "public.seq1" of
subscription "sub2" set to INIT state
2024-08-12 15:04:30.427 AEST [7306] STATEMENT:  alter subscription
sub2 refresh publication sequences;
2024-08-12 15:04:30.427 AEST [7306] ERROR:  tuple already updated by self
2024-08-12 15:04:30.427 AEST [7306] STATEMENT:  alter subscription
sub2 refresh publication sequences;
ERROR:  tuple already updated by self

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Mon, 12 Aug 2024 at 08:50, Peter Smith <smithpb2250@gmail.com> wrote:
>
> ~~~
>
> 3.
> I am not sure that it was an improvement to move the
> process_syncing_sequences_for_apply() function into the
> sequencesync.c. Calling the sequence code from the tablesync code
> still looks strange. OTOH, I see why you don't want to leave it in
> tablesync.c.
>
> Perhaps it would be better to refactor/move all following functions
> back to the (apply) worker.c instead:
> - process_syncing_relations
> - process_syncing_sequences_for_apply(void)
> - process_syncing_tables_for_apply(void)
>
> Actually, now that there are 2 kinds of 'sync' workers, maybe you
> should introduce a new module (e.g. 'commonsync.c' or
> 'syncworker.c...), where you can put functions such as
> process_syncing_relations() plus any other code common to both
> tablesync and sequencesync. That might make more sense then having one
> call to the other.

I created syncutils.c to consolidate code that supports worker
synchronization, table synchronization, and sequence synchronization.
While it may not align exactly with your suggestion, I included
functions like finish_sync_worker, invalidate_syncing_relation_states,
FetchRelationStates, and process_syncing_relations in this new file. I
believe this organization will make the code easier to review.

The rest of the comments are also fixed in the attached v20240812
version patch attached.

Regards,
Vignesh

On Mon, 12 Aug 2024 at 10:40, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> I found that when 2 subscriptions are both subscribing to a
> publication publishing sequences, an ERROR occurs on refresh.
>
> ======
>
> Publisher:
> ----------
>
> test_pub=# create publication pub1 for all sequences;
>
> Subscriber:
> -----------
>
> test_sub=# create subscription sub1 connection 'dbname=test_pub'
> publication pub1;
>
> test_sub=# create subscription sub2 connection 'dbname=test_pub'
> publication pub1;
>
> test_sub=# alter subscription sub1 refresh publication sequences;
> 2024-08-12 15:04:04.947 AEST [7306] LOG:  sequence "public.seq1" of
> subscription "sub1" set to INIT state
> 2024-08-12 15:04:04.947 AEST [7306] STATEMENT:  alter subscription
> sub1 refresh publication sequences;
> 2024-08-12 15:04:04.947 AEST [7306] LOG:  sequence "public.seq1" of
> subscription "sub1" set to INIT state
> 2024-08-12 15:04:04.947 AEST [7306] STATEMENT:  alter subscription
> sub1 refresh publication sequences;
> 2024-08-12 15:04:04.947 AEST [7306] ERROR:  tuple already updated by self
> 2024-08-12 15:04:04.947 AEST [7306] STATEMENT:  alter subscription
> sub1 refresh publication sequences;
> ERROR:  tuple already updated by self
>
> test_sub=# alter subscription sub2 refresh publication sequences;
> 2024-08-12 15:04:30.427 AEST [7306] LOG:  sequence "public.seq1" of
> subscription "sub2" set to INIT state
> 2024-08-12 15:04:30.427 AEST [7306] STATEMENT:  alter subscription
> sub2 refresh publication sequences;
> 2024-08-12 15:04:30.427 AEST [7306] LOG:  sequence "public.seq1" of
> subscription "sub2" set to INIT state
> 2024-08-12 15:04:30.427 AEST [7306] STATEMENT:  alter subscription
> sub2 refresh publication sequences;
> 2024-08-12 15:04:30.427 AEST [7306] ERROR:  tuple already updated by self
> 2024-08-12 15:04:30.427 AEST [7306] STATEMENT:  alter subscription
> sub2 refresh publication sequences;
> ERROR:  tuple already updated by self

This issue is fixed in the v20240812 version attached at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm3hS58W0RTbgsMTk-YvXwt956uabA%3DkYfLGUs3uRNC2Qg%40mail.gmail.com

Regards,
Vignesh

On Mon, 12 Aug 2024 at 09:59, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> I noticed it is not currently possible (there is no syntax way to do
> it) to ALTER an existing publication so that it will publish
> SEQUENCES.
>
> Isn't that a limitation? Why?
>
> For example,. Why should users be prevented from changing a FOR ALL
> TABLES publication into a FOR ALL TABLES, SEQUENCES one?
>
> Similarly, there are other combinations not possible
> DROP ALL SEQUENCES from a publication that is FOR ALL TABLES, SEQUENCES
> DROP ALL TABLES from a publication that is FOR ALL TABLES, SEQUENCES
> ADD ALL TABLES to a publication that is FOR ALL SEQUENCES

Yes, this should be addressed. However, I'll defer it until the
current set of patches is finalized and all comments have been
resolved.

Regards,
Vignesh

Hi Vignesh, Here are my review comments for latest v20240812* patchset:

patch v20240812-0001. No comments.
patch v20240812-0002. Fixed docs.LGTM
patch v20240812-0003. This is new refactoring. See below.
patch v20240812-0004. (was 0003). See below.
patch v20240812-0005. (was 0004). No comments.

//////

patch v20240812-0003.

3.1. GENERAL

Hmm. I am guessing this was provided as a separate patch to aid review
by showing that existing functions are moved? OTOH you can't really
judge this patch properly without already knowing details of what will
come next in the sequencesync. i.e. As a *standalone* patch without
the sequencesync.c the refactoring doesn't make much sense.

Maybe it is OK later to combine patches 0003 and 0004. Alternatively,
keep this patch separated but give greater emphasis in the comment
header to say this patch only exists separately in order to help the
review.

======
Commit message

3.2.
Reorganized tablesync code to generate a syncutils file which will
help in sequence synchronization worker code.

~

"generate" ??

======
src/backend/replication/logical/syncutils.c

3.3. "common code" ??

FYI - There are multiple code comments mentioning "common code..."
which, in the absence of the sequencesync worker (which comes in the
next patch), have nothing "common" about them at all. Fixing them and
then fixing them again in the next patch might cause unnecessary code
churn, but OTOH they aren't correct as-is either. I have left them
alone for now.

~

3.4. function names

With the re-shuffling that this patch does, and changing several from
static to not-static, should the function names remain as they are?
They look random to me.
- finish_sync_worker(void)
- invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
- FetchTableStates(bool *started_tx)
- process_syncing_tables(XLogRecPtr current_lsn)

I think using a consistent naming convention would be better. e.g.
SyncFinishWorker
SyncInvalidateTableStates
SyncFetchTableStates
SyncProcessTables

~~~

nit - file header comment

======
src/backend/replication/logical/tablesync.c

3.5.
-static void
+void
 process_syncing_tables_for_sync(XLogRecPtr current_lsn)

-static void
+void
 process_syncing_tables_for_apply(XLogRecPtr current_lsn)

Since these functions are no longer static should those function names
be changed to use the CamelCase convention for non-static API?

//////////

patch v20240812-0004.

======
src/backend/replication/logical/syncutils.c

nit - file header comment (made same as patch 0003)

~

FetchRelationStates:
nit - IIUC sequence states are only INIT -> READY. So the comments in
this function dont need to specifically talk about sequence INIT
state.

======
src/backend/utils/misc/guc_tables.c

4.1.
  {"max_sync_workers_per_subscription",
  PGC_SIGHUP,
  REPLICATION_SUBSCRIBERS,
- gettext_noop("Maximum number of table synchronization workers per
subscription."),
+ gettext_noop("Maximum number of relation synchronization workers per
subscription."),
  NULL,
  },

I was wondering if "relation synchronization workers" is meaningful to
the user because that seems like new terminology.
Maybe it should say "... of table + sequence synchronization workers..."

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Mon, Aug 12, 2024 at 11:07 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, 12 Aug 2024 at 10:40, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > Hi Vignesh,
> >
> > I found that when 2 subscriptions are both subscribing to a
> > publication publishing sequences, an ERROR occurs on refresh.
> >
> > ======
> >
> > Publisher:
> > ----------
> >
> > test_pub=# create publication pub1 for all sequences;
> >
> > Subscriber:
> > -----------
> >
> > test_sub=# create subscription sub1 connection 'dbname=test_pub'
> > publication pub1;
> >
> > test_sub=# create subscription sub2 connection 'dbname=test_pub'
> > publication pub1;
> >
> > test_sub=# alter subscription sub1 refresh publication sequences;
> > 2024-08-12 15:04:04.947 AEST [7306] LOG:  sequence "public.seq1" of
> > subscription "sub1" set to INIT state
> > 2024-08-12 15:04:04.947 AEST [7306] STATEMENT:  alter subscription
> > sub1 refresh publication sequences;
> > 2024-08-12 15:04:04.947 AEST [7306] LOG:  sequence "public.seq1" of
> > subscription "sub1" set to INIT state
> > 2024-08-12 15:04:04.947 AEST [7306] STATEMENT:  alter subscription
> > sub1 refresh publication sequences;
> > 2024-08-12 15:04:04.947 AEST [7306] ERROR:  tuple already updated by self
> > 2024-08-12 15:04:04.947 AEST [7306] STATEMENT:  alter subscription
> > sub1 refresh publication sequences;
> > ERROR:  tuple already updated by self
> >
> > test_sub=# alter subscription sub2 refresh publication sequences;
> > 2024-08-12 15:04:30.427 AEST [7306] LOG:  sequence "public.seq1" of
> > subscription "sub2" set to INIT state
> > 2024-08-12 15:04:30.427 AEST [7306] STATEMENT:  alter subscription
> > sub2 refresh publication sequences;
> > 2024-08-12 15:04:30.427 AEST [7306] LOG:  sequence "public.seq1" of
> > subscription "sub2" set to INIT state
> > 2024-08-12 15:04:30.427 AEST [7306] STATEMENT:  alter subscription
> > sub2 refresh publication sequences;
> > 2024-08-12 15:04:30.427 AEST [7306] ERROR:  tuple already updated by self
> > 2024-08-12 15:04:30.427 AEST [7306] STATEMENT:  alter subscription
> > sub2 refresh publication sequences;
> > ERROR:  tuple already updated by self
>
> This issue is fixed in the v20240812 version attached at [1].
> [1] - https://www.postgresql.org/message-id/CALDaNm3hS58W0RTbgsMTk-YvXwt956uabA%3DkYfLGUs3uRNC2Qg%40mail.gmail.com
>

Yes, I confirmed it is now fixed. Thanks!

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi Vignesh,

I have been using the latest patchset, trying a few things using many
(1000) sequences.

Here are some observations, plus some suggestions for consideration.

~~~~~

OBSERVATION #1

When 1000s of sequences are refreshed using REFRESH PUBLICATION
SEQUENCES the logging is excessive. For example, since there is only
one sequencesync worker why does it need to broadcast that it is
"finished" separately for every single sequence. That is giving 1000s
of lines of logs which don't seem to be of much interest to a user.

...
2024-08-13 16:17:04.151 AEST [5002] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0918" has
finished
2024-08-13 16:17:04.151 AEST [5002] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0919" has
finished
2024-08-13 16:17:04.151 AEST [5002] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0920" has
finished
2024-08-13 16:17:04.151 AEST [5002] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0921" has
finished
2024-08-13 16:17:04.151 AEST [5002] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0922" has
finished
2024-08-13 16:17:04.151 AEST [5002] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0923" has
finished
...

Perhaps just LOG when each "batch" is completed, but the individual
sequence finished logs can just be DEBUG information?

~~~~~

OBSERVATION #2

When 1000s of sequences are refreshed (set to INIT) then there are
1000s of logs like below:

...
2024-08-13 16:13:57.873 AEST [10301] LOG:  sequence "public.seq_0698"
of subscription "sub3" set to INIT state
2024-08-13 16:13:57.873 AEST [10301] STATEMENT:  alter subscription
sub3 refresh publication sequences;
2024-08-13 16:13:57.873 AEST [10301] LOG:  sequence "public.seq_0699"
of subscription "sub3" set to INIT state
2024-08-13 16:13:57.873 AEST [10301] STATEMENT:  alter subscription
sub3 refresh publication sequences;
2024-08-13 16:13:57.873 AEST [10301] LOG:  sequence "public.seq_0700"
of subscription "sub3" set to INIT state
2024-08-13 16:13:57.873 AEST [10301] STATEMENT:  alter subscription
sub3 refresh publication sequences;
2024-08-13 16:13:57.873 AEST [10301] LOG:  sequence "public.seq_0701"
of subscription "sub3" set to INIT state
2024-08-13 16:13:57.873 AEST [10301] STATEMENT:  alter subscription
sub3 refresh publication sequences;
2024-08-13 16:13:57.874 AEST [10301] LOG:  sequence "public.seq_0702"
of subscription "sub3" set to INIT state
2024-08-13 16:13:57.874 AEST [10301] STATEMENT:  alter subscription
sub3 refresh publication sequences;
...

I felt that showing the STATEMENT for all of these is overkill. How
about changing that ereport LOG so it does not emit the statement 1000
times? Or, maybe you can implement it as a "dynamic" log that emits
the STATEMENT if there are only a few logs a few times but skips it
for the next 995 logs.

~~~~~

OBSERVATION #3

The WARNING about mismatched sequences currently looks like this:

2024-08-13 16:41:45.496 AEST [10301] WARNING:  Parameters differ for
remote and local sequences "public.seq_0999"
2024-08-13 16:41:45.496 AEST [10301] HINT:  Alter/Re-create the
sequence using the same parameter as in remote.

Although you could probably deduce it from nearby logs, I think it
might be more helpful to also identify the subscription name in this
WARNING message. Otherwise, if there are many publications the user
may have no idea where the mismatched "remote" is coming from.

~~~~

OBSERVATION #4

When 1000s of sequences are refreshed then there are 1000s of
associated logs. But (given there is only one sequencesync worker)
those logs are not always the order that I was expecting to see them.

e.g.
...
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0885" has
finished
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0887" has
finished
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0888" has
finished
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0889" has
finished
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0890" has
finished
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0906" has
finished
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0566" has
finished
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0568" has
finished
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0569" has
finished
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0570" has
finished
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0571" has
finished
2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
synchronization for subscription "sub3", sequence "seq_0582" has
finished
...

Is there a way to refresh sequences in a more natural (e.g.
alphabetical) order to make these logs more readable?

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Tue, 13 Aug 2024 at 09:19, Peter Smith <smithpb2250@gmail.com> wrote:
>
> 3.1. GENERAL
>
> Hmm. I am guessing this was provided as a separate patch to aid review
> by showing that existing functions are moved? OTOH you can't really
> judge this patch properly without already knowing details of what will
> come next in the sequencesync. i.e. As a *standalone* patch without
> the sequencesync.c the refactoring doesn't make much sense.
>
> Maybe it is OK later to combine patches 0003 and 0004. Alternatively,
> keep this patch separated but give greater emphasis in the comment
> header to say this patch only exists separately in order to help the
> review.

I have kept this patch only to show that this patch as such has no
code changes. If we move this to the next patch it will be difficult
for reviewers to know which is new code and which is old code. During
commit we can merge this with the next one. I felt it is better to add
it in the commit message instead of comment header so updated the
commit message.

> ======
> src/backend/replication/logical/syncutils.c
>
> 3.3. "common code" ??
>
> FYI - There are multiple code comments mentioning "common code..."
> which, in the absence of the sequencesync worker (which comes in the
> next patch), have nothing "common" about them at all. Fixing them and
> then fixing them again in the next patch might cause unnecessary code
> churn, but OTOH they aren't correct as-is either. I have left them
> alone for now.

We can ignore this as this will get merged to the next one. If you
have any comments you can give it on top of the next(0004) patch.

> ~
>
> 3.4. function names
>
> With the re-shuffling that this patch does, and changing several from
> static to not-static, should the function names remain as they are?
> They look random to me.
> - finish_sync_worker(void)
> - invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
> - FetchTableStates(bool *started_tx)
> - process_syncing_tables(XLogRecPtr current_lsn)
>
> I think using a consistent naming convention would be better. e.g.
> SyncFinishWorker
> SyncInvalidateTableStates
> SyncFetchTableStates
> SyncProcessTables

One advantage with keeping the existing names the same wherever
possible will help while merging the changes to back-branches. So I'm
not making this change.

> ~~~
>
> nit - file header comment
>
> ======
> src/backend/replication/logical/tablesync.c
>
> 3.5.
> -static void
> +void
>  process_syncing_tables_for_sync(XLogRecPtr current_lsn)
>
> -static void
> +void
>  process_syncing_tables_for_apply(XLogRecPtr current_lsn)
>
> Since these functions are no longer static should those function names
> be changed to use the CamelCase convention for non-static API?

One advantage with keeping the existing names the same wherever
possible will help while merging the changes to back-branches. So I'm
not making this change.

The rest of the comments were fixed, the attached v20240813 has the
changes for the same.

Regards,
Vignesh

On Tue, 13 Aug 2024 at 12:31, Peter Smith <smithpb2250@gmail.com> wrote:
>
> OBSERVATION #2
>
> When 1000s of sequences are refreshed (set to INIT) then there are
> 1000s of logs like below:
>
> ...
> 2024-08-13 16:13:57.873 AEST [10301] LOG:  sequence "public.seq_0698"
> of subscription "sub3" set to INIT state
> 2024-08-13 16:13:57.873 AEST [10301] STATEMENT:  alter subscription
> sub3 refresh publication sequences;
> 2024-08-13 16:13:57.873 AEST [10301] LOG:  sequence "public.seq_0699"
> of subscription "sub3" set to INIT state
> 2024-08-13 16:13:57.873 AEST [10301] STATEMENT:  alter subscription
> sub3 refresh publication sequences;
> 2024-08-13 16:13:57.873 AEST [10301] LOG:  sequence "public.seq_0700"
> of subscription "sub3" set to INIT state
> 2024-08-13 16:13:57.873 AEST [10301] STATEMENT:  alter subscription
> sub3 refresh publication sequences;
> 2024-08-13 16:13:57.873 AEST [10301] LOG:  sequence "public.seq_0701"
> of subscription "sub3" set to INIT state
> 2024-08-13 16:13:57.873 AEST [10301] STATEMENT:  alter subscription
> sub3 refresh publication sequences;
> 2024-08-13 16:13:57.874 AEST [10301] LOG:  sequence "public.seq_0702"
> of subscription "sub3" set to INIT state
> 2024-08-13 16:13:57.874 AEST [10301] STATEMENT:  alter subscription
> sub3 refresh publication sequences;
> ...
>
> I felt that showing the STATEMENT for all of these is overkill. How
> about changing that ereport LOG so it does not emit the statement 1000
> times? Or, maybe you can implement it as a "dynamic" log that emits
> the STATEMENT if there are only a few logs a few times but skips it
> for the next 995 logs.

I have changed it to debug1 log level how we do for tables, so this
will not appear for  default log level

>
> OBSERVATION #4
>
> When 1000s of sequences are refreshed then there are 1000s of
> associated logs. But (given there is only one sequencesync worker)
> those logs are not always the order that I was expecting to see them.
>
> e.g.
> ...
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0885" has
> finished
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0887" has
> finished
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0888" has
> finished
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0889" has
> finished
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0890" has
> finished
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0906" has
> finished
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0566" has
> finished
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0568" has
> finished
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0569" has
> finished
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0570" has
> finished
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0571" has
> finished
> 2024-08-13 16:41:47.436 AEST [11735] LOG:  logical replication
> synchronization for subscription "sub3", sequence "seq_0582" has
> finished
> ...
>
> Is there a way to refresh sequences in a more natural (e.g.
> alphabetical) order to make these logs more readable?

I felt this is ok, no need to order it as it can easily be done using
some scripts if required from logs.

The rest of the issues were fixed, the v20240813 version patch
attached at [1] has the changes for the same.
[1] - https://www.postgresql.org/message-id/CALDaNm1Nr_n9SBB52L8A10Txyb4nqGJWfHUapwzM5BopvjMhjA%40mail.gmail.com

Regards,
Vignesh

On Tue, Aug 13, 2024 at 10:00 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, 13 Aug 2024 at 09:19, Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > 3.1. GENERAL
> >
> > Hmm. I am guessing this was provided as a separate patch to aid review
> > by showing that existing functions are moved? OTOH you can't really
> > judge this patch properly without already knowing details of what will
> > come next in the sequencesync. i.e. As a *standalone* patch without
> > the sequencesync.c the refactoring doesn't make much sense.
> >
> > Maybe it is OK later to combine patches 0003 and 0004. Alternatively,
> > keep this patch separated but give greater emphasis in the comment
> > header to say this patch only exists separately in order to help the
> > review.
>
> I have kept this patch only to show that this patch as such has no
> code changes. If we move this to the next patch it will be difficult
> for reviewers to know which is new code and which is old code. During
> commit we can merge this with the next one. I felt it is better to add
> it in the commit message instead of comment header so updated the
> commit message.
>

Yes, I wrote "comment header" but it was a typo; I meant "commit
header". What you did looks good now. Thanks.

> > ~
> >
> > 3.4. function names
> >
> > With the re-shuffling that this patch does, and changing several from
> > static to not-static, should the function names remain as they are?
> > They look random to me.
> > - finish_sync_worker(void)
> > - invalidate_syncing_table_states(Datum arg, int cacheid, uint32 hashvalue)
> > - FetchTableStates(bool *started_tx)
> > - process_syncing_tables(XLogRecPtr current_lsn)
> >
> > I think using a consistent naming convention would be better. e.g.
> > SyncFinishWorker
> > SyncInvalidateTableStates
> > SyncFetchTableStates
> > SyncProcessTables
>
> One advantage with keeping the existing names the same wherever
> possible will help while merging the changes to back-branches. So I'm
> not making this change.
>

According to my understanding, the logical replication code tries to
maintain name conventions for static functions (snake_case) and for
non-static functions (CamelCase) as an aid for code readability. I
think we should either do our best to abide by those conventions, or
we might as well just forget them and have a naming free-for-all.
Since the new syncutils.c module is being introduced by this patch, my
guess is that any future merging to back-branches will be affected
regardless. IMO this is an ideal opportunity to try to nudge the
function names in the right direction. YMMV.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi Vignesh, Here are my review comments for the latest patchset:

Patch v20240813-0001. No comments
Patch v20240813-0002. No comments
Patch v20240813-0003. No comments
Patch v20240813-0004. See below
Patch v20240813-0005. No comments

//////

Patch v20240813-0004

======
src/backend/catalog/pg_subscription.

GetSubscriptionRelations:
nit - modify a condition for readability

======
src/backend/commands/subscriptioncmds.c

fetch_sequence_list:
nit - changed the WARNING message. /parameters differ
between.../parameters differ for.../ (FYI, Chat-GPT agrees that 2nd
way is more correct)
nit - other minor changes to the message and hint

======
.../replication/logical/sequencesync.c

1. LogicalRepSyncSequences

+ ereport(DEBUG1,
+ errmsg("logical replication synchronization for subscription \"%s\",
sequence \"%s\" has finished",
+    get_subscription_name(subid, false), get_rel_name(done_seq->relid)));

DEBUG logs should use errmsg_internal. (fixed also nitpicks attachment).

~

nit - minor change to the log message counting the batched sequences

~~~

process_syncing_sequences_for_apply:
nit - /sequence sync worker/seqeuencesync worker/

======
src/backend/utils/misc/guc_tables.c

nit - /max workers/maximum number of workers/ (for consistency because
all other GUCs are verbose like this; nothing just says "max".)

======
src/test/subscription/t/034_sequences.pl

nit - adjust the expected WARNING message (which was modified above)

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Here are my review comments for the latest patchset

v20240817-0001. No changes. No comments.
v20240817-0002. No changes. No comments.
v20240817-0003. See below.
v20240817-0004. See below.
v20240817-0005. No changes. No comments.

//////

v20240817-0003 and 0004.

(This is a repeat of the same comment as in previous reviews, but lots
more functions seem affected now)

IIUC, the LR code tries to follow function naming conventions (e.g.
CamelCase/snake_case for exposed/static functions respectively),
intended to make the code more readable. But, this only works if the
conventions are followed.

Now, patches 0003 and 0004 are shuffling more and more functions
between modules while changing them from static to non-static (or vice
versa). So, the function name conventions are being violated many
times. IMO these functions ought to be renamed according to their new
modifiers to avoid the confusion caused by ignoring the name
conventions.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi Vignesh, Here are my review comments for the latest patchset

v20240819-0001. No changes. No comments.
v20240819-0002. No changes. No comments.
v20240819-0003. See below.
v20240819-0004. See below.
v20240819-0005. No changes. No comments.

///////////////////////

PATCH v20240819-0003

======
src/backend/replication/logical/syncutils.c

3.1.
+typedef enum
+{
+ SYNC_RELATION_STATE_NEEDS_REBUILD,
+ SYNC_RELATION_STATE_REBUILD_STARTED,
+ SYNC_RELATION_STATE_VALID,
+} SyncingRelationsState;
+
+static SyncingRelationsState relation_states_validity =
SYNC_RELATION_STATE_NEEDS_REBUILD;

There is some muddle of singular/plural names here. The
typedef/values/var should all match:

e.g. It could be like:
SYNC_RELATION_STATE_xxx --> SYNC_RELATION_STATES_xxx
SyncingRelationsState --> SyncRelationStates

But, a more radical change might be better.

typedef enum
{
RELATION_STATES_SYNC_NEEDED,
RELATION_STATES_SYNC_STARTED,
RELATION_STATES_SYNCED,
} SyncRelationStates;

~~~

3.2. GENERAL refactoring

I don't think all of the functions moved into syncutil.c truly belong there.

This new module was introduced to be for common/util functions for
tablesync and sequencesync, but with each patchset, it has been
sucking in more and more functions that maybe do not quite belong
here.

For example, AFAIK these below have logic that is *solely* for TABLES
(not for SEQUENCES). Perhaps it was convenient to dump them here
because they are statically called, but I felt they still logically
belong in tablesync.c:
- process_syncing_tables_for_sync(XLogRecPtr current_lsn)
- process_syncing_tables_for_apply(XLogRecPtr current_lsn)
- AllTablesyncsReady(void)

~~~

3.3.
+static bool
+FetchRelationStates(bool *started_tx)
+{

If this function can remain static then the name should change to be
like fetch_table_states, right?

======
src/include/replication/worker_internal.h

3.4.
+extern bool wait_for_relation_state_change(Oid relid, char expected_state);

If this previously static function will be exposed now (it may not
need to be if some other functions are returned tablesync.c) then the
function name should also be changed, right?

////////////////////////

PATCH v20240819-0004

======
src/backend/replication/logical/syncutils.c

4.1 GENERAL refactoring

(this is similar to review comment #3.2 above)

Functions like below have logic that is *solely* for SEQUENCES (not
for TABLES). I felt they logically belong in sequencesync.c, not here.
- process_syncing_sequences_for_apply(void)

~~~

FetchRelationStates:
nit - the comment change about "not-READY tables" (instead of
relations) should be already in patch 0003.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Fri, Sep 20, 2024 at 9:36 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 21 Aug 2024 at 11:54, vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, 21 Aug 2024 at 08:33, Peter Smith <smithpb2250@gmail.com> wrote:
> > >
> > > Hi Vignesh, Here are my only review comments for the latest patch set.
> >
> > Thanks, these issues have been addressed in the updated version.
> > Additionally, I have fixed the pgindent problems that were reported
> > and included another advantage of this design in the file header of
> > the sequencesync file.
>
> The patch was not applied on top of head, here is a rebased version of
> the patches.
> I have also removed an invalidation which was  not required for
> sequences and a typo.
>

Thank You for the patches. I would like to understand srsublsn and
page_lsn more. Please see the scenario below:

I have a sequence:
CREATE SEQUENCE myseq0 INCREMENT 5 START 100;

After refresh on sub:
postgres=# ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
ALTER SUBSCRIPTION

postgres=# select * from pg_subscription_rel;
 srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
   16385 |   16384 | r          | 0/152F380 -->pub's page_lsn


postgres=# select * from pg_sequence_state('myseq0');
 page_lsn  | last_value | log_cnt | is_called
-----------+------------+---------+-----------
 0/152D830 |        105 |      31 | t   -->(I am assuming 0/152D830 is
local page_lsn corresponding to value-=105)

Now I assume that *only* after doing next_wal for 31 times,  page_lsn
shall change. But I observe strange behaviour

After running nextval on sub for 7 times:
postgres=# select * from pg_sequence_state('myseq0');
 page_lsn  | last_value | log_cnt | is_called
-----------+------------+---------+-----------
 0/152D830 |        140 |      24 | t   -->correct

After running nextval on sub for 15 more times:
postgres=# select * from pg_sequence_state('myseq0');
 page_lsn  | last_value | log_cnt | is_called
-----------+------------+---------+-----------
 0/152D830 |        215 |       9 | t -->correct
(1 row)

Now after running it 6 more times:
postgres=# select * from pg_sequence_state('myseq0');
 page_lsn  | last_value | log_cnt | is_called
-----------+------------+---------+-----------
 0/152D990 |        245 |      28 | t --> how??

last_value increased in the expected way (6*5), but page_lsn changed
and log_cnt changed before we could complete the remaining runs as
well. Not sure why??

Now if I do refresh again:

postgres=# ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
ALTER SUBSCRIPTION

postgres=# select * from pg_subscription_rel;
 srsubid | srrelid | srsubstate | srsublsn
---------+---------+------------+-----------
   16385 |   16384 | r          | 0/152F380-->pub's page_lsn, same as old one.

postgres=# select * from pg_sequence_state('myseq0');
 page_lsn  | last_value | log_cnt | is_called
-----------+------------+---------+-----------
 0/152DDB8 |        105 |      31 | t
(1 row)

Now, what is this page_lsn = 0/152DDB8? Should it be the one
corresponding to last_value=105 and thus shouldn't it match the
previous value of  0/152D830?

thanks
Shveta

On Thu, 26 Sept 2024 at 11:07, shveta malik <shveta.malik@gmail.com> wrote:
>
> On Fri, Sep 20, 2024 at 9:36 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, 21 Aug 2024 at 11:54, vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Wed, 21 Aug 2024 at 08:33, Peter Smith <smithpb2250@gmail.com> wrote:
> > > >
> > > > Hi Vignesh, Here are my only review comments for the latest patch set.
> > >
> > > Thanks, these issues have been addressed in the updated version.
> > > Additionally, I have fixed the pgindent problems that were reported
> > > and included another advantage of this design in the file header of
> > > the sequencesync file.
> >
> > The patch was not applied on top of head, here is a rebased version of
> > the patches.
> > I have also removed an invalidation which was  not required for
> > sequences and a typo.
> >
>
> Thank You for the patches. I would like to understand srsublsn and
> page_lsn more. Please see the scenario below:
>
> I have a sequence:
> CREATE SEQUENCE myseq0 INCREMENT 5 START 100;
>
> After refresh on sub:
> postgres=# ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
> ALTER SUBSCRIPTION
>
> postgres=# select * from pg_subscription_rel;
>  srsubid | srrelid | srsubstate | srsublsn
> ---------+---------+------------+-----------
>    16385 |   16384 | r          | 0/152F380 -->pub's page_lsn
>
>
> postgres=# select * from pg_sequence_state('myseq0');
>  page_lsn  | last_value | log_cnt | is_called
> -----------+------------+---------+-----------
>  0/152D830 |        105 |      31 | t   -->(I am assuming 0/152D830 is
> local page_lsn corresponding to value-=105)
>
> Now I assume that *only* after doing next_wal for 31 times,  page_lsn
> shall change. But I observe strange behaviour
>
> After running nextval on sub for 7 times:
> postgres=# select * from pg_sequence_state('myseq0');
>  page_lsn  | last_value | log_cnt | is_called
> -----------+------------+---------+-----------
>  0/152D830 |        140 |      24 | t   -->correct
>
> After running nextval on sub for 15 more times:
> postgres=# select * from pg_sequence_state('myseq0');
>  page_lsn  | last_value | log_cnt | is_called
> -----------+------------+---------+-----------
>  0/152D830 |        215 |       9 | t -->correct
> (1 row)
>
> Now after running it 6 more times:
> postgres=# select * from pg_sequence_state('myseq0');
>  page_lsn  | last_value | log_cnt | is_called
> -----------+------------+---------+-----------
>  0/152D990 |        245 |      28 | t --> how??
>
> last_value increased in the expected way (6*5), but page_lsn changed
> and log_cnt changed before we could complete the remaining runs as
> well. Not sure why??

This can occur if a checkpoint happened at that time. The regression
test also has specific handling for this, as noted in a comment within
the sequence.sql test file:
-- log_cnt can be higher if there is a checkpoint just at the right
-- time

> Now if I do refresh again:
>
> postgres=# ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
> ALTER SUBSCRIPTION
>
> postgres=# select * from pg_subscription_rel;
>  srsubid | srrelid | srsubstate | srsublsn
> ---------+---------+------------+-----------
>    16385 |   16384 | r          | 0/152F380-->pub's page_lsn, same as old one.
>
> postgres=# select * from pg_sequence_state('myseq0');
>  page_lsn  | last_value | log_cnt | is_called
> -----------+------------+---------+-----------
>  0/152DDB8 |        105 |      31 | t
> (1 row)
>
> Now, what is this page_lsn = 0/152DDB8? Should it be the one
> corresponding to last_value=105 and thus shouldn't it match the
> previous value of  0/152D830?

After executing REFRESH PUBLICATION SEQUENCES, the publication value
will be resynchronized, and a new LSN will be generated and updated
for the publisher sequence (using the old value). Therefore, this is
not a concern.

Regards,
Vignesh

On Sun, Sep 29, 2024 at 12:34 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 26 Sept 2024 at 11:07, shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Fri, Sep 20, 2024 at 9:36 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Wed, 21 Aug 2024 at 11:54, vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > On Wed, 21 Aug 2024 at 08:33, Peter Smith <smithpb2250@gmail.com> wrote:
> > > > >
> > > > > Hi Vignesh, Here are my only review comments for the latest patch set.
> > > >
> > > > Thanks, these issues have been addressed in the updated version.
> > > > Additionally, I have fixed the pgindent problems that were reported
> > > > and included another advantage of this design in the file header of
> > > > the sequencesync file.
> > >
> > > The patch was not applied on top of head, here is a rebased version of
> > > the patches.
> > > I have also removed an invalidation which was  not required for
> > > sequences and a typo.
> > >
> >
> > Thank You for the patches. I would like to understand srsublsn and
> > page_lsn more. Please see the scenario below:
> >
> > I have a sequence:
> > CREATE SEQUENCE myseq0 INCREMENT 5 START 100;
> >
> > After refresh on sub:
> > postgres=# ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
> > ALTER SUBSCRIPTION
> >
> > postgres=# select * from pg_subscription_rel;
> >  srsubid | srrelid | srsubstate | srsublsn
> > ---------+---------+------------+-----------
> >    16385 |   16384 | r          | 0/152F380 -->pub's page_lsn
> >
> >
> > postgres=# select * from pg_sequence_state('myseq0');
> >  page_lsn  | last_value | log_cnt | is_called
> > -----------+------------+---------+-----------
> >  0/152D830 |        105 |      31 | t   -->(I am assuming 0/152D830 is
> > local page_lsn corresponding to value-=105)
> >
> > Now I assume that *only* after doing next_wal for 31 times,  page_lsn
> > shall change. But I observe strange behaviour
> >
> > After running nextval on sub for 7 times:
> > postgres=# select * from pg_sequence_state('myseq0');
> >  page_lsn  | last_value | log_cnt | is_called
> > -----------+------------+---------+-----------
> >  0/152D830 |        140 |      24 | t   -->correct
> >
> > After running nextval on sub for 15 more times:
> > postgres=# select * from pg_sequence_state('myseq0');
> >  page_lsn  | last_value | log_cnt | is_called
> > -----------+------------+---------+-----------
> >  0/152D830 |        215 |       9 | t -->correct
> > (1 row)
> >
> > Now after running it 6 more times:
> > postgres=# select * from pg_sequence_state('myseq0');
> >  page_lsn  | last_value | log_cnt | is_called
> > -----------+------------+---------+-----------
> >  0/152D990 |        245 |      28 | t --> how??
> >
> > last_value increased in the expected way (6*5), but page_lsn changed
> > and log_cnt changed before we could complete the remaining runs as
> > well. Not sure why??
>
> This can occur if a checkpoint happened at that time. The regression
> test also has specific handling for this, as noted in a comment within
> the sequence.sql test file:
> -- log_cnt can be higher if there is a checkpoint just at the right
> -- time

Okay. I see. I tried by executing 'checkpoint' and can see the same behaviour.

>
> > Now if I do refresh again:
> >
> > postgres=# ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
> > ALTER SUBSCRIPTION
> >
> > postgres=# select * from pg_subscription_rel;
> >  srsubid | srrelid | srsubstate | srsublsn
> > ---------+---------+------------+-----------
> >    16385 |   16384 | r          | 0/152F380-->pub's page_lsn, same as old one.
> >
> > postgres=# select * from pg_sequence_state('myseq0');
> >  page_lsn  | last_value | log_cnt | is_called
> > -----------+------------+---------+-----------
> >  0/152DDB8 |        105 |      31 | t
> > (1 row)
> >
> > Now, what is this page_lsn = 0/152DDB8? Should it be the one
> > corresponding to last_value=105 and thus shouldn't it match the
> > previous value of  0/152D830?
>
> After executing REFRESH PUBLICATION SEQUENCES, the publication value
> will be resynchronized, and a new LSN will be generated and updated
> for the publisher sequence (using the old value). Therefore, this is
> not a concern.
>

Okay.

Few comments:

1)
+static List *
+fetch_sequence_list(WalReceiverConn *wrconn, char *subname, List *publications)

--fetch_sequence_list() is not using the argument subanme anywhere.

2)

+ if (resync_all_sequences)
+ {
+ ereport(DEBUG1,
+ errmsg_internal("sequence \"%s.%s\" of subscription \"%s\" set to INIT state",
+ get_namespace_name(get_rel_namespace(relid)),
+ get_rel_name(relid),
+ sub->name));
+ UpdateSubscriptionRelState(sub->oid, relid, SUBREL_STATE_INIT,
+    InvalidXLogRecPtr);
+ }

--Shall we have DEBUG1 after we are done with
UpdateSubscriptionRelState? Otherwise we may end up putting this log
statement, even if the update fails for some reason.

3)
fetch_remote_sequence_data():

Should we have a macro REMOTE_SEQ_COL_COUNT 10 and use it instead of
direct 10. Also instead of  having 1,2,3 etc in slot_getattr, we can
have ++col and at the end we can have:
Assert(col == REMOTE_SEQ_COL_COUNT);

thanks
Shveta

On Tue, Oct 8, 2024 at 2:46 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 4 Oct 2024 at 15:39, shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Sun, Sep 29, 2024 at 12:34 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Thu, 26 Sept 2024 at 11:07, shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > On Fri, Sep 20, 2024 at 9:36 AM vignesh C <vignesh21@gmail.com> wrote:
> > > > >
> > > > > On Wed, 21 Aug 2024 at 11:54, vignesh C <vignesh21@gmail.com> wrote:
> > > > > >
> > > > > > On Wed, 21 Aug 2024 at 08:33, Peter Smith <smithpb2250@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi Vignesh, Here are my only review comments for the latest patch set.
> > > > > >
> > > > > > Thanks, these issues have been addressed in the updated version.
> > > > > > Additionally, I have fixed the pgindent problems that were reported
> > > > > > and included another advantage of this design in the file header of
> > > > > > the sequencesync file.
> > > > >
> > > > > The patch was not applied on top of head, here is a rebased version of
> > > > > the patches.
> > > > > I have also removed an invalidation which was  not required for
> > > > > sequences and a typo.
> > > > >
> > > >
> > > > Thank You for the patches. I would like to understand srsublsn and
> > > > page_lsn more. Please see the scenario below:
> > > >
> > > > I have a sequence:
> > > > CREATE SEQUENCE myseq0 INCREMENT 5 START 100;
> > > >
> > > > After refresh on sub:
> > > > postgres=# ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
> > > > ALTER SUBSCRIPTION
> > > >
> > > > postgres=# select * from pg_subscription_rel;
> > > >  srsubid | srrelid | srsubstate | srsublsn
> > > > ---------+---------+------------+-----------
> > > >    16385 |   16384 | r          | 0/152F380 -->pub's page_lsn
> > > >
> > > >
> > > > postgres=# select * from pg_sequence_state('myseq0');
> > > >  page_lsn  | last_value | log_cnt | is_called
> > > > -----------+------------+---------+-----------
> > > >  0/152D830 |        105 |      31 | t   -->(I am assuming 0/152D830 is
> > > > local page_lsn corresponding to value-=105)
> > > >
> > > > Now I assume that *only* after doing next_wal for 31 times,  page_lsn
> > > > shall change. But I observe strange behaviour
> > > >
> > > > After running nextval on sub for 7 times:
> > > > postgres=# select * from pg_sequence_state('myseq0');
> > > >  page_lsn  | last_value | log_cnt | is_called
> > > > -----------+------------+---------+-----------
> > > >  0/152D830 |        140 |      24 | t   -->correct
> > > >
> > > > After running nextval on sub for 15 more times:
> > > > postgres=# select * from pg_sequence_state('myseq0');
> > > >  page_lsn  | last_value | log_cnt | is_called
> > > > -----------+------------+---------+-----------
> > > >  0/152D830 |        215 |       9 | t -->correct
> > > > (1 row)
> > > >
> > > > Now after running it 6 more times:
> > > > postgres=# select * from pg_sequence_state('myseq0');
> > > >  page_lsn  | last_value | log_cnt | is_called
> > > > -----------+------------+---------+-----------
> > > >  0/152D990 |        245 |      28 | t --> how??
> > > >
> > > > last_value increased in the expected way (6*5), but page_lsn changed
> > > > and log_cnt changed before we could complete the remaining runs as
> > > > well. Not sure why??
> > >
> > > This can occur if a checkpoint happened at that time. The regression
> > > test also has specific handling for this, as noted in a comment within
> > > the sequence.sql test file:
> > > -- log_cnt can be higher if there is a checkpoint just at the right
> > > -- time
> >
> > Okay. I see. I tried by executing 'checkpoint' and can see the same behaviour.
> >
> > >
> > > > Now if I do refresh again:
> > > >
> > > > postgres=# ALTER SUBSCRIPTION sub1 REFRESH PUBLICATION SEQUENCES;
> > > > ALTER SUBSCRIPTION
> > > >
> > > > postgres=# select * from pg_subscription_rel;
> > > >  srsubid | srrelid | srsubstate | srsublsn
> > > > ---------+---------+------------+-----------
> > > >    16385 |   16384 | r          | 0/152F380-->pub's page_lsn, same as old one.
> > > >
> > > > postgres=# select * from pg_sequence_state('myseq0');
> > > >  page_lsn  | last_value | log_cnt | is_called
> > > > -----------+------------+---------+-----------
> > > >  0/152DDB8 |        105 |      31 | t
> > > > (1 row)
> > > >
> > > > Now, what is this page_lsn = 0/152DDB8? Should it be the one
> > > > corresponding to last_value=105 and thus shouldn't it match the
> > > > previous value of  0/152D830?
> > >
> > > After executing REFRESH PUBLICATION SEQUENCES, the publication value
> > > will be resynchronized, and a new LSN will be generated and updated
> > > for the publisher sequence (using the old value). Therefore, this is
> > > not a concern.
> > >
> >
> > Okay.
> >
> > Few comments:
> >
> > 1)
> > +static List *
> > +fetch_sequence_list(WalReceiverConn *wrconn, char *subname, List *publications)
> >
> > --fetch_sequence_list() is not using the argument subanme anywhere.
> >
> > 2)
> >
> > + if (resync_all_sequences)
> > + {
> > + ereport(DEBUG1,
> > + errmsg_internal("sequence \"%s.%s\" of subscription \"%s\" set to INIT state",
> > + get_namespace_name(get_rel_namespace(relid)),
> > + get_rel_name(relid),
> > + sub->name));
> > + UpdateSubscriptionRelState(sub->oid, relid, SUBREL_STATE_INIT,
> > +    InvalidXLogRecPtr);
> > + }
> >
> > --Shall we have DEBUG1 after we are done with
> > UpdateSubscriptionRelState? Otherwise we may end up putting this log
> > statement, even if the update fails for some reason.
> >
> > 3)
> > fetch_remote_sequence_data():
> >
> > Should we have a macro REMOTE_SEQ_COL_COUNT 10 and use it instead of
> > direct 10. Also instead of  having 1,2,3 etc in slot_getattr, we can
> > have ++col and at the end we can have:
> > Assert(col == REMOTE_SEQ_COL_COUNT);
>
> Thanks for the comments, these are addressed in the attached patch.
>

Here are comments on the 0001 and 0002 patches:

0001 patch:

read_seq_tuple() reads a buffer and acquires a lock on it, and the
buffer is returned to the caller while being locked. So I think it's
possible for the caller to get the page LSN even without changes.
Since pg_sequence_state() is the sole caller that requests lsn_ret to
be set, I think the changes of read_seq_tuples() is not necessarily
necessary.

0002 patch:
+        Assert(all_tables && *all_tables == false);
+        Assert(all_sequences && *all_sequences == false);

I think it's better to set both *all_tables and *all_sequence to false
at the beginning of the function to ensure this function works as
expected regardless of their initial values.

---
        appendPQExpBufferStr(query,
                             "SELECT p.tableoid, p.oid, p.pubname, "
                             "p.pubowner, "
-                            "p.puballtables, p.pubinsert,
p.pubupdate, p.pubdelete, p.pubtruncate, p.pubviaroot "
+                            "p.puballtables, false as
p.puballsequences, p.pubinsert, p.pubupdate, p.pubdelete,
p.pubtruncate, p.pubviaroot "
                             "FROM pg_publication p");
    else if (fout->remoteVersion >= 110000)
        appendPQExpBufferStr(query,
                             "SELECT p.tableoid, p.oid, p.pubname, "
                             "p.pubowner, "
-                            "p.puballtables, p.pubinsert,
p.pubupdate, p.pubdelete, p.pubtruncate, false AS pubviaroot "
+                            "p.puballtables, false as
p.puballsequences, p.pubinsert, p.pubupdate, p.pubdelete,
p.pubtruncate, false AS pubviaroot "
                             "FROM pg_publication p");
    else
        appendPQExpBufferStr(query,
                             "SELECT p.tableoid, p.oid, p.pubname, "
                             "p.pubowner, "
-                            "p.puballtables, p.pubinsert,
p.pubupdate, p.pubdelete, false AS pubtruncate, false AS pubviaroot "
+                            "p.puballtables, false as
p.puballsequences, p.pubinsert, p.pubupdate, p.pubdelete, false AS
pubtruncate, false AS pubviaroot "
                             "FROM pg_publication p");

The column name should be puballsequences, not p.puballsequences.

---
IIUC the changes of describeOneTableDetails() includes two kinds of
changes: refactoring to use printTable() instead of printQuery(), and
adding publications that includes the sequence. Is the first
refactoring necessary for the second change? If not, should it be done
in a separate patch?
fg
Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Hi Vignesh,

Here are some review comments for the patch v20241211-0002.

======
doc/src/sgml/ref/create_publication.sgml

1.
+<phrase>where <replaceable class="parameter">object
type</replaceable> is one of:</phrase>
+
+    TABLES
+    SEQUENCES

The replaceable "object_type" is missing an underscore.

~~~

publish option effect fro SEQUENCE replication?

2.
It's not obvious to me if the SEQUENCE replication stuff is affected
but the setting of pubactions (ie the 'publish' option). I'm thinking
that probably anything to do with SEQUENCEs may no be considered a DML
operation, but if that is true it might be better to explicitly say
so.

Also, we might need to include a test to show even if publish='' that
the SEQUENCE synchronization is not affected.


======
src/backend/commands/publicationcmds.c

CreatePublication:

3.
- /* FOR ALL TABLES requires superuser */
- if (stmt->for_all_tables && !superuser())
+ /* FOR ALL TABLES or FOR ALL SEQUENCES requires superuser */
+ if ((stmt->for_all_tables || stmt->for_all_sequences) && !superuser())
  ereport(ERROR,
  (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
- errmsg("must be superuser to create FOR ALL TABLES publication")));
+ errmsg("must be superuser to create a %s publication",
+ stmt->for_all_tables ? "FOR ALL TABLES" :
+ "FOR ALL SEQUENCES")));

It seems a quirk that if FOR ALL TABLES and FOR ALL SEQUENCES are
specified at the same time then you would only get the "FOR ALL
TABLES" error, but maybe that is OK?

~~~

AlterPublicationOwner_internal:

4.
Ditto quirk as commented above, but maybe it is OK.

======
src/bin/psql/describe.c

describePublications:

5.
It seems the ordering of the local variables, and then the attributes
in the SQL, and the headings in the "describe" output are a bit
muddled.

IMO it might be better to always keep things in the same order as the
eventual display headings. So anything to do with puballsequences
should be immediately after anything to do with puballtables.

(There are multiple changes needed in this function to rearrange
things to be this way).

~~~

6.
The following seems wrong because now somehow there are two lots of
index 9 (???)

-----
printTableAddCell(&cont, PQgetvalue(res, i, 2), false, false);
printTableAddCell(&cont, PQgetvalue(res, i, 3), false, false);
if (has_pubsequence)
  printTableAddCell(&cont, PQgetvalue(res, i, 9), false, false); /*
all sequences */
printTableAddCell(&cont, PQgetvalue(res, i, 4), false, false);
printTableAddCell(&cont, PQgetvalue(res, i, 5), false, false);
printTableAddCell(&cont, PQgetvalue(res, i, 6), false, false);
if (has_pubtruncate)
  printTableAddCell(&cont, PQgetvalue(res, i, 7), false, false);
if (has_pubgencols)
  printTableAddCell(&cont, PQgetvalue(res, i, 8), false, false);
if (has_pubviaroot)
  printTableAddCell(&cont, PQgetvalue(res, i, 9), false, false);
-----


======
src/test/regress/expected/psql.out

7.
+\dRp+ regress_pub_forallsequences1
+                                            Publication
regress_pub_forallsequences1
+          Owner           | All tables | All sequences | Inserts |
Updates | Deletes | Truncates | Generated columns | Via root

+--------------------------+------------+---------------+---------+---------+---------+-----------+-------------------+----------
+ regress_publication_user | f          | f             | t       | t
     | t       | t         | f                 | f
+(1 row)
+

The expected value of 'f' for "All sequences" looks wrong to me. I
think this might be a manifestation of that duplicated '9' index
mentioned in an earlier review comment #6.

~~~

8.
+\dRp+ regress_pub_for_allsequences_alltables
+                                       Publication
regress_pub_for_allsequences_alltables
+          Owner           | All tables | All sequences | Inserts |
Updates | Deletes | Truncates | Generated columns | Via root

+--------------------------+------------+---------------+---------+---------+---------+-----------+-------------------+----------
+ regress_publication_user | t          | f             | t       | t
     | t       | t         | f                 | f
+(1 row)
+

The expected value of 'f' for "All sequences" looks wrong to me. I
think this might be a manifestation of that duplicated '9' index
mentioned in an earlier review comment #6.

~~~

9.
 \dRp+ testpub_forparted
-                                         Publication testpub_forparted
-          Owner           | All tables | Inserts | Updates | Deletes
| Truncates | Generated columns | Via root
---------------------------+------------+---------+---------+---------+-----------+-------------------+----------
- regress_publication_user | f          | t       | t       | t
| t         | f                 | t
+                                                 Publication testpub_forparted
+          Owner           | All tables | All sequences | Inserts |
Updates | Deletes | Truncates | Generated columns | Via root

+--------------------------+------------+---------------+---------+---------+---------+-----------+-------------------+----------
+ regress_publication_user | f          | t             | t       | t
     | t       | t         | f                 | t


AFAIK these partition tests should not be impacting the "All
Sequences" flag value, so the expected value of 't' for "All
sequences" looks wrong to me. I think this might be a manifestation of
that duplicated '9' index mentioned in an earlier review comment #6.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi Vignesh,

Here are some review comments for the patch v20241211-0003.

======
src/backend/replication/logical/syncutils.c

1.
+typedef enum
+{
+ SYNC_RELATIONS_STATE_NEEDS_REBUILD,
+ SYNC_RELATIONS_STATE_REBUILD_STARTED,
+ SYNC_RELATIONS_STATE_VALID,
+} SyncingRelationsState;
+

Even though the patch's intent was only to "move" this from
tablsync.c, this enum probably deserved a comment describing its
purpose.

~~~

2.
+List    *table_states_not_ready = NIL;

Maybe it is convenient to move this, but there is something about this
list being exposed out of a "utils" module back to the tablesync.c
module that seems a bit strange. Would it make more sense for this to
be the other way around e.g. declared in the tablesync.c but exposed
to the syncutils.c? (and then similarly in subsequent patch 0004 the
sequence_states_not_ready would belong in the sequencesync.c)

~~~

3.
+/*
+ * Process possible state change(s) of tables that are being synchronized.
+ */
+void
+SyncProcessRelations(XLogRecPtr current_lsn)
+{

IIUC there was a deliberate effort to rename some comments and
functions to say "relations" instead of "tables". AFAICT that was done
to encompass the SEQUENCES which can fit under the umbrella of
"relations". Anyway, it becomes confusing sometimes when there is a
mismatch. For example, here is a function (relations) with a function
comment (tables).

~~~

4.
+/*
+ * Common code to fetch the up-to-date sync state info into the static lists.
+ *
+ * Returns true if subscription has 1 or more tables, else false.
+ *
+ * Note: If this function started the transaction (indicated by the parameter)
+ * then it is the caller's responsibility to commit it.
+ */
+bool
+FetchRelationStates(bool *started_tx)

Here is another place where the function name is "relations", but the
function comment refers to "tables".

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi Vignesh,

Here are some review comments for the patch v20241211-0004.

======
GENERAL

1.
There are more than a dozen places where the relation (relkind) is
checked to see if it is a SEQUENCE:

e.g. + get_rel_relkind(subrel->srrelid) != RELKIND_SEQUENCE &&
e.g. + if (get_rel_relkind(subrel->srrelid) != RELKIND_SEQUENCE)
e.g. + if (relkind == RELKIND_SEQUENCE && !get_sequences)
e.g. + if (relkind != RELKIND_SEQUENCE && !get_tables)
e.g. + relkind == RELKIND_SEQUENCE ? "sequence" : "table",
e.g. + if (relkind != RELKIND_SEQUENCE)
e.g. + relkind == RELKIND_SEQUENCE ? "sequence" : "table",
e.g. + if (get_rel_relkind(sub_remove_rels[off].relid) == RELKIND_SEQUENCE)
e.g. + if (get_rel_relkind(relid) != RELKIND_SEQUENCE)
e.g. + relkind != RELKIND_SEQUENCE)
e.g. + Assert(get_rel_relkind(rstate->relid) == RELKIND_SEQUENCE);
e.g. + Assert(relkind == RELKIND_SEQUENCE);
e.g. + if (get_rel_relkind(rstate->relid) == RELKIND_SEQUENCE)
e.g. + Assert(get_rel_relkind(rstate->relid) != RELKIND_SEQUENCE);

I am wondering if the code might be improved slightly by adding one new macro:

#define RELKIND_IS_SEQUENCE(relkind) ((relkind) == RELKIND_SEQUENCE)

======
Commit message

2.
1) CREATE SUBSCRIPTION
    - (PG17 command syntax is unchanged)
    - The subscriber retrieves sequences associated with publications.
    - Publisher sequences are added to pg_subscription_rel with INIT state.
    - Initiates the sequencesync worker (see above) to synchronize all
      sequences.

~

Shouldn't this say "Published sequences" instead of "Publisher sequences"?

I guess if the patch currently supports only ALL SEQUENCES then maybe
it amounts to the same thing, but IMO "Published" seems more correct.

~~~

3.
2) ALTER SUBSCRIPTION ... REFRESH PUBLICATION
    - (PG17 command syntax is unchanged)
    - Dropped publisher sequences are removed from pg_subscription_rel.
    - New publisher sequences are added to pg_subscription_rel with INIT state.
    - Initiates the sequencesync worker (see above) to synchronize only
      newly added sequences.

3) ALTER SUBSCRIPTION ... REFRESH PUBLICATION SEQUENCES
    - The patch introduces this new command to refresh all sequences
    - Dropped publisher sequences are removed from pg_subscription_rel.
    - New publisher sequences are added to pg_subscription_rel
    - All sequences in pg_subscription_rel are reset to INIT state.
    - Initiates the sequencesync worker (see above) to synchronize all
      sequences.

~

Ditto previous comment -- maybe those should say "Newly published
sequences" instead of "New publisher sequences"/

~~~

4.
Should there be some mention of the WARNING logged if sequence
parameter differences are detected?

======
src/backend/catalog/pg_subscription.c

GetSubscriptionRelations:

5.
+ * all_states:
+ * If getting tables, if all_states is true get all tables, otherwise
+ * only get tables that have not reached READY state.
+ * If getting sequences, if all_states is true get all sequences,
+ * otherwise only get sequences that are in INIT state.
+ *
+ * The returned list is palloc'ed in the current memory context.

and

- if (not_ready)
+ if (!all_states)
  ScanKeyInit(&skey[nkeys++],
  Anum_pg_subscription_rel_srsubstate,
  BTEqualStrategyNumber, F_CHARNE,
                    CharGetDatum(SUBREL_STATE_READY));
~

It was a bit confusing that the code for (!all_states) is excluding
everything that is not in SUBREL_STATE_READY, but OTOH the function
comment for sequences it said "that are in INIT state". Maybe that
function comment for sequence also should have said "that have not
reached READY state (i.e. are still in INIT state)" to better match
the code.

~~~

6.
+ /* Skip sequences if they were not requested */
+ if (relkind == RELKIND_SEQUENCE && !get_sequences)
+ continue;
+
+ /* Skip tables if they were not requested */
+ if (relkind != RELKIND_SEQUENCE && !get_tables)
+ continue;

Somehow, I feel this logic would seem simpler if expressed differently
to make it more explicit that the relation is either a table or a
sequence. e.g. by adding some variables.

SUGGESTION:

bool is_sequence;
bool is_table;

...

/* Relation is either a sequence or a table */
is_sequence = get_rel_relkind(subrel->srrelid) == RELKIND_SEQUENCE;
is_table = !is_sequence;

/* Skip sequences if they were not requested */
if (!get_sequences && is_sequence)
  continue;

/* Skip tables if they were not requested */
if (!get_tables && is_table)
  continue;

======
src/backend/commands/sequence.c

7.
 /*
- * Implement the 2 arg setval procedure.
- * See do_setval for discussion.
+ * Implement the 2 arg set sequence procedure.
+ * See SetSequence for discussion.
  */
 Datum
 setval_oid(PG_FUNCTION_ARGS)

~

Not sure this function comment is right. Shouldn't it still say
"Implement the 2 arg setval procedure."

~~~

8.
 /*
- * Implement the 3 arg setval procedure.
- * See do_setval for discussion.
+ * Implement the 3 arg set sequence procedure.
+ * See SetSequence for discussion.
  */
 Datum
 setval3_oid(PG_FUNCTION_ARGS)

~

Not sure this function comment is right. Shouldn't it still say
"Implement the 3 arg setval procedure."

======
src/backend/commands/subscriptioncmds.c

AlterSubscription_refresh:

9.
+ * If 'refresh_tables' is true, update the subscription by adding or removing
+ * tables that have been added or removed since the last subscription creation
+ * or refresh publication.
+ *
+ * If 'refresh_sequences' is true, update the subscription by adding
or removing
+ * sequences that have been added or removed since the last subscription
+ * creation or publication refresh.

The first para says "refresh publication". The second para says
"publication refresh", but I guess it should also be saying "refresh
publication".

~~~

10.
+ if (resync_all_sequences)
+ {
+
+ UpdateSubscriptionRelState(sub->oid, relid, SUBREL_STATE_INIT,
+    InvalidXLogRecPtr);
+ ereport(DEBUG1,
+ errmsg_internal("sequence \"%s.%s\" of subscription \"%s\" set to INIT state",
+ get_namespace_name(get_rel_namespace(relid)),
+ get_rel_name(relid),
+ sub->name));
+ }

Has unnecessary blank line.

~~~

AlterSubscription:

11.
Some of the generic error messages in this function are now
potentially misleading.

e.g. There are multiple places in this function that say "ALTER
SUBSCRIPTION ... REFRESH", meaning ALTER SUBSCRIPTION <subname>
REFRESH PUBLICATION, but not meaning ALTER SUBSCRIPTION <subname>
REFRESH PUBLICATION SEQUENCES, so possibly those need to be modified
to eliminate any ambiguity.

(Actually, maybe it is not only in this function -- the short form
"ALTER SUBSCRIPTION ... REFRESH" seems to be scattered in other
comments in this file also).

~~~

12.
- logicalrep_worker_stop(w->subid, w->relid);
+ /* Worker might have exited because of an error */
+ if (w->type == WORKERTYPE_UNKNOWN)
+ continue;
+
+ logicalrep_worker_stop(w->subid, w->relid, w->type);

It may be better to put that special case WORKERTYPE_UNKNOWN condition
as a quick exit within the logicalrep_worker_stop() function.

======
src/backend/replication/logical/launcher.c

logicalrep_worker_find:

13.
+ Assert(wtype == WORKERTYPE_TABLESYNC ||
+    wtype == WORKERTYPE_SEQUENCESYNC ||
+    wtype == WORKERTYPE_APPLY);
+

The master version of this function checked for
isParallelApplyWorker(w), but now if a WORKERTYPE_PARALLEL_APPLY ever
got to here it would fail the above Assert. So, is the patch code OK,
or do we still need to also account for a possible
WORKERTYPE_PARALLEL_APPLY reaching here?

~~~

logicalrep_worker_stop:

14.
worker = logicalrep_worker_find(subid, relid, wtype, false);

if (worker)
{
  Assert(!isParallelApplyWorker(worker));
  logicalrep_worker_stop_internal(worker, SIGTERM);
}

~

This code is not changed much from before, so it first finds the
worker, but then asserts that it must not be not a parallel apply
worker. But now, since the wtype is known and passed to the function
why not move the Assert up-front based on the wtype and before even
doing the 'find'?

======
.../replication/logical/sequencesync.c

ProcessSyncingSequencesForApply:

15.
+ /*
+ * Check if there is a sequence worker already running?
+ */
+ LWLockAcquire(LogicalRepWorkerLock, LW_SHARED);
+
+ syncworker = logicalrep_worker_find(MyLogicalRepWorker->subid,
+ InvalidOid, WORKERTYPE_SEQUENCESYNC,
+ true);
+ if (syncworker)
+ {
+ /* Now safe to release the LWLock */
+ LWLockRelease(LogicalRepWorkerLock);
+ break;
+ }

15a.
Comment: /sequence worker/sequencesync worker/

~

15b.
Maybe it's better to call this variable 'sequencesync_worker' or
similar because sync worker is too generic

~~~

fetch_remote_sequence_data:

16.
+/*
+ * fetch_remote_sequence_data
+ *
+ * Retrieves sequence data (last_value, log_cnt, page_lsn, and is_called)
+ * from a remote node.

The SELECT of this function fetch a lot more columns, so why are only
these few mentioned in the function comment?

~

17.
+ res = walrcv_exec(conn, cmd.data, REMOTE_SEQ_COL_COUNT, tableRow);
+ pfree(cmd.data);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ errmsg("could not receive sequence list from the publisher: %s",
+    res->err));

That error msg does seem quite right. IIUC, this is just the data from
a single sequence; it is not a "sequence list".

~~~

report_mismatched_sequences:

18.
+static void
+report_mismatched_sequences(StringInfo mismatched_seqs)
+{
+ if (mismatched_seqs->len)
+ {
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("parameters differ for the remote and local sequences (%s)
for subscription \"%s\"",
+    mismatched_seqs->data, MySubscription->name),
+ errhint("Alter/Re-create local sequences to have the same parameters
as the remote sequences."));
+
+ resetStringInfo(mismatched_seqs);
+ }
+}

I'm confused. The errhint says "Alter/Re-create local sequences to
have the same parameters", but in the function 'copy_sequence' there
was the code (below) that seems to be already setting the sequence
values (SetSequence) regardless of whether it detected
sequence_mismatch true/false.

+ *sequence_mismatch = !fetch_remote_sequence_data(conn, relid, remoteid,
+ nspname, relname,
+ &seq_log_cnt, &seq_is_called,
+ &seq_page_lsn, &seq_last_value);
+
+ SetSequence(RelationGetRelid(rel), seq_last_value, seq_is_called,
+ seq_log_cnt);

Is the setting regardless like that OK, or just going to lead to weird
integrity errors?

~~~

LogicalRepSyncSequences:

19.
+/*
+ * Start syncing the sequences in the sync worker.
+ */
+static void
+LogicalRepSyncSequences(void)

/sync worker/sequencesync worker/

~~~

20.
+ curr_seq++;
+
+ /*
+ * Have we reached the end of the current batch of sequences, or last
+ * remaining sequences to synchronize?
+ */
+ if (((curr_seq % MAX_SEQUENCES_SYNC_PER_BATCH) == 0) ||
+ curr_seq == seq_count)
+ {
+ /* LOG all the sequences synchronized during current batch. */
+ for (int i = (curr_seq - 1) - ((curr_seq - 1) % MAX_SEQUENCES_SYNC_PER_BATCH);
+ i < curr_seq; i++)
+ {

The calculation is quite tricky.

IMO this might all be simplified if you have another variable like
'batch_seq' that just ranges from 0 -- MAX_SEQUENCES_SYNC_PER_BATCH,
then do something like:

SUGGESTION:

cur_seq++;
batch_seq++;

if (batch_seq >= MAX_SEQUENCES_SYNC_PER_BATCH || cur_seq == seq_count)
{
  /* Process the batch. */

  ...

  /* LOG all the sequences synchronized during the current batch. */
  for (int i = 0; i < batch_seq; i++)
  {
    ...
  }

  /* Prepare for next batch */
  batch_seq = 0;
}

======
src/backend/replication/logical/syncutils.c

21.
 List    *table_states_not_ready = NIL;
+List    *sequence_states_not_ready = NIL;

I thought this declaration belonged more in the sequencesync.c file.

~~~

SyncProcessRelations:

22.
 /*
- * Process possible state change(s) of tables that are being synchronized.
+ * Process possible state change(s) of tables that are being synchronized and
+ * start new tablesync workers for the newly added tables and start new
+ * sequencesync worker for the newly added sequences.
  */

/added tables and start new sequencesync worker/added tables. Also,
start a new sequencesync worker/

~~~

FetchRelationStates:

23.
- * Note: If this function started the transaction (indicated by the parameter)
- * then it is the caller's responsibility to commit it.
+ * Returns true if subscription has 1 or more tables, else false.
  */
 bool
-FetchRelationStates(bool *started_tx)
+FetchRelationStates(void)

Partly because of the name (relations), I felt this might be better to
be a void function and the returned value would be passed back by
references (bool *has_tables).

======
src/backend/replication/logical/worker.c

SetupApplyOrSyncWorker:

24.
+
+ if (am_sequencesync_worker())
+ before_shmem_exit(logicalrep_seqsyncworker_failuretime, (Datum) 0);

There should be a comment saying what this callback is for.

======
Kind Regards,
Peter Smith.
Fujitsu Australia

Hi Vignesh.

Here are some review comments for the patch v20241211-0005.

======
doc/src/sgml/logical-replication.sgml

Section "29.6.1. Sequence Definition Mismatches"

1.
+   <warning>
+    <para>
+     If there are differences in sequence definitions between the publisher and
+     subscriber, a WARNING is logged.
+    </para>
+   </warning>

Maybe this should say *when* this happens.

SUGGESTION
During sequence synchronization, the sequence definitions of the
publisher and the subscriber are compared. A WARNING is logged if any
differences are detected.

~~~

Section "29.6.3. Examples"

2.
Should the Examples section also have an example of ALTER SUBSCRIPTION
... REFRESH PUBLICATION to demonstrate (like in the TAP tests) that if
the sequences are already known, then those are not synchronised?

~~~

Section "29.8. Restrictions"

3.
+     Incremental sequence changes are not replicated.  The data in serial or
+     identity columns backed by sequences will of course be replicated as part
+     of the table, but the sequence itself would still show the start value on
+     the subscriber.  If the subscriber is used as a read-only database, then
+     this should typically not be a problem.  If, however, some kind of
+     switchover or failover to the subscriber database is intended, then the
+     sequences would need to be updated to the latest values, either
by executing
+     <link linkend="sql-altersubscription-params-refresh-publication-sequences">
+     <command>ALTER SUBSCRIPTION ... REFRESH PUBLICATION
SEQUENCES</command></link>
+     or by copying the current data from the publisher (perhaps using
+     <command>pg_dump</command>) or by determining a sufficiently high value
+     from the tables themselves.

I don't know if you need to mention it, or maybe it is too obvious,
but the suggestion here to use "ALTER SUBSCRIPTION ... REFRESH
PUBLICATION SEQUENCES" assumed you've already arranged for the
PUBLICATION to be publishing sequences before this.

======
doc/src/sgml/ref/alter_subscription.sgml

4.
          <para>
-          Specifies whether to copy pre-existing data in the publications
-          that are being subscribed to when the replication starts.
-          The default is <literal>true</literal>.
+          Specifies whether to copy pre-existing data for tables and
synchronize
+          sequences in the publications that are being subscribed to
when the replication
+          starts. The default is <literal>true</literal>.
          </para>

This is talking also about "synchronize sequences" when the
replication starts, but it is a bit confusing. IIUC, the .. REFRESH
PUBLICATION only synchronizes *newly added* sequences anyway, so does
it mean even that will not happen if copy_data=false?

I think this option needs more clarification on how it interacts with
sequences. Also, I don't recall seeing any test for sequences and
copy_data in the patch 0004 TAP tests, so maybe something needs to be
added there too.

~~~

5.
+     <para>
+      See <xref linkend="sequences-out-of-sync"/> for recommendations on how
+      to identify sequences and handle out-of-sync sequences.
+     </para>

/on how to identify sequences and handle out-of-sync sequences./on how
to identify and handle out-of-sync sequences./

======
Kind Regards,
Peter Smith.
Fujitsu Australia

On Thu, 19 Dec 2024 at 04:58, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh,
>
> Here are some review comments for the patch v20241211-0003.
>
> ~~~
>
> 4.
> +/*
> + * Common code to fetch the up-to-date sync state info into the static lists.
> + *
> + * Returns true if subscription has 1 or more tables, else false.
> + *
> + * Note: If this function started the transaction (indicated by the parameter)
> + * then it is the caller's responsibility to commit it.
> + */
> +bool
> +FetchRelationStates(bool *started_tx)
>
> Here is another place where the function name is "relations", but the
> function comment refers to "tables".

In this place the use of tables in comment is intentional, as the
return is based on subscription having any tables, and is not
applicable for sequence.

The rest of the comments are fixed and the changes for the same are
available at the v202412123 version patch shared at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm0FqKMqOdm7tNoT5KgK1BAMeeVnOXrSJ2024TscAbf4Og%40mail.gmail.com

Regards,
Vignesh

On Fri, 20 Dec 2024 at 08:05, Peter Smith <smithpb2250@gmail.com> wrote:
>
> Hi Vignesh.
>
> Here are some review comments for the patch v20241211-0005.
>
> ======
>
> Section "29.6.3. Examples"
>
> 2.
> Should the Examples section also have an example of ALTER SUBSCRIPTION
> ... REFRESH PUBLICATION to demonstrate (like in the TAP tests) that if
> the sequences are already known, then those are not synchronised?

I felt it is not required, let's not add too many examples.

> Section "29.8. Restrictions"
>
> 3.
> +     Incremental sequence changes are not replicated.  The data in serial or
> +     identity columns backed by sequences will of course be replicated as part
> +     of the table, but the sequence itself would still show the start value on
> +     the subscriber.  If the subscriber is used as a read-only database, then
> +     this should typically not be a problem.  If, however, some kind of
> +     switchover or failover to the subscriber database is intended, then the
> +     sequences would need to be updated to the latest values, either
> by executing
> +     <link linkend="sql-altersubscription-params-refresh-publication-sequences">
> +     <command>ALTER SUBSCRIPTION ... REFRESH PUBLICATION
> SEQUENCES</command></link>
> +     or by copying the current data from the publisher (perhaps using
> +     <command>pg_dump</command>) or by determining a sufficiently high value
> +     from the tables themselves.
>
> I don't know if you need to mention it, or maybe it is too obvious,
> but the suggestion here to use "ALTER SUBSCRIPTION ... REFRESH
> PUBLICATION SEQUENCES" assumed you've already arranged for the
> PUBLICATION to be publishing sequences before this.

I felt it is obvious, it need not be mentioned.

> ======
> doc/src/sgml/ref/alter_subscription.sgml
>
> 4.
>           <para>
> -          Specifies whether to copy pre-existing data in the publications
> -          that are being subscribed to when the replication starts.
> -          The default is <literal>true</literal>.
> +          Specifies whether to copy pre-existing data for tables and
> synchronize
> +          sequences in the publications that are being subscribed to
> when the replication
> +          starts. The default is <literal>true</literal>.
>           </para>
>
> This is talking also about "synchronize sequences" when the
> replication starts, but it is a bit confusing. IIUC, the .. REFRESH
> PUBLICATION only synchronizes *newly added* sequences anyway, so does
> it mean even that will not happen if copy_data=false?
>
> I think this option needs more clarification on how it interacts with
> sequences. Also, I don't recall seeing any test for sequences and
> copy_data in the patch 0004 TAP tests, so maybe something needs to be
> added there too.

This case is similar to how tables are handled, that is add the table
in subscription_rel and mark it as ready without updating the data.
Since tables and sequences behave the same way I have kept the
documentation the same. I have added a test for this in the 0004 TAP
test.

The rest of the comments are fixed and the changes for the same are
available in the v202412125 version patch at [1].
[1] - https://www.postgresql.org/message-id/CALDaNm0PbSAQvs34D%2BJ63SgmRUzDQHZ1W4aeW_An9pR_tXJnRA%40mail.gmail.com

Regards,
Vignesh