Thread: Using logical replication with older version subscribers
Hi, Logical replication enables us to replicate data changes to different major version PostgreSQL as the doc says[1]. However the current logical replication can work fine only if replicating to a newer major version PostgreSQL such as from 10 to 11. Regarding using logical replication with older major version, say sending from 11 to 10, it will stop when a subscriber receives a truncate change because it's not supported at PostgreSQL 10. I think there are use cases where using logical replication with a subscriber of an older version PostgreSQL but I'm not sure we should support it. Of course in such case we can set the publication with publish = 'insert, update, delete' to not send truncate changes but it requres users to recognize the feature differences between major vesions and in the future it will get more complex. So I think it would be better to be configured autometically by PostgreSQL. To fix it we can make subscribers send its supporting message types to the publisher at a startup time so that the publisher doesn't send unsupported message types on the subscriber. Or as an another idea, we can make subscribers ignore unsupported logical replication message types instead of raising an error. Feedback is very welcome. [1] https://www.postgresql.org/docs/devel/logical-replication.html Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
On Mon, Jan 7, 2019 at 9:01 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
Hi,
Logical replication enables us to replicate data changes to different
major version PostgreSQL as the doc says[1]. However the current
logical replication can work fine only if replicating to a newer major
version PostgreSQL such as from 10 to 11. Regarding using logical
replication with older major version, say sending from 11 to 10, it
will stop when a subscriber receives a truncate change because it's
not supported at PostgreSQL 10. I think there are use cases where
using logical replication with a subscriber of an older version
PostgreSQL but I'm not sure we should support it.
Of course in such case we can set the publication with publish =
'insert, update, delete' to not send truncate changes but it requres
users to recognize the feature differences between major vesions and
in the future it will get more complex. So I think it would be better
to be configured autometically by PostgreSQL.
To fix it we can make subscribers send its supporting message types to
the publisher at a startup time so that the publisher doesn't send
unsupported message types on the subscriber. Or as an another idea, we
can make subscribers ignore unsupported logical replication message
types instead of raising an error. Feedback is very welcome.
[1] https://www.postgresql.org/docs/devel/logical-replication.html
How would that work in practice?
If an 11 server is sent a message saying "client does not support truncate", and immediately generates an error, then you can no longer replicate even if you turn off truncate. And if it delays it until the actual replication of the item, then you just get the error on the primary ìnstead of the standby?
I assume you are not suggesting a publication with truncation enabled should just ignore replicating truncation if the downstream server doesn't support it? Because if that's the suggestion, then a strong -1 from me on that.
And definitely -1 for having a subscriber ignore messages it doesn't know about. That's setting oneself up for getting invalid data on the subscriber, because it skipped something that the publisher expected to be done.
On 07/01/2019 10:54, Magnus Hagander wrote: > I assume you are not suggesting a publication with truncation enabled > should just ignore replicating truncation if the downstream server > doesn't support it? Because if that's the suggestion, then a strong -1 > from me on that. Yes, that's the reason why we intentionally left it as it is now. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Mon, Jan 7, 2019 at 6:54 PM Magnus Hagander <magnus@hagander.net> wrote: > > On Mon, Jan 7, 2019 at 9:01 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote: >> >> Hi, >> >> Logical replication enables us to replicate data changes to different >> major version PostgreSQL as the doc says[1]. However the current >> logical replication can work fine only if replicating to a newer major >> version PostgreSQL such as from 10 to 11. Regarding using logical >> replication with older major version, say sending from 11 to 10, it >> will stop when a subscriber receives a truncate change because it's >> not supported at PostgreSQL 10. I think there are use cases where >> using logical replication with a subscriber of an older version >> PostgreSQL but I'm not sure we should support it. >> >> Of course in such case we can set the publication with publish = >> 'insert, update, delete' to not send truncate changes but it requres >> users to recognize the feature differences between major vesions and >> in the future it will get more complex. So I think it would be better >> to be configured autometically by PostgreSQL. >> >> To fix it we can make subscribers send its supporting message types to >> the publisher at a startup time so that the publisher doesn't send >> unsupported message types on the subscriber. Or as an another idea, we >> can make subscribers ignore unsupported logical replication message >> types instead of raising an error. Feedback is very welcome. >> >> [1] https://www.postgresql.org/docs/devel/logical-replication.html > > > How would that work in practice? > > If an 11 server is sent a message saying "client does not support truncate", and immediately generates an error, then youcan no longer replicate even if you turn off truncate. And if it delays it until the actual replication of the item, thenyou just get the error on the primary ìnstead of the standby? > > I assume you are not suggesting a publication with truncation enabled should just ignore replicating truncation if thedownstream server doesn't support it? Because if that's the suggestion, then a strong -1 from me on that. > I'm thinking that the we can make the pgoutput plugin recognize that the downstream server doesn't support it and not send it. For example, even if we create a publication with publish = 'truncate' we send nothing due to checking supported message types by pgoutput plugin if the downstream server is PostgreSQL server and its version is older than 10. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
On Mon, Jan 7, 2019 at 3:37 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Mon, Jan 7, 2019 at 6:54 PM Magnus Hagander <magnus@hagander.net> wrote:
>
> On Mon, Jan 7, 2019 at 9:01 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>
>> Hi,
>>
>> Logical replication enables us to replicate data changes to different
>> major version PostgreSQL as the doc says[1]. However the current
>> logical replication can work fine only if replicating to a newer major
>> version PostgreSQL such as from 10 to 11. Regarding using logical
>> replication with older major version, say sending from 11 to 10, it
>> will stop when a subscriber receives a truncate change because it's
>> not supported at PostgreSQL 10. I think there are use cases where
>> using logical replication with a subscriber of an older version
>> PostgreSQL but I'm not sure we should support it.
>>
>> Of course in such case we can set the publication with publish =
>> 'insert, update, delete' to not send truncate changes but it requres
>> users to recognize the feature differences between major vesions and
>> in the future it will get more complex. So I think it would be better
>> to be configured autometically by PostgreSQL.
>>
>> To fix it we can make subscribers send its supporting message types to
>> the publisher at a startup time so that the publisher doesn't send
>> unsupported message types on the subscriber. Or as an another idea, we
>> can make subscribers ignore unsupported logical replication message
>> types instead of raising an error. Feedback is very welcome.
>>
>> [1] https://www.postgresql.org/docs/devel/logical-replication.html
>
>
> How would that work in practice?
>
> If an 11 server is sent a message saying "client does not support truncate", and immediately generates an error, then you can no longer replicate even if you turn off truncate. And if it delays it until the actual replication of the item, then you just get the error on the primary ìnstead of the standby?
>
> I assume you are not suggesting a publication with truncation enabled should just ignore replicating truncation if the downstream server doesn't support it? Because if that's the suggestion, then a strong -1 from me on that.
>
I'm thinking that the we can make the pgoutput plugin recognize that
the downstream server doesn't support it and not send it. For example,
even if we create a publication with publish = 'truncate' we send
nothing due to checking supported message types by pgoutput plugin if
the downstream server is PostgreSQL server and its version is older
than 10.
That's the idea I definitely say a strong -1 to.
Ignoring the truncate message isn't going to make it work. It's just going to mean that the downstream data is incorrect vs what the publisher thought. The correct solution here is to not publish the truncate, which we already have. I can see the point in changing it so the error message becomes more obvious (already when the subscriber connects, and not a random time later when the first truncate replicates), but *silently* ignoring it seems like a terrible choice.
On Tue, Jan 8, 2019 at 1:12 AM Magnus Hagander <magnus@hagander.net> wrote: > > On Mon, Jan 7, 2019 at 3:37 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote: >> >> On Mon, Jan 7, 2019 at 6:54 PM Magnus Hagander <magnus@hagander.net> wrote: >> > >> > On Mon, Jan 7, 2019 at 9:01 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote: >> >> >> >> Hi, >> >> >> >> Logical replication enables us to replicate data changes to different >> >> major version PostgreSQL as the doc says[1]. However the current >> >> logical replication can work fine only if replicating to a newer major >> >> version PostgreSQL such as from 10 to 11. Regarding using logical >> >> replication with older major version, say sending from 11 to 10, it >> >> will stop when a subscriber receives a truncate change because it's >> >> not supported at PostgreSQL 10. I think there are use cases where >> >> using logical replication with a subscriber of an older version >> >> PostgreSQL but I'm not sure we should support it. >> >> >> >> Of course in such case we can set the publication with publish = >> >> 'insert, update, delete' to not send truncate changes but it requres >> >> users to recognize the feature differences between major vesions and >> >> in the future it will get more complex. So I think it would be better >> >> to be configured autometically by PostgreSQL. >> >> >> >> To fix it we can make subscribers send its supporting message types to >> >> the publisher at a startup time so that the publisher doesn't send >> >> unsupported message types on the subscriber. Or as an another idea, we >> >> can make subscribers ignore unsupported logical replication message >> >> types instead of raising an error. Feedback is very welcome. >> >> >> >> [1] https://www.postgresql.org/docs/devel/logical-replication.html >> > >> > >> > How would that work in practice? >> > >> > If an 11 server is sent a message saying "client does not support truncate", and immediately generates an error, thenyou can no longer replicate even if you turn off truncate. And if it delays it until the actual replication of the item,then you just get the error on the primary ìnstead of the standby? >> > >> > I assume you are not suggesting a publication with truncation enabled should just ignore replicating truncation if thedownstream server doesn't support it? Because if that's the suggestion, then a strong -1 from me on that. >> > >> >> I'm thinking that the we can make the pgoutput plugin recognize that >> the downstream server doesn't support it and not send it. For example, >> even if we create a publication with publish = 'truncate' we send >> nothing due to checking supported message types by pgoutput plugin if >> the downstream server is PostgreSQL server and its version is older >> than 10. > > > That's the idea I definitely say a strong -1 to. > > Ignoring the truncate message isn't going to make it work. It's just going to mean that the downstream data is incorrectvs what the publisher thought. The correct solution here is to not publish the truncate, which we already have.I can see the point in changing it so the error message becomes more obvious (already when the subscriber connects,and not a random time later when the first truncate replicates), but *silently* ignoring it seems like a terriblechoice. I understood that that makes more sense. And the raising the error when connection seems good to me. Thank you! Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center