Thread: Behaviour of take over the synchronous replication
Hi all,
I understand that setting synchronous_standby_name to '*' means that
all priority of standby server are same.
and the standby server, which connected to the master server at first,
become SYNC standby, another server become
ASYNC standby as potential server.
So, how to set the priority to standby server is different between '*'
and setting all standby names.
For example, if there are two standby servers (AAA and BBB)
(a) synchronous_standby_names = 'AAA,BBB'
application_name | sync_priority | sync_state
--------------------------+-------------------+------------ AAA | 1 | sync BBB
| 2 | potential
(b) synchronous_standby_names = '*'
application_name | sync_priority | sync_state
--------------------------+-------------------+------------ AAA | 1 | sync BBB
| 1 | potential
If the AAA server is failed, BBB server take over SYNC replication.
After take over, when AAA server starts, AAA server re-take over SYNC
replication from BBB server in both case (a) and case(b).
That is, AAA server starts as SYNC server again.
in case (a), those priority is clear. So I think that re-taking over
is correct behaviour.
OHOT, in case (b), even if AAA and BBB are set same priority, AAA
server steals SYNC replication.
I think it is better that BBB server continue behaviour SYNC standby,
and AAA should become potential server.
What am I missing?
Please give me feedback.
Regards,
-------
Sawada Masahiko
On 08/23/2013 12:42 AM, Sawada Masahiko wrote: > in case (a), those priority is clear. So I think that re-taking over > is correct behaviour. > OHOT, in case (b), even if AAA and BBB are set same priority, AAA > server steals SYNC replication. > I think it is better that BBB server continue behaviour SYNC standby, > and AAA should become potential server. So, you're saying that: 1) synchronous_standby_names = '*' 2) replica 'BBB' is the current sync standby 3) replica 'AAA' comes online 4) replica 'AAA' grabs sync status ? If that's the case, I'm not really sure that's undesirable behavior. One could argue fairly persuasively that if you care about the precendence order of sync replicas, you shouldn't use '*'. And the rule of "if using *, the lowest-sorted replica name has sync" is actually a predictable, easy-to-understand rule. So if you want to make this a feature request, you'll need to come up with an argument as to why the current behavior is bad. Otherwise, you're just asking us to document it better (which is a good idea). -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
<div dir="ltr"><p dir="ltr"><p dir="ltr">On Sat, Aug 24, 2013 at 3:14 AM, Josh Berkus <<a href="mailto:josh@agliodbs.com"target="_blank">josh@agliodbs.com</a>> wrote:<br /> > On 08/23/2013 12:42 AM, SawadaMasahiko wrote:<br /> >> in case (a), those priority is clear. So I think that re-taking over<br /> >>is correct behaviour.<br /> >> OHOT, in case (b), even if AAA and BBB are set same priority, AAA<br /> >>server steals SYNC replication.<br /> >> I think it is better that BBB server continue behaviour SYNC standby,<br/> >> and AAA should become potential server.<br /> ><br /> > So, you're saying that:<br /> ><br/> > 1) synchronous_standby_names = '*'<br /> ><br /> > 2) replica 'BBB' is the current sync standby<br />><br /> > 3) replica 'AAA' comes online<br /> ><br /> > 4) replica 'AAA' grabs sync status<br /> ><br />> ?<br />I'm sorry that you are confuse.<br /> It means that<p dir="ltr">1) synchronous_standby_names = '*'<p dir="ltr">2)replica 'AAA' is the current sync standby<p dir="ltr">3) replica 'BBB' is the current async standby (potentialsync standby)<p dir="ltr">4) replica 'AAA' fail. after that, replica 'BBB' is current sync standby.<p dir="ltr">5)replica 'AAA' comes online<p dir="ltr">6) replica 'AAA' grabs sync status<p dir="ltr">><br /> > If that'sthe case, I'm not really sure that's undesirable behavior.<br /> > One could argue fairly persuasively that if youcare about the<br /> > precendence order of sync replicas, you shouldn't use '*'. And the rule<br /> > of "if using*, the lowest-sorted replica name has sync" is actually a<br /> > predictable, easy-to-understand rule.<br /> ><br/> > So if you want to make this a feature request, you'll need to come up<br /> > with an argument as to whythe current behavior is bad. Otherwise,<br /> > you're just asking us to document it better (which is a good idea).<br/> It is not depend on name of standby server. That is, The standby server, which was connected to the master serverduring initial configration replication, is top priority even if priority of two server are same.<p dir="ltr">Usermust remember that which standby server connected to master server at first.<br /> I think that this behaviorconfuse user.<br /> so I think that we need to modify this behaviour or if '*' is used, priority of server is notsame (modifying manual is also good).<p dir="ltr">Regards,<p dir="ltr">-------<br /> Sawada Masahiko</div>
On Sat, Aug 24, 2013 at 2:46 PM, Sawada Masahiko <sawada.mshk@gmail.com> wrote: > On Sat, Aug 24, 2013 at 3:14 AM, Josh Berkus <josh@agliodbs.com> wrote: >> On 08/23/2013 12:42 AM, Sawada Masahiko wrote: >>> in case (a), those priority is clear. So I think that re-taking over >>> is correct behaviour. >>> OHOT, in case (b), even if AAA and BBB are set same priority, AAA >>> server steals SYNC replication. >>> I think it is better that BBB server continue behaviour SYNC standby, >>> and AAA should become potential server. >> >> So, you're saying that: >> >> 1) synchronous_standby_names = '*' >> >> 2) replica 'BBB' is the current sync standby >> >> 3) replica 'AAA' comes online >> >> 4) replica 'AAA' grabs sync status >> >> ? > I'm sorry that you are confuse. > It means that > > 1) synchronous_standby_names = '*' > > 2) replica 'AAA' is the current sync standby > > 3) replica 'BBB' is the current async standby (potential sync standby) > > 4) replica 'AAA' fail. after that, replica 'BBB' is current sync standby. > > 5) replica 'AAA' comes online > > 6) replica 'AAA' grabs sync status > >> > > >> If that's the case, I'm not really sure that's undesirable behavior. >> One could argue fairly persuasively that if you care about the >> precendence order of sync replicas, you shouldn't use '*'. And the rule >> of "if using *, the lowest-sorted replica name has sync" is actually a >> predictable, easy-to-understand rule. >> >> So if you want to make this a feature request, you'll need to come up >> with an argument as to why the current behavior is bad. Otherwise, >> you're just asking us to document it better (which is a good idea). > It is not depend on name of standby server. That is, The standby server, > which was connected to the master server during initial configration > replication, is top priority even if priority of two server are same. What is happening here is that incase of '*' as priority of both are same, system will choose whichever comes in list of registered standby's first (list is maintained in structure WalSndCtl). Each standby is registered with WalSndCtl when a new WALSender is started in function InitWalSenderSlot(). As 'AAA' has been registered first it becomes preferred sync standby even if priorities of both are same. When 'AAA' goes down, it marks that Slot entry as free (by setting pid=0 in function WalSndKill), now when 'AAA' comes back again, it gets that free Slot entry and again becomes preferred sync standby. Now if we want to fix as you are suggesting which I don't think is necessary, we might need to change WalSndKill and some other place so that whenever any standby goes down, it changes slots for already registered standby's. > User must remember that which standby server connected to master server at > first. > I think that this behavior confuse user. > so I think that we need to modify this behaviour or if '*' is used, priority > of server is not same (modifying manual is also good). Here user has done the settings (setting synchronous_standby_names = '*'), after which he will not have any control which standby will become sync standby, so ideally he should not complain. It might be case that for some users current behavior is good enough which means that with '*' whichever standby has become sync standby first, it will be the sync standby always if alive. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
On Sun, Aug 25, 2013 at 3:21 PM, Amit Kapila <amit.kapila16@gmail.com> wrote: > On Sat, Aug 24, 2013 at 2:46 PM, Sawada Masahiko <sawada.mshk@gmail.com> wrote: >> On Sat, Aug 24, 2013 at 3:14 AM, Josh Berkus <josh@agliodbs.com> wrote: >>> On 08/23/2013 12:42 AM, Sawada Masahiko wrote: >>>> in case (a), those priority is clear. So I think that re-taking over >>>> is correct behaviour. >>>> OHOT, in case (b), even if AAA and BBB are set same priority, AAA >>>> server steals SYNC replication. >>>> I think it is better that BBB server continue behaviour SYNC standby, >>>> and AAA should become potential server. >>> >>> So, you're saying that: >>> >>> 1) synchronous_standby_names = '*' >>> >>> 2) replica 'BBB' is the current sync standby >>> >>> 3) replica 'AAA' comes online >>> >>> 4) replica 'AAA' grabs sync status >>> >>> ? >> I'm sorry that you are confuse. >> It means that >> >> 1) synchronous_standby_names = '*' >> >> 2) replica 'AAA' is the current sync standby >> >> 3) replica 'BBB' is the current async standby (potential sync standby) >> >> 4) replica 'AAA' fail. after that, replica 'BBB' is current sync standby. >> >> 5) replica 'AAA' comes online >> >> 6) replica 'AAA' grabs sync status >> >>> >> >> >>> If that's the case, I'm not really sure that's undesirable behavior. >>> One could argue fairly persuasively that if you care about the >>> precendence order of sync replicas, you shouldn't use '*'. And the rule >>> of "if using *, the lowest-sorted replica name has sync" is actually a >>> predictable, easy-to-understand rule. >>> >>> So if you want to make this a feature request, you'll need to come up >>> with an argument as to why the current behavior is bad. Otherwise, >>> you're just asking us to document it better (which is a good idea). >> It is not depend on name of standby server. That is, The standby server, >> which was connected to the master server during initial configration >> replication, is top priority even if priority of two server are same. > > What is happening here is that incase of '*' as priority of both are > same, system will choose whichever > comes in list of registered standby's first (list is maintained in > structure WalSndCtl). > Each standby is registered with WalSndCtl when a new WALSender is > started in function InitWalSenderSlot(). > As 'AAA' has been registered first it becomes preferred sync standby > even if priorities of both are same. > When 'AAA' goes down, it marks that Slot entry as free (by setting > pid=0 in function WalSndKill), > now when 'AAA' comes back again, it gets that free Slot entry and > again becomes preferred sync standby. > > Now if we want to fix as you are suggesting which I don't think is > necessary, we might need to change WalSndKill and some other place so > that whenever any standby goes down, it changes slots for already > registered standby's. >> User must remember that which standby server connected to master server at >> first. >> I think that this behavior confuse user. >> so I think that we need to modify this behaviour or if '*' is used, priority >> of server is not same (modifying manual is also good). > > Here user has done the settings (setting synchronous_standby_names = > '*'), after which he will not have any control which standby will > become sync standby, so ideally he should not complain. > > It might be case that for some users current behavior is good enough > which means that with '*' whichever standby has become sync standby > first, it will be the sync standby always if alive. I'm thinking that it is not necessary to change WalSndKill. For example, we add the value (e.g., sync_standby) which have that which wal sender is active SYNC rep. And if sync_standby is already set and it is active, server doesn't looking for active standby. Only if sync_standby is not set and it is inactive, server looking for that which server is active SYNC rep. If so, we also prevent to find active SYNC rep whenever SyncRepReleaseWaiters() is called. In pg_stat_get_walsenders(), looking for active SYNC rep is already used. I think that it is also able to here. Thought? -- Regards, ------- Sawada Masahiko
Amit Kapila <amit.kapila16@gmail.com> wrote: > What is happening here is that incase of '*' as priority of both > are same, system will choose whichever comes in list of > registered standby's first (list is maintained in structure > WalSndCtl). Each standby is registered with WalSndCtl when a new > WALSender is started in function InitWalSenderSlot(). As 'AAA' > has been registered first it becomes preferred sync standby even > if priorities of both are same. When 'AAA' goes down, it marks > that Slot entry as free (by setting pid=0 in function > WalSndKill), now when 'AAA' comes back again, it gets that free > Slot entry and again becomes preferred sync standby. So, when a user says they don't care about which standby is in sync mode, the system can decide (based on internal implementation details not visible to the user) that any node which comes online could now become the sync node, and freeze commits on the source database until that newly added node catches up? Or does the existing sync node continue in that role until the new "preferred" node is caught up? -- Kevin Grittner EDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Wed, Aug 28, 2013 at 5:51 AM, Kevin Grittner <kgrittn@ymail.com> wrote: > Amit Kapila <amit.kapila16@gmail.com> wrote: > >> What is happening here is that incase of '*' as priority of both >> are same, system will choose whichever comes in list of >> registered standby's first (list is maintained in structure >> WalSndCtl). Each standby is registered with WalSndCtl when a new >> WALSender is started in function InitWalSenderSlot(). As 'AAA' >> has been registered first it becomes preferred sync standby even >> if priorities of both are same. When 'AAA' goes down, it marks >> that Slot entry as free (by setting pid=0 in function >> WalSndKill), now when 'AAA' comes back again, it gets that free >> Slot entry and again becomes preferred sync standby. > > So, when a user says they don't care about which standby is in sync > mode, the system can decide (based on internal implementation > details not visible to the user) that any node which comes online > could now become the sync node, and freeze commits on the source > database until that newly added node catches up? Or does the > existing sync node continue in that role until the new "preferred" > node is caught up? Existing sync node continue in that role until the new preferred node is streamed. For example, When 'AAA' comes back again and reached consistency point, 'AAA' wal sender state is 'catchup'. IWO 'AAA' doesn't has authority to operate the queue. 'BBB' server is preferred sync standby server while 'AAA' wal sender state is 'catchup'. So even if user execute query, master server doesn't freeze commits When 'AAA' walsender state become 'streaming', 'AAA' becomes preferred sync standby again. Regards, ------- Sawada Masahiko
On Tue, Aug 27, 2013 at 4:51 PM, Sawada Masahiko <sawada.mshk@gmail.com> wrote: > On Sun, Aug 25, 2013 at 3:21 PM, Amit Kapila <amit.kapila16@gmail.com> wrote: >> On Sat, Aug 24, 2013 at 2:46 PM, Sawada Masahiko <sawada.mshk@gmail.com> wrote: >>> On Sat, Aug 24, 2013 at 3:14 AM, Josh Berkus <josh@agliodbs.com> wrote: >>>> On 08/23/2013 12:42 AM, Sawada Masahiko wrote: >>>>> in case (a), those priority is clear. So I think that re-taking over >>>>> is correct behaviour. >>>>> OHOT, in case (b), even if AAA and BBB are set same priority, AAA >>>>> server steals SYNC replication. >>>>> I think it is better that BBB server continue behaviour SYNC standby, >>>>> and AAA should become potential server. >>>> >>>> So, you're saying that: >>>> >>>> 1) synchronous_standby_names = '*' >>>> >>>> 2) replica 'BBB' is the current sync standby >>>> >>>> 3) replica 'AAA' comes online >>>> >>>> 4) replica 'AAA' grabs sync status >>>> >>>> ? >>> I'm sorry that you are confuse. >>> It means that >>> >>> 1) synchronous_standby_names = '*' >>> >>> 2) replica 'AAA' is the current sync standby >>> >>> 3) replica 'BBB' is the current async standby (potential sync standby) >>> >>> 4) replica 'AAA' fail. after that, replica 'BBB' is current sync standby. >>> >>> 5) replica 'AAA' comes online >>> >>> 6) replica 'AAA' grabs sync status >>> >>>> >>> >>> >>>> If that's the case, I'm not really sure that's undesirable behavior. >>>> One could argue fairly persuasively that if you care about the >>>> precendence order of sync replicas, you shouldn't use '*'. And the rule >>>> of "if using *, the lowest-sorted replica name has sync" is actually a >>>> predictable, easy-to-understand rule. >>>> >>>> So if you want to make this a feature request, you'll need to come up >>>> with an argument as to why the current behavior is bad. Otherwise, >>>> you're just asking us to document it better (which is a good idea). >>> It is not depend on name of standby server. That is, The standby server, >>> which was connected to the master server during initial configration >>> replication, is top priority even if priority of two server are same. >> >> What is happening here is that incase of '*' as priority of both are >> same, system will choose whichever >> comes in list of registered standby's first (list is maintained in >> structure WalSndCtl). >> Each standby is registered with WalSndCtl when a new WALSender is >> started in function InitWalSenderSlot(). >> As 'AAA' has been registered first it becomes preferred sync standby >> even if priorities of both are same. >> When 'AAA' goes down, it marks that Slot entry as free (by setting >> pid=0 in function WalSndKill), >> now when 'AAA' comes back again, it gets that free Slot entry and >> again becomes preferred sync standby. >> >> Now if we want to fix as you are suggesting which I don't think is >> necessary, we might need to change WalSndKill and some other place so >> that whenever any standby goes down, it changes slots for already >> registered standby's. >>> User must remember that which standby server connected to master server at >>> first. >>> I think that this behavior confuse user. >>> so I think that we need to modify this behaviour or if '*' is used, priority >>> of server is not same (modifying manual is also good). >> >> Here user has done the settings (setting synchronous_standby_names = >> '*'), after which he will not have any control which standby will >> become sync standby, so ideally he should not complain. >> >> It might be case that for some users current behavior is good enough >> which means that with '*' whichever standby has become sync standby >> first, it will be the sync standby always if alive. > I'm thinking that it is not necessary to change WalSndKill. > For example, we add the value (e.g., sync_standby) which have that > which wal sender is active SYNC rep. > And if sync_standby is already set and it is active, server doesn't > looking for active standby. > Only if sync_standby is not set and it is inactive, server looking for > that which server is active SYNC rep. > If so, we also prevent to find active SYNC rep whenever > SyncRepReleaseWaiters() is called. For '*' case, it will be okay, but when the user has given proper names, in that case even if there is any active Sync Rep, it has to be changed based on priority. I think here how to provide a fix, so that behavior gets changed to what you describe is a second priority work, first is to show the value of use-case. Do you really know where people actually setup using '*' as configuration and if yes, are they annoyed with current behavior? I have thought about it, but could imagine a scenario where people will be using '*' in their production configurations, may be it will be useful in test labs. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
On Wed, Aug 28, 2013 at 10:59 PM, Amit Kapila <amit.kapila16@gmail.com> wrote: > On Tue, Aug 27, 2013 at 4:51 PM, Sawada Masahiko <sawada.mshk@gmail.com> wrote: >> On Sun, Aug 25, 2013 at 3:21 PM, Amit Kapila <amit.kapila16@gmail.com> wrote: >>> On Sat, Aug 24, 2013 at 2:46 PM, Sawada Masahiko <sawada.mshk@gmail.com> wrote: >>>> On Sat, Aug 24, 2013 at 3:14 AM, Josh Berkus <josh@agliodbs.com> wrote: >>>>> On 08/23/2013 12:42 AM, Sawada Masahiko wrote: >>>>>> in case (a), those priority is clear. So I think that re-taking over >>>>>> is correct behaviour. >>>>>> OHOT, in case (b), even if AAA and BBB are set same priority, AAA >>>>>> server steals SYNC replication. >>>>>> I think it is better that BBB server continue behaviour SYNC standby, >>>>>> and AAA should become potential server. >>>>> >>>>> So, you're saying that: >>>>> >>>>> 1) synchronous_standby_names = '*' >>>>> >>>>> 2) replica 'BBB' is the current sync standby >>>>> >>>>> 3) replica 'AAA' comes online >>>>> >>>>> 4) replica 'AAA' grabs sync status >>>>> >>>>> ? >>>> I'm sorry that you are confuse. >>>> It means that >>>> >>>> 1) synchronous_standby_names = '*' >>>> >>>> 2) replica 'AAA' is the current sync standby >>>> >>>> 3) replica 'BBB' is the current async standby (potential sync standby) >>>> >>>> 4) replica 'AAA' fail. after that, replica 'BBB' is current sync standby. >>>> >>>> 5) replica 'AAA' comes online >>>> >>>> 6) replica 'AAA' grabs sync status >>>> >>>>> >>>> >>>> >>>>> If that's the case, I'm not really sure that's undesirable behavior. >>>>> One could argue fairly persuasively that if you care about the >>>>> precendence order of sync replicas, you shouldn't use '*'. And the rule >>>>> of "if using *, the lowest-sorted replica name has sync" is actually a >>>>> predictable, easy-to-understand rule. >>>>> >>>>> So if you want to make this a feature request, you'll need to come up >>>>> with an argument as to why the current behavior is bad. Otherwise, >>>>> you're just asking us to document it better (which is a good idea). >>>> It is not depend on name of standby server. That is, The standby server, >>>> which was connected to the master server during initial configration >>>> replication, is top priority even if priority of two server are same. >>> >>> What is happening here is that incase of '*' as priority of both are >>> same, system will choose whichever >>> comes in list of registered standby's first (list is maintained in >>> structure WalSndCtl). >>> Each standby is registered with WalSndCtl when a new WALSender is >>> started in function InitWalSenderSlot(). >>> As 'AAA' has been registered first it becomes preferred sync standby >>> even if priorities of both are same. >>> When 'AAA' goes down, it marks that Slot entry as free (by setting >>> pid=0 in function WalSndKill), >>> now when 'AAA' comes back again, it gets that free Slot entry and >>> again becomes preferred sync standby. >>> >>> Now if we want to fix as you are suggesting which I don't think is >>> necessary, we might need to change WalSndKill and some other place so >>> that whenever any standby goes down, it changes slots for already >>> registered standby's. >>>> User must remember that which standby server connected to master server at >>>> first. >>>> I think that this behavior confuse user. >>>> so I think that we need to modify this behaviour or if '*' is used, priority >>>> of server is not same (modifying manual is also good). >>> >>> Here user has done the settings (setting synchronous_standby_names = >>> '*'), after which he will not have any control which standby will >>> become sync standby, so ideally he should not complain. >>> >>> It might be case that for some users current behavior is good enough >>> which means that with '*' whichever standby has become sync standby >>> first, it will be the sync standby always if alive. > >> I'm thinking that it is not necessary to change WalSndKill. >> For example, we add the value (e.g., sync_standby) which have that >> which wal sender is active SYNC rep. >> And if sync_standby is already set and it is active, server doesn't >> looking for active standby. >> Only if sync_standby is not set and it is inactive, server looking for >> that which server is active SYNC rep. >> If so, we also prevent to find active SYNC rep whenever >> SyncRepReleaseWaiters() is called. > For '*' case, it will be okay, but when the user has given proper > names, in that case even if there is any active Sync > Rep, it has to be changed based on priority. > > I think here how to provide a fix, so that behavior gets changed to > what you describe is a second priority work, first > is to show the value of use-case. Do you really know where people > actually setup using '*' as configuration and if > yes, are they annoyed with current behavior? > I have thought about it, but could imagine a scenario where people > will be using '*' in their production > configurations, may be it will be useful in test labs. Thank you for your feedback. I have implemented the patch which change how to put priority on each walsender, based on I suggested. I added sync_standby value into WalCtl value. This value has that which walsender is active sync rep. This patch handle also for case that user has given proper names. Regards, ------- Sawada Masahiko