Thread: Need inputs on postgresql HA with consul cluster
Hi Team,
We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup.
We are seeing the issue below.
our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover.
Is there any option to prevent this failure even if the consul cluster is down also?
and also any effective alternative for HA for docker based PostgreSQL setup?
> On Dec 19, 2023, at 7:12 AM, Rambabu V <ram.wissen@gmail.com> wrote: > > Hi Team, > > We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup. > > We are seeing the issue below. > > our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover. > Is there any option to prevent this failure even if the consul cluster is down also? > > and also any effective alternative for HA for docker based PostgreSQL setup? How do you expect to have working HA without working networking?
Thanks, any alternative solution for my requirement .
In above case if consul down , db master node going down, replica nodes remains recovery mode.its complete downtime.
On Tue, Dec 19, 2023, 22:23 Scott Ribe <scott_ribe@elevated-dev.com> wrote:
> On Dec 19, 2023, at 7:12 AM, Rambabu V <ram.wissen@gmail.com> wrote:
>
> Hi Team,
>
> We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup.
>
> We are seeing the issue below.
>
> our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover.
> Is there any option to prevent this failure even if the consul cluster is down also?
>
> and also any effective alternative for HA for docker based PostgreSQL setup?
How do you expect to have working HA without working networking?
Have you asked the consul developers?
On Tue, Dec 19, 2023 at 12:33 PM Rambabu V <ram.wissen@gmail.com> wrote:
Thanks, any alternative solution for my requirement .In above case if consul down , db master node going down, replica nodes remains recovery mode.its complete downtime.On Tue, Dec 19, 2023, 22:23 Scott Ribe <scott_ribe@elevated-dev.com> wrote:> On Dec 19, 2023, at 7:12 AM, Rambabu V <ram.wissen@gmail.com> wrote:
>
> Hi Team,
>
> We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup.
>
> We are seeing the issue below.
>
> our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover.
> Is there any option to prevent this failure even if the consul cluster is down also?
>
> and also any effective alternative for HA for docker based PostgreSQL setup?
How do you expect to have working HA without working networking?
May be you can pg auto failover..
On Tue, Dec 19, 2023 at 9:33 AM Rambabu V <ram.wissen@gmail.com> wrote:
Thanks, any alternative solution for my requirement .In above case if consul down , db master node going down, replica nodes remains recovery mode.its complete downtime.On Tue, Dec 19, 2023, 22:23 Scott Ribe <scott_ribe@elevated-dev.com> wrote:> On Dec 19, 2023, at 7:12 AM, Rambabu V <ram.wissen@gmail.com> wrote:
>
> Hi Team,
>
> We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup.
>
> We are seeing the issue below.
>
> our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover.
> Is there any option to prevent this failure even if the consul cluster is down also?
>
> and also any effective alternative for HA for docker based PostgreSQL setup?
How do you expect to have working HA without working networking?
Thanks & Regards
Karthik Yellapragada
+1 860 830 5235
Karthik Yellapragada
+1 860 830 5235
Hi Karthik,
This is docker based Postgtresql, behavior is different than our normal one for failover and promotion.
On Wed, Dec 20, 2023, 03:05 Karthik Yellapragada <karthik.yellapragada@gmail.com> wrote:
May be you can pg auto failover..--On Tue, Dec 19, 2023 at 9:33 AM Rambabu V <ram.wissen@gmail.com> wrote:Thanks, any alternative solution for my requirement .In above case if consul down , db master node going down, replica nodes remains recovery mode.its complete downtime.On Tue, Dec 19, 2023, 22:23 Scott Ribe <scott_ribe@elevated-dev.com> wrote:> On Dec 19, 2023, at 7:12 AM, Rambabu V <ram.wissen@gmail.com> wrote:
>
> Hi Team,
>
> We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup.
>
> We are seeing the issue below.
>
> our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover.
> Is there any option to prevent this failure even if the consul cluster is down also?
>
> and also any effective alternative for HA for docker based PostgreSQL setup?
How do you expect to have working HA without working networking?Thanks & Regards
Karthik Yellapragada
+1 860 830 5235
Consul is DCS for patroni. If consul cluster is completely down, then patroni is unaware of state of cluster.
And "In above case if consul down , db master node going down, replica nodes remains recovery mode"
is expected. Actually this behave protects database from split brain
And "In above case if consul down , db master node going down, replica nodes remains recovery mode"
is expected. Actually this behave protects database from split brain
br
Kaido
On Wed, 20 Dec 2023 at 03:52, Rambabu V <ram.wissen@gmail.com> wrote:
Hi Karthik,This is docker based Postgtresql, behavior is different than our normal one for failover and promotion.On Wed, Dec 20, 2023, 03:05 Karthik Yellapragada <karthik.yellapragada@gmail.com> wrote:May be you can pg auto failover..--On Tue, Dec 19, 2023 at 9:33 AM Rambabu V <ram.wissen@gmail.com> wrote:Thanks, any alternative solution for my requirement .In above case if consul down , db master node going down, replica nodes remains recovery mode.its complete downtime.On Tue, Dec 19, 2023, 22:23 Scott Ribe <scott_ribe@elevated-dev.com> wrote:> On Dec 19, 2023, at 7:12 AM, Rambabu V <ram.wissen@gmail.com> wrote:
>
> Hi Team,
>
> We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup.
>
> We are seeing the issue below.
>
> our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover.
> Is there any option to prevent this failure even if the consul cluster is down also?
>
> and also any effective alternative for HA for docker based PostgreSQL setup?
How do you expect to have working HA without working networking?Thanks & Regards
Karthik Yellapragada
+1 860 830 5235
@Rambabu
You can try this: DCS Failsafe Mode
https://patroni.readthedocs.io/en/master/dcs_failsafe_mode.html#id2
https://speakerdeck.com/cyberdemn/failsafe-patroni-3-dot-0
https://patroni.readthedocs.io/en/master/dcs_failsafe_mode.html#id2
https://speakerdeck.com/cyberdemn/failsafe-patroni-3-dot-0
br
Kaido
Kaido
On Wed, 20 Dec 2023 at 19:42, kaido vaikla <kaido.vaikla@gmail.com> wrote:
Consul is DCS for patroni. If consul cluster is completely down, then patroni is unaware of state of cluster.
And "In above case if consul down , db master node going down, replica nodes remains recovery mode"
is expected. Actually this behave protects database from split brain
br
KaidoOn Wed, 20 Dec 2023 at 03:52, Rambabu V <ram.wissen@gmail.com> wrote:Hi Karthik,This is docker based Postgtresql, behavior is different than our normal one for failover and promotion.On Wed, Dec 20, 2023, 03:05 Karthik Yellapragada <karthik.yellapragada@gmail.com> wrote:May be you can pg auto failover..--On Tue, Dec 19, 2023 at 9:33 AM Rambabu V <ram.wissen@gmail.com> wrote:Thanks, any alternative solution for my requirement .In above case if consul down , db master node going down, replica nodes remains recovery mode.its complete downtime.On Tue, Dec 19, 2023, 22:23 Scott Ribe <scott_ribe@elevated-dev.com> wrote:> On Dec 19, 2023, at 7:12 AM, Rambabu V <ram.wissen@gmail.com> wrote:
>
> Hi Team,
>
> We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup.
>
> We are seeing the issue below.
>
> our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover.
> Is there any option to prevent this failure even if the consul cluster is down also?
>
> and also any effective alternative for HA for docker based PostgreSQL setup?
How do you expect to have working HA without working networking?Thanks & Regards
Karthik Yellapragada
+1 860 830 5235
thanks Kaido, applied to the above option. getting good result.
On Thu, Dec 21, 2023 at 2:30 AM kaido vaikla <kaido.vaikla@gmail.com> wrote:
@RambabuYou can try this: DCS Failsafe Mode
https://patroni.readthedocs.io/en/master/dcs_failsafe_mode.html#id2
https://speakerdeck.com/cyberdemn/failsafe-patroni-3-dot-0br
Kaido
On Wed, 20 Dec 2023 at 19:42, kaido vaikla <kaido.vaikla@gmail.com> wrote:Consul is DCS for patroni. If consul cluster is completely down, then patroni is unaware of state of cluster.
And "In above case if consul down , db master node going down, replica nodes remains recovery mode"
is expected. Actually this behave protects database from split brain
br
KaidoOn Wed, 20 Dec 2023 at 03:52, Rambabu V <ram.wissen@gmail.com> wrote:Hi Karthik,This is docker based Postgtresql, behavior is different than our normal one for failover and promotion.On Wed, Dec 20, 2023, 03:05 Karthik Yellapragada <karthik.yellapragada@gmail.com> wrote:May be you can pg auto failover..--On Tue, Dec 19, 2023 at 9:33 AM Rambabu V <ram.wissen@gmail.com> wrote:Thanks, any alternative solution for my requirement .In above case if consul down , db master node going down, replica nodes remains recovery mode.its complete downtime.On Tue, Dec 19, 2023, 22:23 Scott Ribe <scott_ribe@elevated-dev.com> wrote:> On Dec 19, 2023, at 7:12 AM, Rambabu V <ram.wissen@gmail.com> wrote:
>
> Hi Team,
>
> We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup.
>
> We are seeing the issue below.
>
> our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover.
> Is there any option to prevent this failure even if the consul cluster is down also?
>
> and also any effective alternative for HA for docker based PostgreSQL setup?
How do you expect to have working HA without working networking?Thanks & Regards
Karthik Yellapragada
+1 860 830 5235
Hi Kaido,
Using patroni 3+ version, even though enabled failsafe_mide, still if consul cluster down, master db going down. Replicas remaining same.
On Thu, Dec 21, 2023, 09:49 Rambabu V <ram.wissen@gmail.com> wrote:
thanks Kaido, applied to the above option. getting good result.On Thu, Dec 21, 2023 at 2:30 AM kaido vaikla <kaido.vaikla@gmail.com> wrote:@RambabuYou can try this: DCS Failsafe Mode
https://patroni.readthedocs.io/en/master/dcs_failsafe_mode.html#id2
https://speakerdeck.com/cyberdemn/failsafe-patroni-3-dot-0br
Kaido
On Wed, 20 Dec 2023 at 19:42, kaido vaikla <kaido.vaikla@gmail.com> wrote:Consul is DCS for patroni. If consul cluster is completely down, then patroni is unaware of state of cluster.
And "In above case if consul down , db master node going down, replica nodes remains recovery mode"
is expected. Actually this behave protects database from split brain
br
KaidoOn Wed, 20 Dec 2023 at 03:52, Rambabu V <ram.wissen@gmail.com> wrote:Hi Karthik,This is docker based Postgtresql, behavior is different than our normal one for failover and promotion.On Wed, Dec 20, 2023, 03:05 Karthik Yellapragada <karthik.yellapragada@gmail.com> wrote:May be you can pg auto failover..--On Tue, Dec 19, 2023 at 9:33 AM Rambabu V <ram.wissen@gmail.com> wrote:Thanks, any alternative solution for my requirement .In above case if consul down , db master node going down, replica nodes remains recovery mode.its complete downtime.On Tue, Dec 19, 2023, 22:23 Scott Ribe <scott_ribe@elevated-dev.com> wrote:> On Dec 19, 2023, at 7:12 AM, Rambabu V <ram.wissen@gmail.com> wrote:
>
> Hi Team,
>
> We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup.
>
> We are seeing the issue below.
>
> our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover.
> Is there any option to prevent this failure even if the consul cluster is down also?
>
> and also any effective alternative for HA for docker based PostgreSQL setup?
How do you expect to have working HA without working networking?Thanks & Regards
Karthik Yellapragada
+1 860 830 5235
On Sat, Dec 23, 2023, 14:11 Rambabu V <ram.wissen@gmail.com> wrote:
Hi Kaido,Using patroni 3+ version, even though enabled failsafe_mide, still if consul cluster down, master db going down. Replicas remaining same.On Thu, Dec 21, 2023, 09:49 Rambabu V <ram.wissen@gmail.com> wrote:thanks Kaido, applied to the above option. getting good result.On Thu, Dec 21, 2023 at 2:30 AM kaido vaikla <kaido.vaikla@gmail.com> wrote:@RambabuYou can try this: DCS Failsafe Mode
https://patroni.readthedocs.io/en/master/dcs_failsafe_mode.html#id2
https://speakerdeck.com/cyberdemn/failsafe-patroni-3-dot-0br
Kaido
On Wed, 20 Dec 2023 at 19:42, kaido vaikla <kaido.vaikla@gmail.com> wrote:Consul is DCS for patroni. If consul cluster is completely down, then patroni is unaware of state of cluster.
And "In above case if consul down , db master node going down, replica nodes remains recovery mode"
is expected. Actually this behave protects database from split brain
br
KaidoOn Wed, 20 Dec 2023 at 03:52, Rambabu V <ram.wissen@gmail.com> wrote:Hi Karthik,This is docker based Postgtresql, behavior is different than our normal one for failover and promotion.On Wed, Dec 20, 2023, 03:05 Karthik Yellapragada <karthik.yellapragada@gmail.com> wrote:May be you can pg auto failover..--On Tue, Dec 19, 2023 at 9:33 AM Rambabu V <ram.wissen@gmail.com> wrote:Thanks, any alternative solution for my requirement .In above case if consul down , db master node going down, replica nodes remains recovery mode.its complete downtime.On Tue, Dec 19, 2023, 22:23 Scott Ribe <scott_ribe@elevated-dev.com> wrote:> On Dec 19, 2023, at 7:12 AM, Rambabu V <ram.wissen@gmail.com> wrote:
>
> Hi Team,
>
> We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup.
>
> We are seeing the issue below.
>
> our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover.
> Is there any option to prevent this failure even if the consul cluster is down also?
>
> and also any effective alternative for HA for docker based PostgreSQL setup?
How do you expect to have working HA without working networking?Thanks & Regards
Karthik Yellapragada
+1 860 830 5235
Attachment for the same.
On Sat, Dec 23, 2023, 14:11 Rambabu V <ram.wissen@gmail.com> wrote:
Hi Kaido,Using patroni 3+ version, even though enabled failsafe_mide, still if consul cluster down, master db going down. Replicas remaining same.On Thu, Dec 21, 2023, 09:49 Rambabu V <ram.wissen@gmail.com> wrote:thanks Kaido, applied to the above option. getting good result.On Thu, Dec 21, 2023 at 2:30 AM kaido vaikla <kaido.vaikla@gmail.com> wrote:@RambabuYou can try this: DCS Failsafe Mode
https://patroni.readthedocs.io/en/master/dcs_failsafe_mode.html#id2
https://speakerdeck.com/cyberdemn/failsafe-patroni-3-dot-0br
Kaido
On Wed, 20 Dec 2023 at 19:42, kaido vaikla <kaido.vaikla@gmail.com> wrote:Consul is DCS for patroni. If consul cluster is completely down, then patroni is unaware of state of cluster.
And "In above case if consul down , db master node going down, replica nodes remains recovery mode"
is expected. Actually this behave protects database from split brain
br
KaidoOn Wed, 20 Dec 2023 at 03:52, Rambabu V <ram.wissen@gmail.com> wrote:Hi Karthik,This is docker based Postgtresql, behavior is different than our normal one for failover and promotion.On Wed, Dec 20, 2023, 03:05 Karthik Yellapragada <karthik.yellapragada@gmail.com> wrote:May be you can pg auto failover..--On Tue, Dec 19, 2023 at 9:33 AM Rambabu V <ram.wissen@gmail.com> wrote:Thanks, any alternative solution for my requirement .In above case if consul down , db master node going down, replica nodes remains recovery mode.its complete downtime.On Tue, Dec 19, 2023, 22:23 Scott Ribe <scott_ribe@elevated-dev.com> wrote:> On Dec 19, 2023, at 7:12 AM, Rambabu V <ram.wissen@gmail.com> wrote:
>
> Hi Team,
>
> We have implemented postgresql HA with Patroni + Consul cluster on Docker based PostgreSQL Setup.
>
> We are seeing the issue below.
>
> our consul cluster built with 3 nodes, if consul cluster completely down, Patroni triggering failover.
> Is there any option to prevent this failure even if the consul cluster is down also?
>
> and also any effective alternative for HA for docker based PostgreSQL setup?
How do you expect to have working HA without working networking?Thanks & Regards
Karthik Yellapragada
+1 860 830 5235
Attachment
> On Dec 22, 2023, at 11:11 PM, Rambabu V <ram.wissen@gmail.com> wrote: > > Using patroni 3+ version, even though enabled failsafe_mide, still if consul cluster down, master db going down. Replicasremaining same. If you don't have working network infrastructure, you cannot have reliable HA on Postgres--nor any other service.
Hi Scott,
but as per the patroni documentation , if we enable failsafe_mode option, incase of dcs down, cluster status should remain the same without any action. but it is not happening.
On Sat, Dec 23, 2023 at 9:29 PM Scott Ribe <scott_ribe@elevated-dev.com> wrote:
> On Dec 22, 2023, at 11:11 PM, Rambabu V <ram.wissen@gmail.com> wrote:
>
> Using patroni 3+ version, even though enabled failsafe_mide, still if consul cluster down, master db going down. Replicas remaining same.
If you don't have working network infrastructure, you cannot have reliable HA on Postgres--nor any other service.
> On Dec 23, 2023, at 6:33 AM, Rambabu V <ram.wissen@gmail.com> wrote: > > but as per the patroni documentation , if we enable failsafe_mode option, incase of dcs down, cluster status should remainthe same without any action. but it is not happening. "...if it can access all known members of the cluster via Patroni REST API." Is that your problem? Really, you're pissing away time poking at a symptom, when the underlying cause needs to be fixed.
> On Dec 22, 2023, at 11:13 PM, Rambabu V <ram.wissen@gmail.com> wrote: > > Attachment for the same. So? You posted that before. As I quoted from the Patroni documentation about failsafe_mode, which you surely read, it onlyworks if the Patroni instances can still communicate with each other over their REST APIs. I am going to say this once more, then I will no longer participate in this thread: IF YOUR NETWORK (AND DCS) ARE NOT WORKING,YOU CANNOT HAVE RELIABLE HA.