Thread: Feature Request: pg_replication_master()
Hackers, Currently we can see each master's current replicas using pg_stat_replication. However, there is no way from a replica, that I know of, to figure out who its master is other than to look at recovery.conf. We should probably have a function, like pg_replication_master(), which gives the host address of the current master. This would help DBAs for large replication clusters a lot. Obviously, this would only work in streaming. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
<p dir="ltr"><br /> On Dec 19, 2012 4:43 AM, "Josh Berkus" <<a href="mailto:josh@agliodbs.com">josh@agliodbs.com</a>>wrote:<br /> ><br /> > Hackers,<br /> ><br /> > Currentlywe can see each master's current replicas using<br /> > pg_stat_replication. However, there is no way from areplica, that I<br /> > know of, to figure out who its master is other than to look at<br /> > recovery.conf.<br />><br /> > We should probably have a function, like pg_replication_master(), which<br /> > gives the host addressof the current master. This would help DBAs for<br /> > large replication clusters a lot. Obviously, this wouldonly work in<br /> > streaming.<br /><p dir="ltr">This sounds like my previous suggestion of returning the primaryconninfo value, but with just ip. That one came with a pretty bad patch, and was later postponed until we folded recovery.confinto the main configuration file parsing. I'm not really sure what happened to that project? (the configurationfile one) <p dir="ltr">/Magnus
On 19 December 2012 06:10, Magnus Hagander <magnus@hagander.net> wrote: > This sounds like my previous suggestion of returning the primary conninfo > value, but with just ip. That one came with a pretty bad patch, and was > later postponed until we folded recovery.conf into the main configuration > file parsing. I'm not really sure what happened to that project? (the > configuration file one) It stalled because the patch author decided not to implement the request to detect recovery.conf in data directory, which allows backwards compatibility. I proposed a solution to how to do that, so we can move forwards if people have time. -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
> It stalled because the patch author decided not to implement the > request to detect recovery.conf in data directory, which allows > backwards compatibility. Well, I don't think we had agreement on how important backwards compatibility for recovery.conf was, particularly not onthe whole recovery.conf/recovery.done functionality and the wierd formatting of recovery.conf. However, with "include_if_exists" directives in postgresql.conf, or "include_dir", that would be easy to work around. Don'twe have something like that planned for SET PERSISTENT? --Josh Berkus
> This sounds like my previous suggestion of returning the primary > conninfo value, but with just ip. That one came with a pretty bad > patch, and was later postponed until we folded recovery.conf into > the main configuration file parsing. I'm not really sure what > happened to that project? (the configuration file one) Hmmm, good point. Just having primary_conninfo it in pg_settings would help a lot. --Josh
On 19 December 2012 22:19, Joshua Berkus <josh@agliodbs.com> wrote: > >> It stalled because the patch author decided not to implement the >> request to detect recovery.conf in data directory, which allows >> backwards compatibility. > > Well, I don't think we had agreement on how important backwards compatibility for recovery.conf was, particularly not onthe whole recovery.conf/recovery.done functionality and the wierd formatting of recovery.conf. As ever, we spent much energy on debating backwards compatibility rather than just solving the problem it posed, which is fairly easy to solve. -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Wed, Dec 19, 2012 at 5:34 PM, Simon Riggs <simon@2ndquadrant.com> wrote: > As ever, we spent much energy on debating backwards compatibility > rather than just solving the problem it posed, which is fairly easy to > solve. I'm still of the opinion (as were a lot of people on the previous thread, IIRC) that just making them GUCs and throwing backward compatibility under the bus is acceptable in this case. Changes that break application code are anathema to me, because people can have a LOT of application code and updating it can be REALLY hard. The same cannot be said about recovery.conf - you have at most one of those per standby, and if it needs to be changed in some way, you can do it with a very small Perl script. Yes, third-party tools will need to be updated; that is surely a downside, but I think it might be a tolerable one in this case. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Thursday, December 20, 2012 3:50 AM Joshua Berkus wrote: > > > It stalled because the patch author decided not to implement the > > request to detect recovery.conf in data directory, which allows > > backwards compatibility. > > Well, I don't think we had agreement on how important backwards > compatibility for recovery.conf was, particularly not on the whole > recovery.conf/recovery.done functionality and the wierd formatting of > recovery.conf. > > However, with "include_if_exists" directives in postgresql.conf, or > "include_dir", that would be easy to work around. Don't we have > something like that planned for SET PERSISTENT? Yes in SET PERSISTENT patch, it uses "include_dir". I wonder why can't we get this information from WALRcvData structure? It has the required information. With Regards, Amit Kapila.
On Wed, Dec 19, 2012 at 07:32:33PM -0500, Robert Haas wrote: > On Wed, Dec 19, 2012 at 5:34 PM, Simon Riggs <simon@2ndquadrant.com> wrote: > > As ever, we spent much energy on debating backwards compatibility > > rather than just solving the problem it posed, which is fairly easy to > > solve. > > I'm still of the opinion (as were a lot of people on the previous > thread, IIRC) that just making them GUCs and throwing backward > compatibility under the bus is acceptable in this case. Changes that > break application code are anathema to me, because people can have a > LOT of application code and updating it can be REALLY hard. The same > cannot be said about recovery.conf - you have at most one of those per > standby, and if it needs to be changed in some way, you can do it with > a very small Perl script. Yes, third-party tools will need to be > updated; that is surely a downside, but I think it might be a > tolerable one in this case. Agreed. We have always has a more lax requirement of changing the Postgres administration interface. I am not saying to ignore backward compatibility, but future admin interface clarity overrules backward compatibility in most cases. If people are really worried about this, they can write a Perl script to convert from the old to new format. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On Wed, Dec 19, 2012 at 10:34:14PM +0000, Simon Riggs wrote: > On 19 December 2012 22:19, Joshua Berkus <josh@agliodbs.com> wrote: > > > >> It stalled because the patch author decided not to implement the > >> request to detect recovery.conf in data directory, which allows > >> backwards compatibility. > > > > Well, I don't think we had agreement on how important backwards compatibility for recovery.conf was, particularly noton the whole recovery.conf/recovery.done functionality and the wierd formatting of recovery.conf. > > As ever, we spent much energy on debating backwards compatibility > rather than just solving the problem it posed, which is fairly easy to > solve. Let me also add that I am tired of having recovery.conf improvement stalled by backward compatibility concerns. At this point, let's just trash recovery.conf backward compatibility and move on. And I don't want to hear complaints about tool breakage either. These are external tools, not shipped with community Postgres, and they will just have to adjust. I will be glad to beat all complainants into the ground for the good of the community. ;-) We just can't operate like this, and if we allowed these things to block us in the past, Postgres would be a royal mess today! At this point backward compatibility has paralized us from fixing a recovery.conf API that everyone agrees is non-optimal, and this has gone on for multiple major releases. I don't care what we have to do, just clean this up for 9.3! -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On Thu, Dec 20, 2012 at 02:29:49PM -0500, Bruce Momjian wrote: > Let me also add that I am tired of having recovery.conf improvement > stalled by backward compatibility concerns. At this point, let's just > trash recovery.conf backward compatibility and move on. > > And I don't want to hear complaints about tool breakage either. These > are external tools, not shipped with community Postgres, and they will > just have to adjust. I will be glad to beat all complainants into the > ground for the good of the community. ;-) We just can't operate like > this, and if we allowed these things to block us in the past, Postgres > would be a royal mess today! > > At this point backward compatibility has paralized us from fixing a > recovery.conf API that everyone agrees is non-optimal, and this has > gone on for multiple major releases. I don't care what we have to do, > just clean this up for 9.3! Let me add two more things. First, no matter how many people you interact with who use Postgres, there are many more who you do not interact with. Please don't give excessive weight to those people you know, or who use your tools, or who are your customers, and forget the many more Postgres users who you will never know. They are not using your tools or scripts --- they just want Postgres to be simple to use. Second, no matter how successful Postgres is now, there are many more users in our future, and we have a responsibility to give them a database that is as easy to configure as possible, without hampering them with decisions to avoid disruption for current Postgres users. I am not saying we should ignore current users, or our customers or our tool users, but it is very clear to me that we have lost the proper balance in this area. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
> Let me also add that I am tired of having recovery.conf improvement > stalled by backward compatibility concerns. At this point, let's just > trash recovery.conf backward compatibility and move on. > > And I don't want to hear complaints about tool breakage either. These are > external tools, not shipped with community Postgres, and they will just have > to adjust. I will be glad to beat all complainants into the > ground for the good of the community. ;-) We just can't operate like > this, and if we allowed these things to block us in the past, Postgres would be > a royal mess today! > > At this point backward compatibility has paralized us from fixing a > recovery.conf API that everyone agrees is non-optimal, and this has gone on > for multiple major releases. I don't care what we have to do, just clean this > up for 9.3! > +1 And the sooner we do it before release, the more time will those external tools have to adjust. Regards Petr Jelinek
> As ever, we spent much energy on debating backwards compatibility > rather than just solving the problem it posed, which is fairly easy > to > solve. Well, IIRC, the debate was primarily of *your* making. Almost everyone else on the thread was fine with the original patch,and it was nearly done for 9.2 before you stepped in. I can't find anyone else on that thread who thought that backwardscompatibility was more important than fixing the API. --Josh
On 2012-12-18 19:43:09 -0800, Josh Berkus wrote: > We should probably have a function, like pg_replication_master(), which > gives the host address of the current master. This would help DBAs for > large replication clusters a lot. Obviously, this would only work in > streaming. Do you want the node one step up or the top-level in the chain? Because I don't think we can do the latter without complicating the replication protocol noticably. Greetings, Andres Freund --Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Andreas, > Do you want the node one step up or the top-level in the chain? > Because > I don't think we can do the latter without complicating the > replication > protocol noticably. Well, clearly a whole chain would be nice for the user. But even just one step up would be very useful. --Josh
On Thu, Dec 20, 2012 at 5:07 PM, Joshua Berkus <josh@agliodbs.com> wrote: >> As ever, we spent much energy on debating backwards compatibility >> rather than just solving the problem it posed, which is fairly easy >> to >> solve. > > Well, IIRC, the debate was primarily of *your* making. Almost everyone else on the thread was fine with the original patch,and it was nearly done for 9.2 before you stepped in. I can't find anyone else on that thread who thought that backwardscompatibility was more important than fixing the API. +1. Let's JFDI. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 20 December 2012 19:29, Bruce Momjian <bruce@momjian.us> wrote: > On Wed, Dec 19, 2012 at 10:34:14PM +0000, Simon Riggs wrote: >> On 19 December 2012 22:19, Joshua Berkus <josh@agliodbs.com> wrote: >> > >> >> It stalled because the patch author decided not to implement the >> >> request to detect recovery.conf in data directory, which allows >> >> backwards compatibility. >> > >> > Well, I don't think we had agreement on how important backwards compatibility for recovery.conf was, particularly noton the whole recovery.conf/recovery.done functionality and the wierd formatting of recovery.conf. >> >> As ever, we spent much energy on debating backwards compatibility >> rather than just solving the problem it posed, which is fairly easy to >> solve. > > Let me also add that I am tired of having recovery.conf improvement > stalled by backward compatibility concerns. At this point, let's just > trash recovery.conf backward compatibility and move on. No, lets not. The only stall happening is because of a refusal to listen to another person's reasonable request during patch review. That requirement is not a blocker to the idea, it just needs to be programmed. Lets just implement the reasonable request for backwards compatibility, rather than wasting time on reopening the debate. -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On 20 December 2012 19:29, Bruce Momjian <bruce@momjian.us> wrote: > At this point backward compatibility has paralized us from fixing a > recovery.conf API that everyone agrees is non-optimal, and this has > gone on for multiple major releases. I don't care what we have to do, > just clean this up for 9.3! The main stall at this point is that the developer who wrote that patch no longer spends much time working on Postgres. AFAICS there is nobody working on this for 9.3 mainly because its not a priority, nor will implementing that fix the OP's request. There is no paralysis because there never was a blocker, only a request for backwards compatibility, which is easily possible to implement. -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Fri, Dec 21, 2012 at 9:21 AM, Simon Riggs <simon@2ndquadrant.com> wrote: > No, lets not. > > The only stall happening is because of a refusal to listen to another > person's reasonable request during patch review. That requirement is > not a blocker to the idea, it just needs to be programmed. > > Lets just implement the reasonable request for backwards > compatibility, rather than wasting time on reopening the debate. I read this as "let's do it the way I proposed, instead of the way other people proposed". I don't see how that suggestion advances the debate. If I recall correctly, and I might not, because it's been a year, you wanted to implicitly include recovery.conf in postgresql.conf only when the system is recovery mode, but that gave rise to a bunch of thorny definitional issues that were never adequately solved. I would have been willing to tolerate that solution if they had been, but they were not. It is not accurate to suggest that you presented a reasonable proposal and other people refused to listen. That is not what happened. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 21 December 2012 17:12, Robert Haas <robertmhaas@gmail.com> wrote: > On Fri, Dec 21, 2012 at 9:21 AM, Simon Riggs <simon@2ndquadrant.com> wrote: >> No, lets not. >> >> The only stall happening is because of a refusal to listen to another >> person's reasonable request during patch review. That requirement is >> not a blocker to the idea, it just needs to be programmed. >> >> Lets just implement the reasonable request for backwards >> compatibility, rather than wasting time on reopening the debate. > > I read this as "let's do it the way I proposed, instead of the way > other people proposed". I don't see how that suggestion advances the > debate. If I recall correctly, and I might not, because it's been a > year, you wanted to implicitly include recovery.conf in > postgresql.conf only when the system is recovery mode, but that gave > rise to a bunch of thorny definitional issues that were never > adequately solved. I would have been willing to tolerate that > solution if they had been, but they were not. It is not accurate to > suggest that you presented a reasonable proposal and other people > refused to listen. That is not what happened. Having looked into how to solve it, I think its solvable easily enough. If the energy expended on that thread, and now this one was directed to actually solve the problem, it would be solved. Characterising the problem as containing a bunch of thorny definitional issues is just a way to further avoid that, unintentionally or not. "Whether you think you can or think you can't, either way you are right" - Henry Ford -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On 21 December 2012 14:09, Robert Haas <robertmhaas@gmail.com> wrote: > On Thu, Dec 20, 2012 at 5:07 PM, Joshua Berkus <josh@agliodbs.com> wrote: >>> As ever, we spent much energy on debating backwards compatibility >>> rather than just solving the problem it posed, which is fairly easy >>> to >>> solve. >> >> Well, IIRC, the debate was primarily of *your* making. Almost everyone else on the thread was fine with the originalpatch, and it was nearly done for 9.2 before you stepped in. I can't find anyone else on that thread who thoughtthat backwards compatibility was more important than fixing the API. Then you're forgetting the thread. If you reread it you will find I was not alone. > +1. Let's JFDI. Agreed, lets get on with solving the problem rather than continuing a debate that could have been avoided, and can stop any time people choose to let it. -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Fri, Dec 21, 2012 at 02:25:47PM +0000, Simon Riggs wrote: > On 20 December 2012 19:29, Bruce Momjian <bruce@momjian.us> wrote: > > > At this point backward compatibility has paralyzed us from fixing a > > recovery.conf API that everyone agrees is non-optimal, and this has > > gone on for multiple major releases. I don't care what we have to do, > > just clean this up for 9.3! > > The main stall at this point is that the developer who wrote that > patch no longer spends much time working on Postgres. AFAICS there is > nobody working on this for 9.3 mainly because its not a priority, nor > will implementing that fix the OP's request. The job wasn't completed because the demands for backward compatibility were too complex from a coding/user experience perspective, so the original developer just went away. > There is no paralysis because there never was a blocker, only a > request for backwards compatibility, which is easily possible to > implement. OK, and I am saying the request for backwards compatibility is rejected. You want to have a vote on that right now? And don't make your typical demands that you will not 'accept' something that isn't backward compatible. I don't care if you accept anything or not, we are moving ahead, with or without you. If we can't get beyond this, I need to start blogging at how insular our developer team is and how we need new people to joint the hackers list and we can start this discussion all over again with a new group. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 21 December 2012 19:21, Bruce Momjian <bruce@momjian.us> wrote: > On Fri, Dec 21, 2012 at 02:25:47PM +0000, Simon Riggs wrote: >> On 20 December 2012 19:29, Bruce Momjian <bruce@momjian.us> wrote: >> >> > At this point backward compatibility has paralyzed us from fixing a >> > recovery.conf API that everyone agrees is non-optimal, and this has >> > gone on for multiple major releases. I don't care what we have to do, >> > just clean this up for 9.3! >> >> The main stall at this point is that the developer who wrote that >> patch no longer spends much time working on Postgres. AFAICS there is >> nobody working on this for 9.3 mainly because its not a priority, nor >> will implementing that fix the OP's request. > > The job wasn't completed because the demands for backward compatibility > were too complex from a coding/user experience perspective, so the > original developer just went away. It's not too complex. You just want that to be true. The original developer has actually literally gone away, but not because of this. >> There is no paralysis because there never was a blocker, only a >> request for backwards compatibility, which is easily possible to >> implement. > > OK, and I am saying the request for backwards compatibility is rejected. > You want to have a vote on that right now? > > And don't make your typical demands that you will not 'accept' something > that isn't backward compatible. I don't care if you accept anything or > not, we are moving ahead, with or without you. > > If we can't get beyond this, I need to start blogging at how insular our > developer team is and how we need new people to joint the hackers list > and we can start this discussion all over again with a new group. Yes, I think having some people on this list who make decisions after they have heard technical facts would be a welcome change. -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Fri, Dec 21, 2012 at 07:32:48PM +0000, Simon Riggs wrote: > On 21 December 2012 19:21, Bruce Momjian <bruce@momjian.us> wrote: > > On Fri, Dec 21, 2012 at 02:25:47PM +0000, Simon Riggs wrote: > >> On 20 December 2012 19:29, Bruce Momjian <bruce@momjian.us> wrote: > >> > >> > At this point backward compatibility has paralyzed us from fixing a > >> > recovery.conf API that everyone agrees is non-optimal, and this has > >> > gone on for multiple major releases. I don't care what we have to do, > >> > just clean this up for 9.3! > >> > >> The main stall at this point is that the developer who wrote that > >> patch no longer spends much time working on Postgres. AFAICS there is > >> nobody working on this for 9.3 mainly because its not a priority, nor > >> will implementing that fix the OP's request. > > > > The job wasn't completed because the demands for backward compatibility > > were too complex from a coding/user experience perspective, so the > > original developer just went away. > > It's not too complex. You just want that to be true. The original > developer has actually literally gone away, but not because of this. Well, Robert and I remember it differently. Anyway, I will ask for a vote now. > >> There is no paralysis because there never was a blocker, only a > >> request for backwards compatibility, which is easily possible to > >> implement. > > > > OK, and I am saying the request for backwards compatibility is rejected. > > You want to have a vote on that right now? > > > > And don't make your typical demands that you will not 'accept' something > > that isn't backward compatible. I don't care if you accept anything or > > not, we are moving ahead, with or without you. > > > > If we can't get beyond this, I need to start blogging at how insular our > > developer team is and how we need new people to joint the hackers list > > and we can start this discussion all over again with a new group. > > Yes, I think having some people on this list who make decisions after > they have heard technical facts would be a welcome change. OK, I will start blogging too. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 21 December 2012 19:35, Bruce Momjian <bruce@momjian.us> wrote: >> It's not too complex. You just want that to be true. The original >> developer has actually literally gone away, but not because of this. > > Well, Robert and I remember it differently. > > Anyway, I will ask for a vote now. And what will you ask for a vote on? Why not spend that effort on solving the problem? Why is it OK to waste so much time? Having already explained how to do this, I'll add backwards compatibility within 1 day of the commit of the patch you claim was blocked by this. I think it will take me about an hour and not be very invasive, just to prove what a load of hot air is being produced here. >> Yes, I think having some people on this list who make decisions after >> they have heard technical facts would be a welcome change. > > OK, I will start blogging too. Good for you. I'll stick to trying to improve PostgreSQL by coding. -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Fri, Dec 21, 2012 at 07:43:29PM +0000, Simon Riggs wrote: > On 21 December 2012 19:35, Bruce Momjian <bruce@momjian.us> wrote: > > >> It's not too complex. You just want that to be true. The original > >> developer has actually literally gone away, but not because of this. > > > > Well, Robert and I remember it differently. > > > > Anyway, I will ask for a vote now. > > And what will you ask for a vote on? Why not spend that effort on > solving the problem? Why is it OK to waste so much time? I have already sent out the vote request. > Having already explained how to do this, I'll add backwards > compatibility within 1 day of the commit of the patch you claim was > blocked by this. I think it will take me about an hour and not be very > invasive, just to prove what a load of hot air is being produced here. I have seen too many cases where things are derailed. I think we need a clear statement of exactly how important backward compatibility is in this case. I think the fear of you requesting stuff has basically scared everyone away from working on this. Of course, I might be wrong, but I have to make a guess on this one. > >> Yes, I think having some people on this list who make decisions after > >> they have heard technical facts would be a welcome change. > > > > OK, I will start blogging too. > > Good for you. I'll stick to trying to improve PostgreSQL by coding. Well, when our process is broken, coding doesn't really solve the problem, at least from my perspective. Let me explain it this way. A family has a car they love, but the car is in bad shape, so they all go to the auto dealership. They like some features of the new cars, but it doesn't have everying the old car had, so they go to another dealership, and then another. Two years go buy, and they still haven't gotten a new car, and the old car is getting worse. As some point, someone in the family has to stand up and say, "Hey we aren't going to get everything we liked in the old car, but we have to make a decision and move forward." That is where we are now. Overhauling recovery.conf can't be a super-complex job, and we already have a patch we can base it of off. Why is this not done yet! I don't know, but I have seen lots of discussion about it, and no clear conclusions, at least that you agree with. I have realized this could languish for another two years unless I stand up, say the old car is dead, and force us to get a new car! -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 21.12.2012 21:43, Simon Riggs wrote: > On 21 December 2012 19:35, Bruce Momjian<bruce@momjian.us> wrote: > >>> It's not too complex. You just want that to be true. The original >>> developer has actually literally gone away, but not because of this. >> >> Well, Robert and I remember it differently. >> >> Anyway, I will ask for a vote now. > > And what will you ask for a vote on? Why not spend that effort on > solving the problem? Why is it OK to waste so much time? > > Having already explained how to do this, I'll add backwards > compatibility within 1 day of the commit of the patch you claim was > blocked by this. I think it will take me about an hour and not be very > invasive, just to prove what a load of hot air is being produced here. I haven't been following this.. Could you two post a link to the patch we're talking about, and the explanation of how to add backwards compatibility to it? Just by looking at the last few posts, this seems like a no brainer. The impression I get is that there's a patch that's otherwise ready to be applied, but Simon and some others want to have backwards-compatiblity added to it first. And Simon has a plan on how to do it, and can do it in one day. The obvious solution is that Simon posts the patch, with the backwards-compatibility added. We can then discuss that, and assuming no surprises there, commit it. And everyone lives happily ever after. - Heikki
On 12/21/2012 11:56 AM, Bruce Momjian wrote: > That is where we are now. Overhauling recovery.conf can't be a > super-complex job, and we already have a patch we can base it of off. > Why is this not done yet! I don't know, but I have seen lots of > discussion about it, and no clear conclusions, at least that you agree > with. I have realized this could languish for another two years unless > I stand up, say the old car is dead, and force us to get a new car! Forgive me because I have been up for 28 hours on a 9.0 to 9.2 migration with Hot Standby and PgPool-II for load balancing but I was excessively irritated that I had to go into recovery.conf to configure things. I am one of the software authors that breaking recovery.conf will cause problems for. Break it anyway. The fact that anyone has to touch recovery.conf is at this point in software development, backwards. No matter how it gets implemented it shouldn't be anymore complicated than: Master: SET/ENABLE replication FOR db1; Standby: SET/ENABLE hot_standby MASTER db0 Have the logs automatically go to an archive tablespace (so it can be moved) Now I am not expecting this to happen or even saying it is a valid approach. What I am saying is that it should be "THAT" easy [1]. [2] Multi version rpms on the same box still need some help (lib issues) [3] Bruce, you rock... pg_ugrade is the bomb Joshua D. Drake 1. Obviously I am talking about the simplest of configuration and there are a whole slew of other options that needs to be set but have reasonable defaults. -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579
> Forgive me because I have been up for 28 hours on a 9.0 to 9.2 migration > with Hot Standby and PgPool-II for load balancing but I was excessively > irritated that I had to go into recovery.conf to configure things. I am > one of the software authors that breaking recovery.conf will cause > problems for. Break it anyway. Speaking as another vendor who has custom software currently designed to manage recovery.conf: I will *happily* rewrite that software. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On Sat, Dec 22, 2012 at 5:14 AM, Heikki Linnakangas <hlinnakangas@vmware.com> wrote: > On 21.12.2012 21:43, Simon Riggs wrote: >> >> On 21 December 2012 19:35, Bruce Momjian<bruce@momjian.us> wrote: >> >>>> It's not too complex. You just want that to be true. The original >>>> developer has actually literally gone away, but not because of this. >>> >>> >>> Well, Robert and I remember it differently. >>> >>> Anyway, I will ask for a vote now. >> >> >> And what will you ask for a vote on? Why not spend that effort on >> solving the problem? Why is it OK to waste so much time? >> >> Having already explained how to do this, I'll add backwards >> compatibility within 1 day of the commit of the patch you claim was >> blocked by this. I think it will take me about an hour and not be very >> invasive, just to prove what a load of hot air is being produced here. > > > I haven't been following this.. Could you two post a link to the patch we're > talking about, and the explanation of how to add backwards compatibility to > it? The latest patch is the following. Of course, this cannot be applied cleanly in HEAD. http://archives.postgresql.org/message-id/CAHGQGwHB==cJehmE6JiuY-kNutRx-k3YWQziEg1gPNb3CK6N3A@mail.gmail.com > Just by looking at the last few posts, this seems like a no brainer. The > impression I get is that there's a patch that's otherwise ready to be > applied, but Simon and some others want to have backwards-compatiblity added > to it first. And Simon has a plan on how to do it, and can do it in one day. > The obvious solution is that Simon posts the patch, with the > backwards-compatibility added. We can then discuss that, and assuming no > surprises there, commit it. And everyone lives happily ever after. Sounds reasonable. Regards, -- Fujii Masao
On 23 December 2012 15:24, Fujii Masao <masao.fujii@gmail.com> wrote: > The latest patch is the following. Of course, this cannot be applied > cleanly in HEAD. > http://archives.postgresql.org/message-id/CAHGQGwHB==cJehmE6JiuY-kNutRx-k3YWQziEg1gPNb3CK6N3A@mail.gmail.com > >> Just by looking at the last few posts, this seems like a no brainer. The >> impression I get is that there's a patch that's otherwise ready to be >> applied, but Simon and some others want to have backwards-compatiblity added >> to it first. The patch isn't ready to be applied. Reviewing this patch again from scratch, it does the following... 1. Makes recovery.conf parameters into GUCs 2. Moves these parameters to postgresql.conf 3. Changes all the docs referring to recovery.conf What the patch doesn't change is the requirement to have a file that causes the server to place itself into archive recovery. So there is no more recovery.conf and instead we have a file called recovery.trigger instead. We now support the concept of multiple .conf files, so continuing to use a file called recovery.conf for those parameters works just fine now without much effort. Note also that the recovery.conf requirement to put things in quotes still works when they are GUCs, its just no longer a requirement like it used to be. So previous files will work just fine. I don't see any point doing (2) or (3), especially since it makes it a huge patch and especially since the trend is to break down large config files into smaller pieces for use by Chef and Puppet. Given the continuing need for a file to trigger recovery, changing the name of the file from recovery.conf to recovery.trigger just breaks backwards compatibility for absolutely no gain whatsoever (as long as we do (1)). I think it would be much easier to do just (1) in this release. What we do want is the ability to move recovery.conf elsewhere (if we choose), so that data directory is not writable by anything but postgres server. If we do that, we'll have to make the config directory writable by the server, whereas it wasn't before, so that can cause problems also. That's the most useful additional change that I see in this area, even better than (1). (Note that you can't tell which server is the master purely from reading primary_conninfo, since that you can't tell if you're a cascading standby, or not. As discussed elsewhere, we could actually do some more pushups to identify the current From my perspective this patch and its design wasn't well thought through and on balance I'm happy it wasn't committed earlier. From all of the above, I think its worth doing this * allowing recovery.conf to be in a different directory * make recovery config parameters into GUCs * no other changes to way things currently work I don't think that represents enough change to keep people happy, but I don't see anything else useful being suggested in this patch. Other design thoughts welcome, but personally, I think rushing this design through at this stage is likely to require us to change the design again in later releases. Productive comments welcome, Merry Christmas -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Mon, Dec 24, 2012 at 03:13:59PM +0000, Simon Riggs wrote: > I don't think that represents enough change to keep people happy, but > I don't see anything else useful being suggested in this patch. Other > design thoughts welcome, but personally, I think rushing this design > through at this stage is likely to require us to change the design > again in later releases. Simon, you just agreed to: > At this point, backward compatibility seems to be hampering our ability > to move forward. I would like a vote that supports creation of a new > method for setting up streaming replication/point-in-time-recovery, > where backward compatibility is considered only where it is minimally > invasive. Let's figure out the API we want and implement it. If we haven't figured out a perfect answer in 2 years, we never will and we should just do our best. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 24 December 2012 15:48, Bruce Momjian <bruce@momjian.us> wrote: > On Mon, Dec 24, 2012 at 03:13:59PM +0000, Simon Riggs wrote: >> I don't think that represents enough change to keep people happy, but >> I don't see anything else useful being suggested in this patch. Other >> design thoughts welcome, but personally, I think rushing this design >> through at this stage is likely to require us to change the design >> again in later releases. > > Simon, you just agreed to: > >> At this point, backward compatibility seems to be hampering our ability >> to move forward. I would like a vote that supports creation of a new >> method for setting up streaming replication/point-in-time-recovery, >> where backward compatibility is considered only where it is minimally >> invasive. > > Let's figure out the API we want and implement it. That's exactly what I spent the afternoon doing. Merry Christmas -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Mon, Dec 24, 2012 at 03:57:10PM +0000, Simon Riggs wrote: > On 24 December 2012 15:48, Bruce Momjian <bruce@momjian.us> wrote: > > On Mon, Dec 24, 2012 at 03:13:59PM +0000, Simon Riggs wrote: > >> I don't think that represents enough change to keep people happy, but > >> I don't see anything else useful being suggested in this patch. Other > >> design thoughts welcome, but personally, I think rushing this design > >> through at this stage is likely to require us to change the design > >> again in later releases. > > > > Simon, you just agreed to: > > > >> At this point, backward compatibility seems to be hampering our ability > >> to move forward. I would like a vote that supports creation of a new > >> method for setting up streaming replication/point-in-time-recovery, > >> where backward compatibility is considered only where it is minimally > >> invasive. > > > > Let's figure out the API we want and implement it. > > That's exactly what I spent the afternoon doing. OK, is that your 1-3, and you only want #1? > 1. Makes recovery.conf parameters into GUCs > 2. Moves these parameters to postgresql.conf > 3. Changes all the docs referring to recovery.conf Is that what everyone else wants? If that is all, let's do it and close the item. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Bruce Momjian <bruce@momjian.us> writes: > Is that what everyone else wants? If that is all, let's do it and close > the item. What Simon is proposing is quite clear and not what you pasted, if I'm reading that correctly: On Mon, Dec 24, 2012 at 03:13:59PM +0000, Simon Riggs wrote: > From all of the above, I think its worth doing this > * allowing recovery.conf to be in a different directory > * make recovery config parameters into GUCs > * no other changes to way things currently work I happen to like this proposal, +1 from me. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On 2012-12-24 18:06:44 +0100, Dimitri Fontaine wrote: > Bruce Momjian <bruce@momjian.us> writes: > > Is that what everyone else wants? If that is all, let's do it and close > > the item. > > What Simon is proposing is quite clear and not what you pasted, if I'm > reading that correctly: > > On Mon, Dec 24, 2012 at 03:13:59PM +0000, Simon Riggs wrote: > > From all of the above, I think its worth doing this > > * allowing recovery.conf to be in a different directory > > * make recovery config parameters into GUCs > > * no other changes to way things currently work > > I happen to like this proposal, +1 from me. It also seems to be a good building block to do work on the other things asked for in this and related threads. Greetings, Andres Freund --Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Simon, > What the patch doesn't change is the requirement to have a file that > causes the server to place itself into archive recovery. So there is > no more recovery.conf and instead we have a file called > recovery.trigger instead. Requiring a file in order to make a server a replica is what we should be trying to get away from. It should be possible to configure a server as a replica by setting a GUC in PostgreSQL.conf (before first startup, obviously). Naturally, this then links in with SET PERSISTENT or however we're calling it these days in order to take a server out of replica mode. > We now support the concept of multiple .conf files, so continuing to > use a file called recovery.conf for those parameters works just fine > now without much effort. Note also that the recovery.conf requirement > to put things in quotes still works when they are GUCs, its just no > longer a requirement like it used to be. So previous files will work > just fine. You're assuming the current recovery.conf-->recovery.done arrangement, or its analog, is a given. I'm saying that's the biggest thing we want to change. > Given the continuing need for a file to trigger recovery, changing the > name of the file from recovery.conf to recovery.trigger just breaks > backwards compatibility for absolutely no gain whatsoever (as long as > we do (1)). I agree. If we can't dump the trigger file, there's no good reason to change its name. However, I want to get rid of the trigger file. > From all of the above, I think its worth doing this > * allowing recovery.conf to be in a different directory > * make recovery config parameters into GUCs > * no other changes to way things currently work I agree that the above would be worth doing if that's all we can get accomplished in 9.3. Let's determine that it is all we can get accomplished before we settle, though. From my perspective, the best API would be: 1. all current recovery.conf parameters get moved into PostgreSQL.conf. 2. we add a parameter called replication_role (or similar), which can be "master","replica", or "warm_standby" (someone else can make a case for "none" if there is one). 3. replica promotion is accomplished by one of 2 mechanisms: a. user edits postgresql.conf, changes replication_role, and reloads. b. pg_ctl promote, which also rewrites replication_role in postgresql.conf (hence, link to SET PERSISTENT). 4. many current recovery.conf paramters become changeable with a HUP, including primary_conninfo. 5. primary_conninfo supports the URI format as well as the present one. While the replication change mechanisms of recovery.done and the replication-ending trigger file are nice for some users, they aren't (IMHO) valuable enough to be worth preserving if they would compromise the above, much cleaner, API. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Josh Berkus <josh@agliodbs.com> writes:
>> What the patch doesn't change is the requirement to have a file that
>> causes the server to place itself into archive recovery. So there is
>> no more recovery.conf and instead we have a file called
>> recovery.trigger instead.
> Requiring a file in order to make a server a replica is what we should
> be trying to get away from. It should be possible to configure a server
> as a replica by setting a GUC in PostgreSQL.conf (before first startup,
> obviously).
I'm not entirely convinced about that, because if we do it like that, we
will *never*, *ever* be able to store GUC settings except in a flat,
editable textfile. Now, that's fine by me personally, but there seem to
be a lot of people around here with ambitions to bury those settings in
not-so-editable places. Including you, to judge by your next sentence:
> Naturally, this then links in with SET PERSISTENT or
> however we're calling it these days in order to take a server out of
> replica mode.
People are going to want to be able to push a server into, and possibly
out of, replica mode without necessarily having the server up at the
time. So I'm not real convinced that we want that flag to be a GUC.
A trigger file is a lot easier to manipulate from places like shell
scripts.
regards, tom lane
On Mon, Dec 24, 2012 at 7:04 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Josh Berkus <josh@agliodbs.com> writes: >>> What the patch doesn't change is the requirement to have a file that >>> causes the server to place itself into archive recovery. So there is >>> no more recovery.conf and instead we have a file called >>> recovery.trigger instead. > >> Requiring a file in order to make a server a replica is what we should >> be trying to get away from. It should be possible to configure a server >> as a replica by setting a GUC in PostgreSQL.conf (before first startup, >> obviously). > > I'm not entirely convinced about that, because if we do it like that, we > will *never*, *ever* be able to store GUC settings except in a flat, > editable textfile. Now, that's fine by me personally, but there seem to > be a lot of people around here with ambitions to bury those settings in > not-so-editable places. Including you, to judge by your next sentence: > >> Naturally, this then links in with SET PERSISTENT or >> however we're calling it these days in order to take a server out of >> replica mode. > > People are going to want to be able to push a server into, and possibly > out of, replica mode without necessarily having the server up at the > time. So I'm not real convinced that we want that flag to be a GUC. > A trigger file is a lot easier to manipulate from places like shell > scripts. > I'm not sure that my POV exactly matches up with Tom's, but on the last point, I strongly agree that the use of the trigger file makes it trivial to integrate Postgres warm standby management into 3rd party tools. I'm not against coming up with a new API that's better for postgres dedicated tools, but I think you're going to really make it harder for people if you eliminate the trigger file method for coming out of recovery. Robert Treat play: xzilla.net work: omniti.com
> I'm not sure that my POV exactly matches up with Tom's, but on the > last point, I strongly agree that the use of the trigger file makes it > trivial to integrate Postgres warm standby management into 3rd party > tools. I'm not against coming up with a new API that's better for > postgres dedicated tools, but I think you're going to really make it > harder for people if you eliminate the trigger file method for coming > out of recovery. Huh. My experience integrating PostgreSQL with Puppet or SALT infrastructures is that they don't understand trigger files, but they do understand configuration+restart/reload. Before we get off on an argument about which is better, though, here's an important question: how difficult would it be to make the trigger file optional, but still effective? That is, I personally don't care if other people use trigger files, I just hate to be forced to use them myself. Is it possible to support both options without making either the code or the API hopelessly confusing? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On 26.12.2012 21:55, Josh Berkus wrote: > >> I'm not sure that my POV exactly matches up with Tom's, but on the >> last point, I strongly agree that the use of the trigger file makes it >> trivial to integrate Postgres warm standby management into 3rd party >> tools. I'm not against coming up with a new API that's better for >> postgres dedicated tools, but I think you're going to really make it >> harder for people if you eliminate the trigger file method for coming >> out of recovery. > > Huh. My experience integrating PostgreSQL with Puppet or SALT > infrastructures is that they don't understand trigger files, but they do > understand configuration+restart/reload. Before we get off on an > argument about which is better, though, here's an important question: > how difficult would it be to make the trigger file optional, but still > effective? > > That is, I personally don't care if other people use trigger files, I > just hate to be forced to use them myself. Is it possible to support > both options without making either the code or the API hopelessly > confusing? There already are two ways to promote a server out of recovery. One is creating the trigger file. The other is "pg_ctl promote". (it uses a trigger file called $PGDATA/promote internally, but that's invisible to the user). - Heikki
> There already are two ways to promote a server out of recovery. One is > creating the trigger file. The other is "pg_ctl promote". (it uses a > trigger file called $PGDATA/promote internally, but that's invisible to > the user). Right, I was thinking of the trigger file to put a server *into* replication. That is, recovery.conf. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On Wed, Dec 26, 2012 at 3:32 PM, Josh Berkus <josh@agliodbs.com> wrote: >> There already are two ways to promote a server out of recovery. One is >> creating the trigger file. The other is "pg_ctl promote". (it uses a >> trigger file called $PGDATA/promote internally, but that's invisible to >> the user). > > Right, I was thinking of the trigger file to put a server *into* > replication. That is, recovery.conf. Well, we can't put the server back into recovery once it's up and running. You have to restart for that, and I can't see that changing any time soon. I think we could support something like: pg_ctl start -m recover Of course, that would require that primary_conninfo et. al. be available from somewhere. If we got rid of all the recovery.conf parameters and made them GUCs, then that'd be pretty easy: the setting would always be available from postgresql.conf, but would be ignored except when starting in recovery mode. In my view, the biggest problem with recovery.conf is that the parameters in there are not GUCs, which means that all of the infrastructure that we've built for managing GUCs does not work with them. As an example, when we converted recovery.conf to use the same lexer that the GUC machinery uses, it allowed recovery.conf values to be specified unquoted in the same circumstances where that was already possible for postgresql.conf. But, you still can't use SHOW or pg_settings with recovery.conf parameters, and I think pg_ctl reload doesn't work either. If we make these parameters into GUCs, then they'll work the same way everything else works. Even if (as seems likely) we end up still needing a trigger file (or a special pg_ctl mode) to initiate recovery, I think that's probably a win. Other people's mileage may vary, of course. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert, > In my view, the biggest problem with recovery.conf is that the > parameters in there are not GUCs, which means that all of the > infrastructure that we've built for managing GUCs does not work with > them. As an example, when we converted recovery.conf to use the same > lexer that the GUC machinery uses, it allowed recovery.conf values to > be specified unquoted in the same circumstances where that was already > possible for postgresql.conf. But, you still can't use SHOW or > pg_settings with recovery.conf parameters, and I think pg_ctl reload > doesn't work either. If we make these parameters into GUCs, then > they'll work the same way everything else works. Even if (as seems > likely) we end up still needing a trigger file (or a special pg_ctl > mode) to initiate recovery, I think that's probably a win. I agree that it would be an improvement, and I would be happy just to see the parameters become GUCs. I'm just saying that I'll still be pushing to get rid of the requirement for recovery.conf in 9.4, that's all. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On 3 January 2013 18:35, Josh Berkus <josh@agliodbs.com> wrote: > Robert, > >> In my view, the biggest problem with recovery.conf is that the >> parameters in there are not GUCs, which means that all of the >> infrastructure that we've built for managing GUCs does not work with >> them. As an example, when we converted recovery.conf to use the same >> lexer that the GUC machinery uses, it allowed recovery.conf values to >> be specified unquoted in the same circumstances where that was already >> possible for postgresql.conf. But, you still can't use SHOW or >> pg_settings with recovery.conf parameters, and I think pg_ctl reload >> doesn't work either. If we make these parameters into GUCs, then >> they'll work the same way everything else works. Even if (as seems >> likely) we end up still needing a trigger file (or a special pg_ctl >> mode) to initiate recovery, I think that's probably a win. > > I agree that it would be an improvement, and I would be happy just to > see the parameters become GUCs. That may be possible in 9.3 since we have a patch from Fujii-san. I'll hack that down to just the GUC part once we start the next CF. My personal priority is the shutdown checkpoint patch over that though. > I'm just saying that I'll still be pushing to get rid of the requirement > for recovery.conf in 9.4, that's all. No pushing required. When we have a reasonable proposal that improves on the current state, we can implement that. -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Thu, Jan 3, 2013 at 07:45:32PM +0000, Simon Riggs wrote: > On 3 January 2013 18:35, Josh Berkus <josh@agliodbs.com> wrote: > > Robert, > > > >> In my view, the biggest problem with recovery.conf is that the > >> parameters in there are not GUCs, which means that all of the > >> infrastructure that we've built for managing GUCs does not work with > >> them. As an example, when we converted recovery.conf to use the same > >> lexer that the GUC machinery uses, it allowed recovery.conf values to > >> be specified unquoted in the same circumstances where that was already > >> possible for postgresql.conf. But, you still can't use SHOW or > >> pg_settings with recovery.conf parameters, and I think pg_ctl reload > >> doesn't work either. If we make these parameters into GUCs, then > >> they'll work the same way everything else works. Even if (as seems > >> likely) we end up still needing a trigger file (or a special pg_ctl > >> mode) to initiate recovery, I think that's probably a win. > > > > I agree that it would be an improvement, and I would be happy just to > > see the parameters become GUCs. > > That may be possible in 9.3 since we have a patch from Fujii-san. I'll > hack that down to just the GUC part once we start the next CF. > > My personal priority is the shutdown checkpoint patch over that though. > > > I'm just saying that I'll still be pushing to get rid of the requirement > > for recovery.conf in 9.4, that's all. > > No pushing required. When we have a reasonable proposal that improves > on the current state, we can implement that. Sounds like we are all in agreement and on a good track to completion. I apologize for overreacting and thinking we were not addressing this issue objectively. Thanks for the discussion. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 24 December 2012 17:15, Andres Freund <andres@2ndquadrant.com> wrote: >> On Mon, Dec 24, 2012 at 03:13:59PM +0000, Simon Riggs wrote: >> > From all of the above, I think its worth doing this >> > * allowing recovery.conf to be in a different directory >> > * make recovery config parameters into GUCs >> > * no other changes to way things currently work >> >> I happen to like this proposal, +1 from me. > > It also seems to be a good building block to do work on the other things > asked for in this and related threads. As discussed previously, I've committed a patch for * allowing recovery.conf to be in a different directory and am now starting work on * make recovery config parameters into GUCs using Fujii's patch as a start These are separate thoughts and hence separate patches. -- Simon Riggs http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services