Thread: Mammoth in Core?
Hello, O.k. I know there is no way we will hit this for 8.5. So this is more of a future discussion more than anything. We at CMD have been working diligently on our next version of Mammoth Replicator, 1.9. It is currently revved at 8.4. I expect that we will be close to done if not done, by the release of 8.5. My question is, do we have any interest in working on getting this into core? To give those that don't have any background with Mammoth here is the run down: 1. It is a patch to .Org. E.g; it integrates with the backend unlike Slony or Londiste. 2. 1.9 remove the SPOF problem of the 1.8 series by adding forwarder capabilities within the postmaster itself. (1.8 used a secondary daemon) 3. It has been developed for years as a proprietary product, but was released as BSD about a year ago. It supports the following features: * Data replication * Partial replication (to multiple different slaves) * Large Object replication * ACL (GRANT/REVOKE) replication * ALTER/CREATE ROLE * Promotion (And promote back) * Firing triggers on a slave with replicated relations (for reporting, materialized views etc...) * Monitoring The docs are here: https://projects.commandprompt.com/public/replicator/wiki/Documentation/current There are some limitations, which could be addressed. I would have to talk with Alvaro and Alexey further on them but this is more of a field test. If the community is interested in having a full scale replication system in the backend (HS and SR provide different facilities) then CMD is interested in making this community ready. If the community isn't interested, we are likely to start putting our efforts elsewhere (as opposed to Mammoth Replicator). Sincerely, Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
"Joshua D. Drake" <jd@commandprompt.com> writes:
> O.k. I know there is no way we will hit this for 8.5. So this is more of
> a future discussion more than anything.
Well, this is not really the time to be having such a discussion; right
now we need to all have our noses to the grindstone dealing with the
already-submitted 8.5 features. The start of the next devel cycle would
be a more appropriate time to think about it.
(Personally, I suspect we're going to have our hands full dealing with
HS+SR for quite some time to come, which implies we should not scatter
our energies across multiple replication solutions. But that will be
clearer in a few months when we see what emerges from beta.)
regards, tom lane
"Joshua D. Drake" <jd@commandprompt.com> wrote: > My question is, do we have any interest in working on getting this into > core? We had a discussion how replication projects work together with the core in the developer meeting on PGCon 2009 Japan. http://wiki.postgresql.org/wiki/PGCon2009JapanClusterDeveloperMeeting The conclusion is splitting existing projects into some 'modules', and getting the modules one by one into core. Voted features are here: http://wiki.postgresql.org/wiki/ClusterFeatures Mammoth Replicator seems to modify the core heavily, no? I hope you would split the mammoth patch into several independent features, and submit them separately. It is much better if some of them can be shared by other replication projects. Regards, --- Takahiro Itagaki NTT Open Source Software Center
Takahiro Itagaki wrote: > The conclusion is splitting existing projects into some 'modules', > and getting the modules one by one into core. Voted features are here: > http://wiki.postgresql.org/wiki/ClusterFeatures > This page was a bit messy for someone who didn't attend the meeting to follow. I just cleaned it up so that the features are listed in voting order, and to have more inter-page links. It's better, but could use some more work still. There are a few things I noted that I didn't see in the voting order; I moved those all into another section. If anyone who was there can help prioritize those it would be helpful. Also, the voting included "XID set", but I didn't see any section that clearly matched that, so there may be a missing description section there too. Stepping back for a second, there are three big picture things that I think need to be sorted out before it's possible for this to be useful guidance for something like suggesting how JD might best fit his Mammoth work into the broad work being done in this area, which I consider a subset of the larger important question "how can people help here?": -There are dependencies that exist regardless of what order people would like to see the features at. For example, the one I thought I saw listed and therefore documented is that the "Modification trigger into core" feature presumes you've already resolved "Generalized Data Queue". Are there more of those? Does "Function scan push-down" depend on "Export snapshots" already being done? Those are the kind of questions I'm left with when reading. -Who if anyone is actually working in this area already with prototypes or at all? In some cases these are listed (like notes referencing Slony/Londiste etc.); it would be helpful to explicitly nail those down in every case. For the Mammoth example, that might help identify which components they have a uniquely useful piece for. -Is there anyone who's taken on responsibility for working on any of these specific parts yet? When we had a similar work prioritization session at the PGCon developer's meeting, most of the action items there came with a clearly labeled person on the hook who was working on them. I don't see any notion like that from this meeting's outcome. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com
Hi, Greg Smith wrote: > Takahiro Itagaki wrote: >> The conclusion is splitting existing projects into some 'modules', >> and getting the modules one by one into core. Voted features are here: >> http://wiki.postgresql.org/wiki/ClusterFeatures That's certainly been one of the outcomes, however, there was a lot more to it that just that. You find a bit more covered on http://wiki.postgresql.org/wiki/PGCon2009JapanClusterDeveloperMeeting (and some of its links), but I understand it's hard to follow. It's unfortunate that none of the Mammoth developers have been around. It would have been interesting hearing their POV as well. > There are a few things I noted that I didn't see in the voting order; I > moved those all into another section. If anyone who was there can help > prioritize those it would be helpful. Also, the voting included "XID > set", but I didn't see any section that clearly matched that, so there > may be a missing description section there too. I don't remember, sorry. Note that this is just a wish-list of things that the clustering hackers possibly want in core. There are lots of little, but important details that still need to be fleshed out for every single item. > Stepping back for a second, there are three big picture things that I > think need to be sorted out before it's possible for this to be useful > guidance for something like suggesting how JD might best fit his Mammoth > work into the broad work being done in this area, which I consider a > subset of the larger important question "how can people help here?" Well, that's not exactly the kind of question JD posed. He offered to help with making Mammoth "community ready" - whatever that is. (And I bet not even "the community" knows. You certainly won't get a single answer). However, I'm in a somewhat similar situation with Postgres-R. And since the clustering meeting, I'm even more convinced that I need to break it apart and split into single modules, which might be of use for others. I didn't have no luck with the imessages patch, so far. But Kevin seems to be interested in the testing framework I'm putting together [1] - and he's not even using it for a clustering project. (And another module that's core code is in the queue, so stay tuned). So, that's what I'd recommend the Mammoth developers to do as well: cherry-picking, sort of. Maybe that fulfills one or the other item on our wish-list (in one way or another)... Regards Markus Wanner [1]: dtester http://www.bluegap.ch/projects/dtester
On Tue, 2010-01-19 at 21:55 +0100, Markus Wanner wrote: > Hi, > So, that's what I'd recommend the Mammoth developers to do as well: > cherry-picking, sort of. Maybe that fulfills one or the other item on > our wish-list (in one way or another)... > I doubt we are going to spend the time to do that. Mammoth is BSD and open source. If people want to jump in and help, that would be interesting, but on our own... it isn't on our priority list. Our priority list with mammoth is simple. Finish 1.9 Fix things we know the community as hackers (not feature set) will grump about. Make it work on 8.4 ad 8.5 But again, as Tom says... none of this is relevant until 8.5 is released at which point, we can talk about all of this again. Hopefully 1.9 will be done by then. Done. Joshua D. Drake --
On 1/19/10 9:28 AM, Greg Smith wrote: > Takahiro Itagaki wrote: >> The conclusion is splitting existing projects into some 'modules', >> and getting the modules one by one into core. Voted features are here: >> http://wiki.postgresql.org/wiki/ClusterFeatures >> > This page was a bit messy for someone who didn't attend the meeting to > follow. I just cleaned it up so that the features are listed in voting > order, and to have more inter-page links. It's better, but could use > some more work still. Yeah, we'll be cleaning it up and fleshing it out more from now until pgCon. --Josh Berkus
> On 1/19/10 9:28 AM, Greg Smith wrote: > > Takahiro Itagaki wrote: > >> The conclusion is splitting existing projects into some 'modules', > >> and getting the modules one by one into core. Voted features are here: > >> http://wiki.postgresql.org/wiki/ClusterFeatures "splitting existing projects into some 'modules', and getting the modules one by one into core" was not the concluion, actually. For example, see below from above URL: This means that we expect PostgreSQL exports it's parser so that existing cluster softwares can use it. Not opposite direction. > API into the Parser / Parser as an independent module > > * just first step, before going to the catalog > * statement based replication need to reply certain constructs with CONSTAnT values you provide > * Figure out which you need to replace... quite difficult > * but if we have the plan, could do better -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp > > This page was a bit messy for someone who didn't attend the meeting to > > follow. I just cleaned it up so that the features are listed in voting > > order, and to have more inter-page links. It's better, but could use > > some more work still. > > Yeah, we'll be cleaning it up and fleshing it out more from now until pgCon. > > --Josh Berkus > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers
Tatsuo Ishii <ishii@postgresql.org> wrote: > "splitting existing projects into some 'modules', and getting the > modules one by one into core" was not the concluion, actually. > > For example, see below from above URL: This means that we expect > PostgreSQL exports it's parser so that existing cluster softwares can > use it. Not opposite direction. I think they says the same practically -- at least have the same impact. It says postgres need to export the the internal feature *only for* some of external cluster softwares. So, if you are thinking about exporting some features from the core, the exported features would better to be stable enough and shared by several third-party tools. Regards, --- Takahiro Itagaki NTT Open Source Software Center
Takahiro Itagaki <itagaki.takahiro@oss.ntt.co.jp> writes:
> Tatsuo Ishii <ishii@postgresql.org> wrote:
>> For example, see below from above URL: This means that we expect
>> PostgreSQL exports it's parser so that existing cluster softwares can
>> use it. Not opposite direction.
> I think they says the same practically -- at least have the same impact.
> It says postgres need to export the the internal feature *only for* some
> of external cluster softwares. So, if you are thinking about exporting
> some features from the core, the exported features would better to be
> stable enough and shared by several third-party tools.
[ raised eyebrow... ]
It's going to be a really, really, *really* hard sell to get us to
export any sort of external API to the parser internals. At least
if by "API" you mean something other than "we will whack this around
to an indefinite degree on no notice, and don't even think about
complaining".
What exactly is the goal that you think such a thing would serve,
anyway? The fragments on the referenced web page don't leave me with
any warm feelings about how well the idea has been thought through.
regards, tom lane
Tom Lane <tgl@sss.pgh.pa.us> wrote: > It's going to be a really, really, *really* hard sell to get us to > export any sort of external API to the parser internals. At least > if by "API" you mean something other than "we will whack this around > to an indefinite degree on no notice, and don't even think about > complaining". > > What exactly is the goal that you think such a thing would serve, > anyway? The fragments on the referenced web page don't leave me with > any warm feelings about how well the idea has been thought through. Some of items in the referenced web page are just voted results form cluster projects. At this time, we should read them as "what is needed", but not "how to do it". They have been not reviewd yet and not well-considered to be official TODO items. I知 not sure what pgpool team think about, but I do NOT intend to export the existing internal functions as-is. As for my personal goal, I think pgpool should be re-implemented on the layers of SQL/MED FDW or planner/executor hooks. I'd say the SQL/MED FDW apporach is "one by one into core (from projects)", and the hook apporach is "external API (from core)". Regards, --- Takahiro Itagaki NTT Open Source Software Center
All, Please rename this thread. I think that several people to whom it's relevant aren't replying to it because of the deceptive name. Thanks. --Josh
On Mon, 2010-01-25 at 10:50 -0800, Josh Berkus wrote: > All, > > Please rename this thread. I think that several people to whom it's > relevant aren't replying to it because of the deceptive name. Thanks. > Except that it is. This thread was about whether or not, Mammoth would be considered to be in core. Which is fairly independent of the clustering docs. My suggestion would be to start a new thread entirely. Joshua D. Drake > --Josh > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
Joshua D. Drake wrote: > My suggestion would be to start a new thread entirely. > And mine would be to drop this whole topic altogether until after the CommitFest is over, and instead focus on the replication features already committed that need work before beta can even start. Only reason I said something is that I objected to the idea that the information on the wiki was complete enough to be considered a roadmap for others to follow in this area and wanted to clarify that it wasn't--it's helpful, but everybody knows it still needs work yet. Actually improving things enough that it is, that work I think is well outside of what this list should be focusing on right now. We've got pgsql-cluster-hackers to discuss this particular area. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com
On Mon, 2010-01-25 at 10:50 -0800, Josh Berkus wrote: > All, > > Please rename this thread. I think that several people to whom it's > relevant aren't replying to it because of the deceptive name. Thanks. > Except that it is. This thread was about whether or not, Mammoth would be considered to be in core. Which is fairly independent of the clustering docs. My suggestion would be to start a new thread entirely. Joshua D. Drake > --Josh > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.
Greg Smith wrote: > We've got pgsql-cluster-hackers to discuss this > particular area. Huh, is this a new list? It wasn't added to wwwmaster's list of lists, apparently, right? -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
Alvaro Herrera wrote: > Greg Smith wrote: > > >> We've got pgsql-cluster-hackers to discuss this >> particular area. >> > > Huh, is this a new list? It wasn't added to wwwmaster's list of lists, > apparently, right? > The archives are at http://archives.postgresql.org/pgsql-cluster-hackers/ but it's not listed at http://archives.postgresql.org/ for some reason. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com
Greg Smith wrote: > Alvaro Herrera wrote: > >Greg Smith wrote: > > > >>We've got pgsql-cluster-hackers to discuss this > >>particular area. > > > >Huh, is this a new list? It wasn't added to wwwmaster's list of lists, > >apparently, right? > > The archives are at > http://archives.postgresql.org/pgsql-cluster-hackers/ but it's not > listed at http://archives.postgresql.org/ for some reason. Because it wasn't added to the database. It seems we've gotten sloppy about the list creation process (though I admit it doesn't seem to be documented anywhere). If you give me a description for the list I'll add it. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
Alvaro Herrera wrote: > Greg Smith wrote: > >> Alvaro Herrera wrote: >> >>> Greg Smith wrote: >>> >>> >>>> We've got pgsql-cluster-hackers to discuss this >>>> particular area. >>>> >>> Huh, is this a new list? It wasn't added to wwwmaster's list of lists, >>> apparently, right? >>> >> The archives are at >> http://archives.postgresql.org/pgsql-cluster-hackers/ but it's not >> listed at http://archives.postgresql.org/ for some reason. >> > > Because it wasn't added to the database. It seems we've gotten sloppy > about the list creation process (though I admit it doesn't seem to be > documented anywhere). > > If you give me a description for the list I'll add it. > Discussion of adding replication and clustering features to PostgreSQL, both inside the database and via integration with additional software. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com
Greg Smith wrote: > Alvaro Herrera wrote: > >Greg Smith wrote: > >>The archives are at > >>http://archives.postgresql.org/pgsql-cluster-hackers/ but it's not > >>listed at http://archives.postgresql.org/ for some reason. > > > >Because it wasn't added to the database. It seems we've gotten sloppy > >about the list creation process (though I admit it doesn't seem to be > >documented anywhere). > > > >If you give me a description for the list I'll add it. > > Discussion of adding replication and clustering features to > PostgreSQL, both inside the database and via integration with > additional software. Added, thanks. -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Joshua D. Drake wrote: > On Tue, 2010-01-19 at 21:55 +0100, Markus Wanner wrote: > > Hi, > > > So, that's what I'd recommend the Mammoth developers to do as well: > > cherry-picking, sort of. Maybe that fulfills one or the other item on > > our wish-list (in one way or another)... > > > > I doubt we are going to spend the time to do that. Mammoth is BSD and > open source. If people want to jump in and help, that would be > interesting, but on our own... it isn't on our priority list. > > Our priority list with mammoth is simple. > > Finish 1.9 > > Fix things we know the community as hackers (not feature set) will grump > about. > > Make it work on 8.4 ad 8.5 > > But again, as Tom says... none of this is relevant until 8.5 is released > at which point, we can talk about all of this again. Hopefully 1.9 will > be done by then. I wanted to comment on this, however late I am. I think the writing is on the wall and it is now questionable if there is much community value to continue development of Mammoth Replicator, though we might want to pull some ideas from the code at some point. Company-sponsored development is a valuable resource for this community and I would love to see new things worked on. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +