Thread: Re: xReader, double-effort (was: Temporary tables under hot standby)
On Thu, Apr 26, 2012 at 3:40 PM, Kevin Grittner <Kevin.Grittner@wicourts.gov> wrote: > [resending because of postgresql.org bounces on first try] > > Simon Riggs <simon@2ndquadrant.com> wrote: >> Kevin Grittner <Kevin.Grittner@wicourts.gov> wrote: > >>> The GSoC xReader project is intended to be a major step toward >>> that, by providing a way to translate the WAL stream to a series >>> of notifications of logical events to clients which register with >>> xReader. <sigh> Yes, it appears very similar. I didn't know about this. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Thu, Apr 26, 2012 at 11:02 AM, Simon Riggs <simon@2ndquadrant.com> wrote: > On Thu, Apr 26, 2012 at 3:40 PM, Kevin Grittner > <Kevin.Grittner@wicourts.gov> wrote: >> [resending because of postgresql.org bounces on first try] >> >> Simon Riggs <simon@2ndquadrant.com> wrote: >>> Kevin Grittner <Kevin.Grittner@wicourts.gov> wrote: >> >>>> The GSoC xReader project is intended to be a major step toward >>>> that, by providing a way to translate the WAL stream to a series >>>> of notifications of logical events to clients which register with >>>> xReader. > > <sigh> Yes, it appears very similar. I didn't know about this. I don't recall seeing any previous postings about the 2Q project, either. We should try to keep these discussions on-list. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Thu, Apr 26, 2012 at 5:57 PM, Robert Haas <robertmhaas@gmail.com> wrote: > On Thu, Apr 26, 2012 at 11:02 AM, Simon Riggs <simon@2ndquadrant.com> wrote: >> On Thu, Apr 26, 2012 at 3:40 PM, Kevin Grittner >> <Kevin.Grittner@wicourts.gov> wrote: >>> [resending because of postgresql.org bounces on first try] >>> >>> Simon Riggs <simon@2ndquadrant.com> wrote: >>>> Kevin Grittner <Kevin.Grittner@wicourts.gov> wrote: >>> >>>>> The GSoC xReader project is intended to be a major step toward >>>>> that, by providing a way to translate the WAL stream to a series >>>>> of notifications of logical events to clients which register with >>>>> xReader. >> >> <sigh> Yes, it appears very similar. I didn't know about this. > > I don't recall seeing any previous postings about the 2Q project, > either. We should try to keep these discussions on-list. While that's true, it wasn't much of a stretch to think that I might be planning to do more work on replication since I've done that for years and years now. Also, I do keep requesting GSOC doesn't put such major tasks out there, but not much heed taken. Beautiful illustration of a multi-master conflict case though. Not sure just yet what the resolution is, and it will probably take a few weeks to resolve that happily. Glad we're all facing in the same direction, which is the most important thing. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
>>>>>> The GSoC xReader project is intended to be a major step toward >>>>>> that, by providing a way to translate the WAL stream to a series >>>>>> of notifications of logical events to clients which register with >>>>>> xReader. >>> >>> <sigh> Yes, it appears very similar. I didn't know about this. This isn't a problem. In fact, it's the *exact opposite* of a problem. The student wanted to do more advanced things with the xlog stream, and we convinced him to narrow the scope in order to make the project achievable. With 2Q having done some/most of the basic work, the student can focus on more advanced applications of the data again. Win! However, it would be great if someone from 2Q co-mentored him. --Josh
On 04/26/2012 12:57 PM, Robert Haas wrote: > I don't recall seeing any previous postings about the 2Q project, > either. We should try to keep these discussions on-list. First brought up two months ago to the list it seemed most relevant to at the time: http://archives.postgresql.org/pgsql-cluster-hackers/2012-02/msg00001.php and the solid commentary started coming out yesterday, right near the approach of May as promised. There hasn't been much posted on hackers about this out of respect for the extended CommitFest. I personally suggested Aakash vent ideas out on pgsql-students (or here) at the beginning of the month: http://archives.postgresql.org/pgsql-students/2012-04/msg00002.php but the next time I heard about the subject was the acceptance notice. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Greg Smith <greg@2ndQuadrant.com> wrote: > On 04/26/2012 12:57 PM, Robert Haas wrote: >> I don't recall seeing any previous postings about the 2Q project, >> either. We should try to keep these discussions on-list. > > First brought up two months ago to the list it seemed most > relevant to at the time: > http://archives.postgresql.org/pgsql-cluster-hackers/2012-02/msg00001.php > and the solid commentary started coming out yesterday, right near > the approach of May as promised. There hasn't been much posted on > hackers about this out of respect for the extended CommitFest. > > I personally suggested Aakash vent ideas out on pgsql-students (or > here) at the beginning of the month: > http://archives.postgresql.org/pgsql-students/2012-04/msg00002.php > but the next time I heard about the subject was the acceptance > notice. That's largely my fault. Aakash's original proposal was naively assuming that logical records were available in our xlog stream and wanted to work on tools to do something useful with that data. There was some confusion among the various core team members, committers, and other contributors who volunteered to review the proposals about whether he was talking about analyzing the server log or xlog. We had a wide-ranging discussion on the freenode #postgresql channel about it, and brainstormed to the current idea -- at which point Aakash had about a week and a half to rewrite his proposal along the lines discussed on irc. A week after that the acceptance notices came out. If we had had any idea that 2Q was working on something along these lines, we would have coordinated or guided him in another direction. The post you cite from a couple months ago was way too vague to have even an inkling that it had anything to do with providing feeds of the logical interpretation of the WAL stream. I don't recall having heard the idea even mentioned before. The timing in regard to the release process seems to have had an effect on both groups here, since I suggested he read up on walsender and walreceiver and a few other areas, put together a Wiki page with the plan, and *then* describe the preliminary plan to the community (a month or so before starting to code), so that details could be hashed out in public before time was spent coding. Had there not been so many people heads-down trying to get the release out, I might have advised him to post a hand-wavy description of the effort before he got the Wiki page set up and reviewed the relevant code. As Josh points out, his first choice was to write utilities to *do* something with the logical xlog information, and he only switched over to working on *producing* it when he heard that no such thing was currently available. I think there's a lot of room to coordinate these efforts so that there *isn't* a waste of effort as long as communication is open enough. -Kevin
On 04/27/2012 06:48 PM, Kevin Grittner wrote: > That's largely my fault. It may not have come out that way, but I was trying more to point out the complexity of the story rather than assign blame. I find it hard to point at any obvious "don't do that again" target here. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com