Thread: Re: Re: xReader, double-effort (was: Temporary tables under hot standby)

Simon Riggs  wrote:
> Though it isn't a given that logical change records (LCRs) will
> require more bandwidth than physical WAL.
It would help if you could share a little more of what you have in
mind.  It's not yet clear to me whether you're talking about adding
logical records to the WAL stream, replacing page-oriented records in
the WAL stream with logical records, or leaving the WAL stream alone
and generating a second, new stream of logical data.
> WAL contains full page images, index changes and other information
> that would be absent from the LCR stream.
Other than full page images, which could be eliminated in other ways
before streaming the WAL, what is in the WAL file that would not be
needed to maintain an exact replica of the cluster, suitable for
quick deployment in place of a lost cluster for disaster recovery? Or
do you foresee eliminating some of the current page-image oriented
WAL records in favor of logical WAL records, with an expectation that
the logical records will be smaller?
> measurements and detailed analysis is required to justify how to
> proceed. Which is what is in progress now.
Could you clarify?  Are you referring to this discussion or some
effort at 2Q that hasn't yet been shared with the community?
-Kevin

On Sun, Apr 29, 2012 at 7:36 PM, Kevin Grittner
<Kevin.Grittner@wicourts.gov> wrote:
> Simon Riggs  wrote:
>
>> Though it isn't a given that logical change records (LCRs) will
>> require more bandwidth than physical WAL.
>
> It would help if you could share a little more of what you have in
> mind.  It's not yet clear to me whether you're talking about adding
> logical records to the WAL stream, replacing page-oriented records in
> the WAL stream with logical records, or leaving the WAL stream alone
> and generating a second, new stream of logical data.
>
>> WAL contains full page images, index changes and other information
>> that would be absent from the LCR stream.
>
> Other than full page images, which could be eliminated in other ways
> before streaming the WAL, what is in the WAL file that would not be
> needed to maintain an exact replica of the cluster, suitable for
> quick deployment in place of a lost cluster for disaster recovery? Or
> do you foresee eliminating some of the current page-image oriented
> WAL records in favor of logical WAL records, with an expectation that
> the logical records will be smaller?
>
>> measurements and detailed analysis is required to justify how to
>> proceed. Which is what is in progress now.
>
> Could you clarify?  Are you referring to this discussion or some
> effort at 2Q that hasn't yet been shared with the community?

These are early days yet and I've been deliberately vague on the
format and content of the LCRs. That is to allow the idea that we have
choices to bed in a little, so we can discuss them. The one thing that
does stand clear in my mind at this stage is that the physical
transport of LCRs should be the current streaming infrastructure. So
the different aspects of the design are at differing stages of
certainty (in my mind).

Much of today has been spent on a presentation explaining the thought
processes and options available to us. Deciding between those options
needs to be fact based rather than just a matter of opinion and I want
to ensure we make the right choices, whatever they are.

I've mentioned a few times already that we're working on prototypes
all of which will be shared with the community. R&D, in that order.

--
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services