Re: Timeline following for logical slots - Mailing list pgsql-hackers
| From | Craig Ringer |
|---|---|
| Subject | Re: Timeline following for logical slots |
| Date | |
| Msg-id | CAMsr+YGc9h1o4Xj2wBxpPCVqky=RSipySbrMev7HhV5Ojwo6mQ@mail.gmail.com Whole thread Raw |
| In response to | Re: Timeline following for logical slots (Robert Haas <robertmhaas@gmail.com>) |
| Responses |
Re: Timeline following for logical slots
(Robert Haas <robertmhaas@gmail.com>)
|
| List | pgsql-hackers |
On 6 April 2016 at 09:29, Robert Haas <robertmhaas@gmail.com> wrote:
> Right now if you're doing any kind of logical deocoding from a master server
> that fails over to a standby the client just dies. The slot vanishes. You're
> stuffed. Gee, I hope you didn't need all that nice consistent ordering,
> because you're going to have to start from scratch and somehow reconcile the
> data in the new master with what you've already received ... and haven't.
Right, but right now you probably *aren't* do doing any kind of
logical decoding from a master server to a standby, because there's
squat in the core distribution that could make use of that capability.
So you never get as far as discovering this problem.
OK, I'm starting to understand where this is going.
pglogical and pglogical_output are irrelevant so long as they haven't yet landed in core. There is no logical replication worth considering, therefore anything that improves logical replication is not worth the effort because there's really nothing to improve. Why bother with enhancements like timeline following when nothing uses it.
Right?
That's pretty much what I'm hearing from a number of others too. Obviously I don't wholly agree, but I do see where you're coming from. Especially since pglogical is in a sort of twilight zone where it's neither a public, open project with a proper website, public SCM, etc that encourages external contribution, nor is it strictly and only a submission to core.
It's not like, say, PostGIS, in that it's always been aimed at core (or stated to be) and is something that can be realistically considered for core. It doesn't have much of a life as an independent extension, and at least from the "outside" it doesn't look like there's much energy going into giving it one, right? Yet it's also had limited time and effort put into getting it in core in this release cycle.
While I don't like that, I have to say I understand it.
First, it wasn't until 9.3 that physical standbys could follow
timeline switches, but that doesn't mean that streaming replication
was useless in 9.0 - 9.2
I don't think there's much of a parallel there. It's closer to being like having _cascading_ replication without the ability to follow timeline switches via WAL archive replay or streaming.
Second, I'm not sure whether it was a good design decision to make
logical slots a special kind of object that sit off to the side,
neither configuration (like postgresql.conf) nor WAL-protected data
(like pg_clog and the data files themselves), but it was certainly a
very deliberate decision. I sort of expected them to be WAL-logged,
but Andres argued (not unconvincingly) that we'd want to have slots on
standbys, and making them WAL-logged would preclude that.
Yeah. I understand the reasons for that decision. Per an earlier reply I think we can avoid making them WAL-logged so they can be used on standbys and still achieve usable failover support on physical replicas.
It's more like unlogged tables, where a deliberate
design decision to lose data was made in order to meet some other
goal.
Yeah ... but it's like unlogged tables without having the option of logged tables.
Nice while it lasts but you're in bad trouble when it goes wrong.
(The parallel isn't perfect of course, since unlogged tables aren't crash safe wheras slots are, they just aren't replicated).
If you retype a column from text to integer,
you probably aren't storing anything in it other than integers, in
which case it is not necessarily the case that you are locked into
applying that change at a particular point in the change stream.
... if you're replicating in text format, not send/recv format, yeah. You get away with it in that particular case. But that *only* works where you're using text format and the types are convertable via text representation.
It's also just one example. Adding a NOT NULL column bites you, hard, for example.
Asking users to do this themselves is incredibly fragile and requires much more knowledge of PostgreSQL innards than I think is realistic. It's part of why Slony-I is difficult too.
Sure, you can avoid having DDL replication if you're really, really careful and know exactly what you're doing. I don't think it's a pre-req for getting logical replication into core, but I do think it's a pretty strong requirement for taking the use of logical replication for HA and failover seriously for general use.
To be honest, I was shocked that pglogical and pglogical_output didn't
go into this release. I assumed that you and other folks at
2ndQuadrant were going to make a big push to get that done. I did
take a brief look at one of them - pglogical, I think - a week or two
ago but there were unaddressed review comments that had been pending
for months and there were a lot of fairly obvious things that needed
to be done before it could be seriously considered as a core
submission. Like, for example, rewriting the documentation heavily
and making it look like the rest of our docs, and putting it in SGML
format. The code seemed to need quite a bit of cleanup, too.
Yes, I agree, and I think those comments are largely fair. Both Petr and I had more limited time to work on them than either of us had expected, so timely follow-up was a challenge. We also got them ready for initial submission later in the cycle than we'd hoped. That said, there was also very little review response or interest for a long time after they were published... and by the time there was I think both Petr and I were bogged a bit in other project work.
In retrospect I think the separate submission of the output plugin was a mistake too, at least when called pglogical_output; in part because of the name it was completely ignored as meaningless without the downstream and written off as useful for nothing except logical replication. So no momentum was built by the time the downstream was published.
One thing that'd help a lot from my PoV would be if PGXS supported building SGML docs. It's a major pain that maintaining a version as an extension (so people can use it for older versions) requires a different documentation format. But all in all that's relatively minor.
Now,
logical replication is a sufficiently important feature that if the
only way it's going to get into core is if I work on it myself, or get
other people at EnterpriseDB to do so, then I'll try to make that
happen. But I was assuming that that was your/2ndQuadrant's patch,
that you were going to get it in shape, and that me poking my nose
into it wasn't going to be particularly welcome. Maybe I've misread
the whole dynamic here.
I think getting more people involved is useful in a number of ways. Not just time and resources, but also getting more agreement on the direction things should be going in and making sure that more people have a direct hand in the core and are happy with it. Working from a common repo where interested people can push directly would be an interesting start.
I'll need to discuss this inside 2ndQ and follow up as there are things involved that I can't make determinations for myself.
> Do you want to get a logical replication system into core that doesn't work
> properly with lots of the other features in PostgreSQL? That's historically
> not how we've done things here, and sometimes massive amounts of work have
> been required to make new feature X work with obscure/awkward existing
> feature Y. Inheritance comes strongly to mind as an exciting challenge for
> many new features to support properly. Arguments are usually raised that
> reference things like the mess created by MySQL's storage engines and how
> we're better off making everything work right once and for all.
So, let me be clear, here. You have every right to decide which
feature you want to work on, and if, at the current time, failover
slots is that thing and if, at the current time, in-core logical
replication solution is not that thing, then that is entirely up to
you.
It's not entirely up to me personally, but I take your point.
My own priorities _have_ been influenced by how many people told me not to bother with getting logical rep in core since it can just live as an extension, though. I thought it reasonable to focus on things that can't be done as an extension since pglogical looked like it had little hope of landing in 9.6. Now I'm being told by others that it may as well not exist until it's in core. That's ... a tad frustrating. But that's also part of working in a community of individuals with different views and opinions.
Of course, I'm also biased to see pglogical as a going real-world concern which is relevant whether in or out of tree, so to me it seems obviously that improvements in core to make logical rep using it work better are worthwhile. If the general viewpoint of the wider community is "not in core, doesn't matter" then others will see it very differently.
FWIW, I'm also constantly keeping in mind that while I really, really want pglogical in core, there'll certainly be a backport as an extension for 9.4/9.5 and now 9.6. Given the massive lead times involved in getting anything into user hands I've really been hoping to get some of those foundations into 9.6 so that the backport from 9.7+ works better on 9.6. I suspect this either wasn't obvious or that others here don't see that as interesting.
That being said, if we get a logical replication system into core that
doesn't do DDL, doesn't do multi-master, doesn't know squat about
sequences, and rolls over and dies if a timeline switch happens, I
would consider that a huge step forward and I think a lot of other
people would, too.
I can understand that.
I think it's pretty wrong for it to die on timeline switch. I also think we'd really need *something* in place for sequences (even Londiste-style sync-with-fudge-factor), but the rest is tolerable as far as I'm concerned. I absolutely agree that multi-master isn't a requirement, and I think a "queue some DDL for me" function is ugly but an acceptable workaround for now. That's part of why pglogical's boundaries were set where they were when separating it from BDR.
We have a long history of building features
incrementally. Parallel query in 9.6 doesn't parallelize every query
that can be parallelized
Right. But it doesn't just arbitrarily break on what it doesn't understand. That's the difference here.
But
plenty of people also make the opposite mistake, of thinking that they
have to fix everything at once in order to have anything worthwhile,
and that's just as much of a trap as the other thing.
Yeah, that's fair. Personally I think I pushed to cut pglogical to the bone in terms of the minimum viable feature set that was targeted for it, though. It goes very, very far from solving everything, and intentionally ignores hard problems like DDL.
Even with lots
of limitations, built-in logical replication could still be good
enough to be used long enough to manage a major version upgrade
... with some form of sequence handling added, yes. Not necessarily nice sequence decoding, it can be worked around.
Well, as noted above, I think the main thing we need to figure out is
who is going to do the work. The main thing blocking other people
from working on it is the belief that we are just waiting for you or
someone else at 2ndQuadrant to land the necessary patches, but if you
aren't really working on that in a focused way, then somebody else can
step up.
It's a big job, and there are lots of differing viewpoints on it.
It does seem like having 2ndQ submit a patch, wait for feedback, be told "oh, don't do it that way, come back with <something else>", then wait for us to respond and address the issues isn't working. Certainly isn't going to get us there in a timely manner.
I want to explore some options here, but I have to talk to internal team members. Sorry for the handwavyness of that.
pgsql-hackers by date: