Re: pglogical_output - a general purpose logical decoding output plugin - Mailing list pgsql-hackers

From Craig Ringer
Subject Re: pglogical_output - a general purpose logical decoding output plugin
Date
Msg-id CAMsr+YF8mu63hnhA6uFes=Y7aAjNU8g0o2g2PuUyYR+T7VVXQQ@mail.gmail.com
Whole thread Raw
In response to pglogical_output - a general purpose logical decoding output plugin  (Craig Ringer <craig@2ndquadrant.com>)
Responses Re: pglogical_output - a general purpose logical decoding output plugin  (Tomasz Rybak <tomasz.rybak@post.pl>)
List pgsql-hackers
Hi all

Here's an updated pglogical_output patch.

Selected changes since v1:

    - add json protocol output support
    - fix encoding comparisons to use parsed encoding not string name
    - import protocol documentation
    - significantly expand pg_regress tests
    - move pglogical_output_plhooks to a top-level contrib
    - remove 9.4 compatibility


Notably, it now has support for a basic json-format text-based protocol as well as the main binary protocol. Mostly for debugging, testing and diagnostics.

I've used the newly added json support in pglogical_output to significantly expand the pg_regress tests, since the json mode can support text-mode decoding in the SQL interface to logical decoding, where the lsn, xid, etc can be skipped.

I've removed the Python based test framework. Test coverage in-tree is reduced as a result, with no coverage of the details of the binary protocol, no coverage of walsender based use, etc. I'll have a look at whether it'll be practical to convert the tests to Perl code driving pg_recvlogical as a co-process but don't think evaluation should wait on this.

I've also removed the 9.4 backward compatibility code from the version submitted for 9.6.

Docs conversion to SGML is still pending/WIP.

I've been unable to find a reasonable way to send the startup message before raising an ERROR when there's an issue with parameter/protocol negotiation. Advice/suggestions appreciated. The main issue is that the setup callback can't write to the protocol stream since in walsender mode the walsender isn't ready for writes yet. (Otherwise we could just write a record with InvalidXLogRecPtr, etc). Delaying the startup msg to the first begin callback works, as is done now. But delaying an error would involve allowing the begin callback to complete, then ERRORing at the *next* callback, which could be anything. Very ugly for what should be an uncommon case. So for now ERRORs are immediate, even though that makes negotiation much more difficult. Ideas welcomed.

(Cc'd a few people who expressed interest)

Attachment

pgsql-hackers by date:

Previous
From: Robert Haas
Date:
Subject: Re: eXtensible Transaction Manager API
Next
From: Fabien COELHO
Date:
Subject: Re: checkpointer continuous flushing