Re: logical changeset generation v3 - Mailing list pgsql-hackers

From Michael Paquier
Subject Re: logical changeset generation v3
Date
Msg-id CAB7nPqSyXsLavmNga4sZPkSdts-do_rfs1j6VTUjVuzeXjTXuQ@mail.gmail.com
Whole thread Raw
In response to Re: logical changeset generation v3  (Andres Freund <andres@2ndquadrant.com>)
Responses Re: logical changeset generation v3  (Andres Freund <andres@2ndquadrant.com>)
List pgsql-hackers


On Mon, Nov 19, 2012 at 5:50 PM, Andres Freund <andres@2ndquadrant.com> wrote:
Hi Michael,


On 2012-11-19 16:28:55 +0900, Michael Paquier wrote:
> I have been able to fetch your code (thanks Andrea!) and some it. For the
> time being I am spending some time reading the code and understanding the
> whole set of features you are trying to implement inside core, even if I
> got some background from what you presented at PGCon and from the hackers
> ML.

Cool.

> Btw, as a first approach, I tried to run the logical log receiver plugged
> on a postgres server, and I am not able to make it work.

> Well, I am using settings similar to yours.
> # Run master
> rm -r ~/bin/pgsql/master/
> initdb -D ~/bin/pgsql/master/
> echo "local replication $USER trust" >> ~/bin/pgsql/master/pg_hba.conf
> postgres -D ~/bin/pgsql/master \
>           -c wal_level=logical \
>           -c max_wal_senders=10 \
>           -c max_logical_slots=10 \
>           -c wal_keep_segments=100 \
>           -c log_line_prefix="[%p %x] "
> # Logical log receiver
> pg_receivellog -f $HOME/output.txt -d postgres -v
>
> After launching some SQLs, the logical receiver is stuck just after sending
> INIT_LOGICAL_REPLICATION, please see bt of process waiting:

Its waiting till it sees initial an initial xl_running_xacts record. The
easiest way to do that is manually issue a checkpoint. Sorry, should
have included that in the description.
Otherwise you can wait till the next routine checkpoint comes arround...

I plan to cause more xl_running_xacts record to be logged in the
future. I think the timing of those currently is non-optimal, you have
the same problem as here in normal streaming replication as well :(

> I am just looking at this patch and will provide some comments.
> By the way, you forgot the installation part of pg_receivellog, please see
> patch attached.

That actually was somewhat intended, I thought people wouldn't like the
name and I didn't want a binary that's going to be replaced anyway lying
around ;)
OK no problem. For sure this is going to happen, I was wondering myself if it could be possible to merge pg_receivexlog and pg_receivellog into a single utility with multiple modes :)

Btw, here are some extra comments based on my progress, hope it will be useful for other people playing around with your patches.
1) Necessary to install the contrib module test_decoding on server side or the test case will not work.
2) Obtention of the following logs on server:
LOG:  forced to assume catalog changes for xid 1370 because it was running to early
WARNING:  ABORT 1370
Actually I saw that there are many warnings like this.
3) Assertion failure while running pgbench, I was  just curious to see how it reacted when logical replication was put under a little bit of load.
TRAP: FailedAssertion("!(((xid) >= ((TransactionId) 3)) && ((snapstate->xmin_running) >= ((TransactionId) 3)))", File: "snapbuild.c", Line: 877)
=> pgbench -i postgres; pgbench -T 500 -c 8 postgres
4) Mentionned by Andres above, but logical replication begins only there is a xl_running_xacts record. I just enforced a checkpoint manually.

With all those things done, I have been able to set up the system, for example those queries:
postgres=# create table ac (a int);
CREATE TABLE
postgres=# insert into ac values (1);
INSERT 0 1
created the expected output:
BEGIN 32135
COMMIT 32135
BEGIN 32136
table "ac": INSERT: a[int4]:1
COMMIT 32136

Now it is time to dig into the code...
--
Michael Paquier
http://michael.otacoo.com

pgsql-hackers by date:

Previous
From: Peter Eisentraut
Date:
Subject: Re: MySQL search query is not executing in Postgres DB
Next
From: Steve Singer
Date:
Subject: Re: logical changeset generation v3 - Source for Slony