Thread: BDR Error restarted

Hi Guys.

First of all, BDR is cool, should have tried it earlier.

Environment: CentOS 6.5, PostgreSQL 9.4.0 with BDR from yum repository

Done the PostgreSQL 9.4 with BDR setup successfully by following the User
Guide and Admin Doc, but got a issue when tried to do postgresql service
restart on any of the cluster nodes, it always lost items in the relation
"pg_stat_replication"  after the service restarted, means lost replication
connections, like normally we have 26 items(13 databases and 3 servers in
cluster, so 13x2 items in pg_stat_replication on each node, yes, lots dbs on
one cluster), but after the restart, got some random count of item in the
relation, like 20, or 16 or 4, but not 26 as expected, i.e. not all the
replication connections running well. And the logs showing replication is
"wait until the node has caught up", but they never catch up again.

BDR Settings(replaced the real db name here):
1. on node 01, the replication src one:

2. on node 02, mostly like 03


Error logs:

Detail on the pgsql relation said above:






Thanks for any help/suggestion on this!
-Dean



--
View this message in context: http://postgresql.nabble.com/BDR-Error-restarted-tp5833139.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

On 01/07/2015 11:28 AM, deans wrote:
> Hi Guys.
>
> First of all, BDR is cool, should have tried it earlier.
>
> Environment: CentOS 6.5, PostgreSQL 9.4.0 with BDR from yum repository
>
> Done the PostgreSQL 9.4 with BDR setup successfully by following the User
> Guide and Admin Doc, but got a issue when tried to do postgresql service
> restart on any of the cluster nodes, it always lost items in the relation
> "pg_stat_replication"  after the service restarted

It'd be weird if it didn't. That's not a table, it's a view over a
function that reports server status information.

It's like pg_stat_activity for replication.

How exactly do you restart the node?

> means lost replication
> connections, like normally we have 26 items(13 databases and 3 servers in
> cluster, so 13x2 items in pg_stat_replication on each node, yes, lots dbs on
> one cluster), but after the restart, got some random count of item in the
> relation, like 20, or 16 or 4, but not 26 as expected, i.e. not all the
> replication connections running well.

That's normal, in that it can take time for peers to notice and
re-connect. This is true whether it's BDR, other logical replication, or
conventional streaming replication.

> And the logs showing replication is
> "wait until the node has caught up", but they never catch up again.

Please show the full log line, not just an excerpt, along with the lines
around it.

If possible run with log_error_verbosity=verbose and show the extra
line(s) printed after each log line too, please.

> BDR Settings(replaced the real db name here):
> 1. on node 01, the replication src one:
>
> 2. on node 02, mostly like 03

You forgot these.

> Error logs:

and these.

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

---

View this message in context: Re: BDR Error restarted
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

Hi Craig, any insight on this issue? :)

Thanks,
Dean



--
View this message in context: http://postgresql.nabble.com/BDR-Error-restarted-tp5833139p5833722.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

Hi Dean

Did you manage do get any help. I am having the exact issue at the moment.





--
View this message in context: http://postgresql.nabble.com/BDR-Error-restarted-tp5833139p5834722.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

:( not yet, still having the issue.



--
View this message in context: http://postgresql.nabble.com/BDR-Error-restarted-tp5833139p5834821.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

Hi,

On 2015-01-13 02:43:26 -0700, deans wrote:
> Hi Craig, any insight on this issue? :)

You've, afaics, not posted your configuration. Could you do so?

Greetings,

Andres Freund

--
 Andres Freund                       http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Hi Andres,

Any other conf should I list in addition to the bdr settings below?

BDR Settings(replaced the real db name here):

1. on node 01, the replication src one:
# Generic settings required for BDR
max_replication_slots = 60
max_wal_senders = 60
wal_level = 'logical'
track_commit_timestamp = on
shared_preload_libraries = 'bdr'
max_worker_processes = 100

# These aren't required, but are useful for diagnosing problems
#log_error_verbosity = verbose
#log_min_messages = debug1
#log_line_prefix = 'd=%d p=%p a=%a%q '

# Useful options for playing with conflicts
#bdr.default_apply_delay=2000   # milliseconds
#bdr.log_conflicts_to_table=on

bdr.connections =

'bdr02db1,bdr02db2,bdr02db3,bdr02db4,bdr02db5,bdr02db6,bdr02db7,bdr02db8,bdr02db9,bdr02db10,bdr02db11,bdr02db12,bdr02db13,bdr03db1,bdr03db2,bdr03db3,bdr03db4,bdr03db5,bdr03db6,bdr03db7,bdr03db8,bdr03db9,bdr03db10,bdr03db11,bdr03db12,bdr03db13'

bdr.bdr02db1_dsn = 'dbname=db1 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db2_dsn = 'dbname=db2 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db3_dsn = 'dbname=db3 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db4_dsn = 'dbname=db4 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db5_dsn = 'dbname=db5 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db6_dsn = 'dbname=db6 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db7_dsn = 'dbname=db7 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db8_dsn = 'dbname=db8 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db8_dsn = 'dbname=db9 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db10_dsn = 'dbname=db10 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db11_dsn = 'dbname=db11 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db12_dsn = 'dbname=db12 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr02db13_dsn = 'dbname=db13 host=pgsql-bdr-cluster-02.ol.xxx.net
user=postgres'
bdr.bdr03db1_dsn = 'dbname=db1 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db2_dsn = 'dbname=db2 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db3_dsn = 'dbname=db3 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db4_dsn = 'dbname=db4 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db5_dsn = 'dbname=db5 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db6_dsn = 'dbname=db6 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db7_dsn = 'dbname=db7 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db8_dsn = 'dbname=db8 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db9_dsn = 'dbname=db9 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db10_dsn = 'dbname=db10 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db11_dsn = 'dbname=db11 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db12_dsn = 'dbname=db12 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db13_dsn = 'dbname=db13 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'

2. on node 02
# Generic settings required for BDR
max_replication_slots = 60
max_wal_senders = 60
wal_level = 'logical'
track_commit_timestamp = on
shared_preload_libraries = 'bdr'
max_worker_processes = 100

# These aren't required, but are useful for diagnosing problems
#log_error_verbosity = verbose
#log_min_messages = debug1
#log_line_prefix = 'd=%d p=%p a=%a%q '

# Useful options for playing with conflicts
#bdr.default_apply_delay=2000   # milliseconds
#bdr.log_conflicts_to_table=on

bdr.connections =

'bdr01db1,bdr01db2,bdr01db3,bdr01db4,bdr01db5,bdr01db6,bdr01db7,bdr01db8,bdr01db9,bdr01db10,bdr01db11,bdr01db12,bdr01db13,bdr03db1,bdr03db2,bdr03db3,bdr03db4,bdr03db5,bdr03db6,bdr03db7,bdr03db8,bdr03db9,bdr03db10,bdr03db11,bdr03db12,bdr03db13'

bdr.bdr01db1_dsn = 'dbname=db1 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db2_dsn = 'dbname=db2 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db3_dsn = 'dbname=db3 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db4_dsn = 'dbname=db4 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db5_dsn = 'dbname=db5 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db6_dsn = 'dbname=db6 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db7_dsn = 'dbname=db7 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db8_dsn = 'dbname=db8 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db8_dsn = 'dbname=db9 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db10_dsn = 'dbname=db10 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db11_dsn = 'dbname=db11 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db12_dsn = 'dbname=db12 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db13_dsn = 'dbname=db13 host=pgsql-bdr-cluster-01.ol.xxx.net
user=postgres'
bdr.bdr01db1_dsn = 'dbname=db1 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db2_dsn = 'dbname=db2 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db3_dsn = 'dbname=db3 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db4_dsn = 'dbname=db4 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db5_dsn = 'dbname=db5 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db6_dsn = 'dbname=db6 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db7_dsn = 'dbname=db7 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db8_dsn = 'dbname=db8 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db9_dsn = 'dbname=db9 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db10_dsn = 'dbname=db10 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db11_dsn = 'dbname=db11 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db12_dsn = 'dbname=db12 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'
bdr.bdr03db13_dsn = 'dbname=db13 host=pgsql-bdr-cluster-03.ol.xxx.net
user=postgres'

bdr.bdr01db1_init_replica = on
bdr.bbdr01db1_replica_local_dsn = 'dbname=db1 user=postgres'
bdr.bdr01db2_init_replica = on
bdr.bbdr01db2_replica_local_dsn = 'dbname=db2 user=postgres'
bdr.bdr01db3_init_replica = on
bdr.bbdr01db3_replica_local_dsn = 'dbname=db3 user=postgres'
bdr.bdr01db4_init_replica = on
bdr.bbdr01db4_replica_local_dsn = 'dbname=db4 user=postgres'
bdr.bdr01db5_init_replica = on
bdr.bbdr01db5_replica_local_dsn = 'dbname=db5 user=postgres'
bdr.bdr01db6_init_replica = on
bdr.bbdr01db6_replica_local_dsn = 'dbname=db6 user=postgres'
bdr.bdr01db7_init_replica = on
bdr.bbdr01db7_replica_local_dsn = 'dbname=db7 user=postgres'
bdr.bdr01db8_init_replica = on
bdr.bbdr01db8_replica_local_dsn = 'dbname=db8 user=postgres'
bdr.bdr01db9_init_replica = on
bdr.bbdr01db9_replica_local_dsn = 'dbname=db9 user=postgres'
bdr.bdr01db10_init_replica = on
bdr.bbdr01db10_replica_local_dsn = 'dbname=db10 user=postgres'
bdr.bdr01db11_init_replica = on
bdr.bbdr01db11_replica_local_dsn = 'dbname=db11 user=postgres'
bdr.bdr01db12_init_replica = on
bdr.bbdr01db12_replica_local_dsn = 'dbname=db12 user=postgres'
bdr.bdr01db13_init_replica = on
bdr.bbdr01db13_replica_local_dsn = 'dbname=db13 user=postgres'



--
View this message in context: http://postgresql.nabble.com/BDR-Error-restarted-tp5833139p5834859.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

Hi Craig I have a similar issue with a rather small number of servers involved. Node 1: max_replication_slots = 4 max_wal_senders = 6 wal_level = 'logical' track_commit_timestamp = on shared_preload_libraries = 'bdr' max_worker_processes = 10 log_error_verbosity = verbose log_min_messages = debug1 log_line_prefix = 'd=%d p=%p a=%a%q ' bdr.default_apply_delay = 2000 bdr.log_conflicts_to_table = on bdr.connections = 'bdrnode3,bdrnode3' bdr.bdrnode2_dsn = 'dbname=db user=postgres host=host2 port=5599' bdr.bdrnode3_dsn = 'dbname=db user=postgres host=host3 port=5599' Node 2: max_replication_slots = 4 max_wal_senders = 6 wal_level = 'logical' track_commit_timestamp = on shared_preload_libraries = 'bdr' max_worker_processes = 10 log_error_verbosity = verbose log_min_messages = debug1 log_line_prefix = 'd=%d p=%p a=%a%q ' bdr.default_apply_delay = 2000 bdr.log_conflicts_to_table = on bdr.connections = 'bdrnode1,bdrnode3' bdr.bdrnode1_dsn = 'dbname=db user=postgres host=host1 port=5599' bdr.bdrnode3_dsn = 'dbname=db user=postgres host=host3 port=5599' bdr.bdrnode1_init_replica = on bdr.bdrnode1_replica_local_dsn = 'dbname=db user=postgres host=localhost port=5599' Node 3: max_replication_slots = 4 max_wal_senders = 6 wal_level = 'logical' track_commit_timestamp = on shared_preload_libraries = 'bdr' max_worker_processes = 10 log_error_verbosity = verbose log_min_messages = debug1 log_line_prefix = 'd=%d p=%p a=%a%q ' bdr.default_apply_delay = 2000 bdr.log_conflicts_to_table = on bdr.connections = 'bdrnode1,bdrnode2' bdr.bdrnode1_dsn = 'dbname=db user=postgres host=host1 port=5599' bdr.bdrnode2_dsn = 'dbname=db user=postgres host=host2 port=5599' bdr.bdrnode1_init_replica = on bdr.bdrnode1_replica_local_dsn = 'dbname=db user=postgres host=localhost port=5599' The above setup is an attempt for a "3-remote site simple Multi-Master Plex" configuration. Node1 and Node2 are replicating, but Node3 does not work, and the following data is is from Node3 logs: d=db_name p=3173 a=bdr (6106457559585933042,1,16398,): receive CONTEXT: slot "bdr_16396_6106457559585933042_1_16398__", output plugin "bdr", in the startup callback d=db_name p=3173 a=bdr (6106457559585933042,1,16398,): receive LOCATION: pg_decode_startup, bdr_output.c:450 d=db_name p=3173 a=bdr (6106457559585933042,1,16398,): receive ERROR: 55000: bdr output plugin: slot creation rejected, bdr.bdr_nodes entry for local node (sysid=6107134089288229725, timelineid=1, dboid=16396): status='i', bdr still starting up: applying initial dump of remote node d=db_name p=3173 a=bdr (6106457559585933042,1,16398,): receive HINT: Monitor pg_stat_activity and the logs, wait until the node has caught up d=db_name p=3173 a=bdr (6106457559585933042,1,16398,): receive CONTEXT: slot "bdr_16396_6106457559585933042_1_16398__", output plugin "bdr", in the startup callback d=db_name p=3173 a=bdr (6106457559585933042,1,16398,): receive LOCATION: bdr_ensure_node_ready, bdr_output.c:268 d=db_name p=3173 a=bdr (6106457559585933042,1,16398,): receive LOG: 08006: could not receive data from client: Connection reset by peer d=db_name p=3173 a=bdr (6106457559585933042,1,16398,): receive LOCATION: pq_recvbuf, pqcomm.c:869 d=db_name p=3173 a=bdr (6106457559585933042,1,16398,): receive DEBUG: 08003: unexpected EOF on client connection d=db_name p=3173 a=bdr (6106457559585933042,1,16398,): receive LOCATION: SocketBackend, postgres.c:353 d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive DEBUG: 00000: received replication command: IDENTIFY_SYSTEM d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive LOCATION: exec_replication_command, walsender.c:1292 d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive DEBUG: 00000: received replication command: START_REPLICATION SLOT "bdr_16396_6106458869483394081_1_16396__" LOGICAL 0/0 (pg_version '90400', pg_catversion '201408161', bdr_version '701', min_bdr_version '700', sizeof_int '4', sizeof_long '8', sizeof_datum '8', maxalign '8', float4_byval '1', float8_byval '1', integer_datetimes '1', bigendian '0', db_encoding 'UTF8') d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive LOCATION: exec_replication_command, walsender.c:1292 d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive DEBUG: 00000: bdr.bdr_conflict_handlers OID set to 16991 d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive CONTEXT: slot "bdr_16396_6106458869483394081_1_16396__", output plugin "bdr", in the startup callback d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive LOCATION: pg_decode_startup, bdr_output.c:450 d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive ERROR: 55000: bdr output plugin: slot creation rejected, bdr.bdr_nodes entry for local node (sysid=6107134089288229725, timelineid=1, dboid=16396): status='i', bdr still starting up: applying initial dump of remote node d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive HINT: Monitor pg_stat_activity and the logs, wait until the node has caught up d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive CONTEXT: slot "bdr_16396_6106458869483394081_1_16396__", output plugin "bdr", in the startup callback d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive LOCATION: bdr_ensure_node_ready, bdr_output.c:268 d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive LOG: 08006: could not receive data from client: Connection reset by peer d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive LOCATION: pq_recvbuf, pqcomm.c:869 d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive DEBUG: 08003: unexpected EOF on client connection d=db_name p=3175 a=bdr (6106458869483394081,1,16396,): receive LOCATION: SocketBackend, postgres.c:353 d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive DEBUG: 00000: received replication command: IDENTIFY_SYSTEM d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive LOCATION: exec_replication_command, walsender.c:1292 d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive DEBUG: 00000: received replication command: START_REPLICATION SLOT "bdr_16396_6106457559585933042_1_16398__" LOGICAL 0/0 (pg_version '90400', pg_catversion '201408161', bdr_version '701', min_bdr_version '700', sizeof_int '4', sizeof_long '8', sizeof_datum '8', maxalign '8', float4_byval '1', float8_byval '1', integer_datetimes '1', bigendian '0', db_encoding 'UTF8') d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive LOCATION: exec_replication_command, walsender.c:1292 d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive DEBUG: 00000: bdr.bdr_conflict_handlers OID set to 16991 d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive CONTEXT: slot "bdr_16396_6106457559585933042_1_16398__", output plugin "bdr", in the startup callback d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive LOCATION: pg_decode_startup, bdr_output.c:450 d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive ERROR: 55000: bdr output plugin: slot creation rejected, bdr.bdr_nodes entry for local node (sysid=6107134089288229725, timelineid=1, dboid=16396): status='i', bdr still starting up: applying initial dump of remote node d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive HINT: Monitor pg_stat_activity and the logs, wait until the node has caught up d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive CONTEXT: slot "bdr_16396_6106457559585933042_1_16398__", output plugin "bdr", in the startup callback d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive LOCATION: bdr_ensure_node_ready, bdr_output.c:268 d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive LOG: 08006: could not receive data from client: Connection reset by peer d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive LOCATION: pq_recvbuf, pqcomm.c:869 d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive DEBUG: 08003: unexpected EOF on client connection d=db_name p=3189 a=bdr (6106457559585933042,1,16398,): receive LOCATION: SocketBackend, postgres.c:353 d= p=3190 a=DEBUG: 00000: autovacuum: processing database "db_name" d= p=3190 a=LOCATION: AutoVacWorkerMain, autovacuum.c:1670 d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive DEBUG: 00000: received replication command: IDENTIFY_SYSTEM d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive LOCATION: exec_replication_command, walsender.c:1292 d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive DEBUG: 00000: received replication command: START_REPLICATION SLOT "bdr_16396_6106458869483394081_1_16396__" LOGICAL 0/0 (pg_version '90400', pg_catversion '201408161', bdr_version '701', min_bdr_version '700', sizeof_int '4', sizeof_long '8', sizeof_datum '8', maxalign '8', float4_byval '1', float8_byval '1', integer_datetimes '1', bigendian '0', db_encoding 'UTF8') d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive LOCATION: exec_replication_command, walsender.c:1292 d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive DEBUG: 00000: bdr.bdr_conflict_handlers OID set to 16991 d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive CONTEXT: slot "bdr_16396_6106458869483394081_1_16396__", output plugin "bdr", in the startup callback d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive LOCATION: pg_decode_startup, bdr_output.c:450 d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive ERROR: 55000: bdr output plugin: slot creation rejected, bdr.bdr_nodes entry for local node (sysid=6107134089288229725, timelineid=1, dboid=16396): status='i', bdr still starting up: applying initial dump of remote node d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive HINT: Monitor pg_stat_activity and the logs, wait until the node has caught up d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive CONTEXT: slot "bdr_16396_6106458869483394081_1_16396__", output plugin "bdr", in the startup callback d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive LOCATION: bdr_ensure_node_ready, bdr_output.c:268 d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive LOG: 08006: could not receive data from client: Connection reset by peer d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive LOCATION: pq_recvbuf, pqcomm.c:869 d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive DEBUG: 08003: unexpected EOF on client connection d=db_name p=3191 a=bdr (6106458869483394081,1,16396,): receive LOCATION: SocketBackend, postgres.c:353 d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive DEBUG: 00000: received replication command: IDENTIFY_SYSTEM d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive LOCATION: exec_replication_command, walsender.c:1292 d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive DEBUG: 00000: received replication command: START_REPLICATION SLOT "bdr_16396_6106457559585933042_1_16398__" LOGICAL 0/0 (pg_version '90400', pg_catversion '201408161', bdr_version '701', min_bdr_version '700', sizeof_int '4', sizeof_long '8', sizeof_datum '8', maxalign '8', float4_byval '1', float8_byval '1', integer_datetimes '1', bigendian '0', db_encoding 'UTF8') d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive LOCATION: exec_replication_command, walsender.c:1292 d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive DEBUG: 00000: bdr.bdr_conflict_handlers OID set to 16991 d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive CONTEXT: slot "bdr_16396_6106457559585933042_1_16398__", output plugin "bdr", in the startup callback d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive LOCATION: pg_decode_startup, bdr_output.c:450 d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive ERROR: 55000: bdr output plugin: slot creation rejected, bdr.bdr_nodes entry for local node (sysid=6107134089288229725, timelineid=1, dboid=16396): status='i', bdr still starting up: applying initial dump of remote node d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive HINT: Monitor pg_stat_activity and the logs, wait until the node has caught up d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive CONTEXT: slot "bdr_16396_6106457559585933042_1_16398__", output plugin "bdr", in the startup callback d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive LOCATION: bdr_ensure_node_ready, bdr_output.c:268 d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive LOG: 08006: could not receive data from client: Connection reset by peer d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive LOCATION: pq_recvbuf, pqcomm.c:869 d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive DEBUG: 08003: unexpected EOF on client connection d=db_name p=3199 a=bdr (6106457559585933042,1,16398,): receive LOCATION: SocketBackend, postgres.c:353 The persisting error is: 55000: bdr output plugin: slot creation rejected, bdr.bdr_nodes entry for local node (sysid=6107134089288229725, timelineid=1, dboid=16396): status='i', bdr still starting up: applying initial dump of remote node I hope I have provided enough detail and that I am posting at the right forum. Regards

---

View this message in context: Re: BDR Error restarted
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

Hi Chris

When running: git rev-parse --short HEAD

Output: a63674e

The source package I got from:

git clone git://git.postgresql.org/git/2ndquadrant_bdr.git
git clean  -d  -fx ""
git checkout bdr/0.7.1

This is the logs from node 1:

d= p=28495 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28495 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=28495 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28495 a=DEBUG:  00000: bdr (6106457559585933042,1,16398,): node3: apply
initialized on db_name
d= p=28495 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=28495 a=DEBUG:  00000: found valid replication identifier 4
d= p=28495 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=28495 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=28495 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=28495 a=ERROR:  XX000: data stream ended
d= p=28495 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=31417 a=LOG:  00000: worker process: bdr
(6106457559585933042,1,16398,): node3: apply (PID 28495) exited with exit
code 1
d= p=31417 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=31417 a=LOG:  00000: starting background worker process "bdr
(6106457559585933042,1,16398,): node3: apply"
d= p=31417 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=28515 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=28515 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28515 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=28515 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28515 a=DEBUG:  00000: bdr (6106457559585933042,1,16398,): node3: apply
initialized on db_name
d= p=28515 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=28515 a=DEBUG:  00000: found valid replication identifier 4
d= p=28515 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=28515 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=28515 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=28515 a=ERROR:  XX000: data stream ended
d= p=28515 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=31417 a=LOG:  00000: worker process: bdr
(6106457559585933042,1,16398,): node3: apply (PID 28515) exited with exit
code 1
d= p=31417 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=28519 a=DEBUG:  00000: autovacuum: processing database "postgres"
d= p=28519 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=31417 a=LOG:  00000: starting background worker process "bdr
(6106457559585933042,1,16398,): node3: apply"
d= p=31417 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=28520 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=28520 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28520 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=28520 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28520 a=DEBUG:  00000: bdr (6106457559585933042,1,16398,): node3: apply
initialized on db_name
d= p=28520 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=28520 a=DEBUG:  00000: found valid replication identifier 4
d= p=28520 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=28520 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=28520 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=28520 a=ERROR:  XX000: data stream ended
d= p=28520 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=31417 a=LOG:  00000: worker process: bdr
(6106457559585933042,1,16398,): node3: apply (PID 28520) exited with exit
code 1
d= p=31417 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=31417 a=LOG:  00000: starting background worker process "bdr
(6106457559585933042,1,16398,): node3: apply"
d= p=31417 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=28524 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=28524 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28524 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=28524 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28524 a=DEBUG:  00000: bdr (6106457559585933042,1,16398,): node3: apply
initialized on db_name
d= p=28524 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=28524 a=DEBUG:  00000: found valid replication identifier 4
d= p=28524 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=28524 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=28524 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=28524 a=ERROR:  XX000: data stream ended
d= p=28524 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=31417 a=LOG:  00000: worker process: bdr
(6106457559585933042,1,16398,): node3: apply (PID 28524) exited with exit
code 1
d= p=31417 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=31417 a=LOG:  00000: starting background worker process "bdr
(6106457559585933042,1,16398,): node3: apply"
d= p=31417 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=28550 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=28550 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28550 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=28550 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28550 a=DEBUG:  00000: bdr (6106457559585933042,1,16398,): node3: apply
initialized on db_name
d= p=28550 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=28550 a=DEBUG:  00000: found valid replication identifier 4
d= p=28550 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=28550 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=28550 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=28550 a=ERROR:  XX000: data stream ended
d= p=28550 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=31417 a=LOG:  00000: worker process: bdr
(6106457559585933042,1,16398,): node3: apply (PID 28550) exited with exit
code 1
d= p=31417 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=28566 a=DEBUG:  00000: autovacuum: processing database "template1"
d= p=28566 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=31417 a=LOG:  00000: starting background worker process "bdr
(6106457559585933042,1,16398,): node3: apply"
d= p=31417 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=28591 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=28591 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28591 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=28591 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28591 a=DEBUG:  00000: bdr (6106457559585933042,1,16398,): node3: apply
initialized on db_name
d= p=28591 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=28591 a=DEBUG:  00000: found valid replication identifier 4
d= p=28591 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=28591 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=28591 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=28591 a=ERROR:  XX000: data stream ended
d= p=28591 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=31417 a=LOG:  00000: worker process: bdr
(6106457559585933042,1,16398,): node3: apply (PID 28591) exited with exit
code 1
d= p=31417 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=31417 a=LOG:  00000: starting background worker process "bdr
(6106457559585933042,1,16398,): node3: apply"
d= p=31417 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=28595 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=28595 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28595 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=28595 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28595 a=DEBUG:  00000: bdr (6106457559585933042,1,16398,): node3: apply
initialized on db_name
d= p=28595 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=28595 a=DEBUG:  00000: found valid replication identifier 4
d= p=28595 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=28595 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=28595 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=28595 a=ERROR:  XX000: data stream ended
d= p=28595 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=31417 a=LOG:  00000: worker process: bdr
(6106457559585933042,1,16398,): node3: apply (PID 28595) exited with exit
code 1
d= p=31417 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=31417 a=LOG:  00000: starting background worker process "bdr
(6106457559585933042,1,16398,): node3: apply"
d= p=31417 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=28599 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=28599 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28599 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=28599 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=28599 a=DEBUG:  00000: bdr (6106457559585933042,1,16398,): node3: apply
initialized on db_name
d= p=28599 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=28599 a=DEBUG:  00000: found valid replication identifier 4
d= p=28599 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=28599 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=28599 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=28599 a=ERROR:  XX000: data stream ended
d= p=28599 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=31417 a=LOG:  00000: worker process: bdr
(6106457559585933042,1,16398,): node3: apply (PID 28599) exited with exit
code 1
d= p=31417 a=LOCATION:  LogChildExit, postmaster.c:3287


node2:

d= p=4813 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4813) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4823 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4823 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4823 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4823 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4823 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4823 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4823 a=DEBUG:  00000: found valid replication identifier 4
d= p=4823 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4823 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4823 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4823 a=ERROR:  XX000: data stream ended
d= p=4823 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4823) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=4827 a=DEBUG:  00000: autovacuum: processing database
"trustpay_null_production"
d= p=4827 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4850 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4850 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4850 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4850 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4850 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4850 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4850 a=DEBUG:  00000: found valid replication identifier 4
d= p=4850 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4850 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4850 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4850 a=ERROR:  XX000: data stream ended
d= p=4850 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4850) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4854 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4854 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4854 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4854 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4854 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4854 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4854 a=DEBUG:  00000: found valid replication identifier 4
d= p=4854 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4854 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4854 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4854 a=ERROR:  XX000: data stream ended
d= p=4854 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4854) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4868 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4868 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4868 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4868 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4868 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4868 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4868 a=DEBUG:  00000: found valid replication identifier 4
d= p=4868 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4868 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4868 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4868 a=ERROR:  XX000: data stream ended
d= p=4868 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4868) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=4876 a=DEBUG:  00000: autovacuum: processing database "db_name"
d= p=4876 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4880 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4880 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4880 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4880 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4880 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4880 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4880 a=DEBUG:  00000: found valid replication identifier 4
d= p=4880 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4880 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4880 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4880 a=ERROR:  XX000: data stream ended
d= p=4880 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4880) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4898 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4898 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4898 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4898 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4898 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4898 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4898 a=DEBUG:  00000: found valid replication identifier 4
d= p=4898 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4898 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4898 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4898 a=ERROR:  XX000: data stream ended
d= p=4898 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4898) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4907 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4907 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4907 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4907 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4907 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4907 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4907 a=DEBUG:  00000: found valid replication identifier 4
d= p=4907 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4907 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4907 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4907 a=ERROR:  XX000: data stream ended
d= p=4907 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4907) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=4912 a=DEBUG:  00000: autovacuum: processing database "postgres"
d= p=4912 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4935 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4935 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4935 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4935 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4935 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4935 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4935 a=DEBUG:  00000: found valid replication identifier 4
d= p=4935 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4935 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4935 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4935 a=ERROR:  XX000: data stream ended
d= p=4935 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4935) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4940 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4940 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4940 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4940 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4940 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4940 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4940 a=DEBUG:  00000: found valid replication identifier 4
d= p=4940 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4940 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4940 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4940 a=ERROR:  XX000: data stream ended
d= p=4940 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4940) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4954 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4954 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4954 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4954 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4954 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4954 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4954 a=DEBUG:  00000: found valid replication identifier 4
d= p=4954 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4954 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4954 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4954 a=ERROR:  XX000: data stream ended
d= p=4954 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4954) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6707 a=DEBUG:  00000: started 0 votes
d= p=6707 a=LOCATION:  bdr_sequencer_vote, bdr_seq.c:715
d= p=6707 a=DEBUG:  00000: tallied 0 elections
d= p=6707 a=LOCATION:  bdr_sequencer_tally, bdr_seq.c:823
d= p=6707 a=DEBUG:  00000: started 0 elections
d= p=6707 a=LOCATION:  bdr_sequencer_start_elections, bdr_seq.c:766
d= p=4964 a=DEBUG:  00000: autovacuum: processing database "template1"
d= p=4964 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4965 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4965 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4965 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4965 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4965 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4965 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4965 a=DEBUG:  00000: found valid replication identifier 4
d= p=4965 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4965 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4965 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4965 a=ERROR:  XX000: data stream ended
d= p=4965 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4965) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4970 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4970 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4970 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4970 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4970 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4970 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4970 a=DEBUG:  00000: found valid replication identifier 4
d= p=4970 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4970 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4970 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4970 a=ERROR:  XX000: data stream ended
d= p=4970 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4970) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=4980 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=4980 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4980 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=4980 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=4980 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=4980 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=4980 a=DEBUG:  00000: found valid replication identifier 4
d= p=4980 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=4980 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=4980 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=4980 a=ERROR:  XX000: data stream ended
d= p=4980 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 4980) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=4984 a=DEBUG:  00000: autovacuum: processing database
"trustpay_null_production"
d= p=4984 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5007 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5007 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5007 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5007 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5007 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5007 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5007 a=DEBUG:  00000: found valid replication identifier 4
d= p=5007 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5007 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5007 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5007 a=ERROR:  XX000: data stream ended
d= p=5007 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5007) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5011 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5011 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5011 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5011 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5011 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5011 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5011 a=DEBUG:  00000: found valid replication identifier 4
d= p=5011 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5011 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5011 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5011 a=ERROR:  XX000: data stream ended
d= p=5011 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5011) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5025 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5025 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5025 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5025 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5025 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5025 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5025 a=DEBUG:  00000: found valid replication identifier 4
d= p=5025 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5025 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5025 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5025 a=ERROR:  XX000: data stream ended
d= p=5025 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5025) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=5033 a=DEBUG:  00000: autovacuum: processing database "db_name"
d= p=5033 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5037 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5037 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5037 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5037 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5037 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5037 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5037 a=DEBUG:  00000: found valid replication identifier 4
d= p=5037 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5037 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5037 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5037 a=ERROR:  XX000: data stream ended
d= p=5037 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5037) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5083 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5083 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5083 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5083 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5083 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5083 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5083 a=DEBUG:  00000: found valid replication identifier 4
d= p=5083 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5083 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5083 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5083 a=ERROR:  XX000: data stream ended
d= p=5083 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5083) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5092 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5092 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5092 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5092 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5092 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5092 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5092 a=DEBUG:  00000: found valid replication identifier 4
d= p=5092 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5092 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5092 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5092 a=ERROR:  XX000: data stream ended
d= p=5092 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5092) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=5097 a=DEBUG:  00000: autovacuum: processing database "postgres"
d= p=5097 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5120 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5120 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5120 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5120 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5120 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5120 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5120 a=DEBUG:  00000: found valid replication identifier 4
d= p=5120 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5120 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5120 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5120 a=ERROR:  XX000: data stream ended
d= p=5120 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5120) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5125 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5125 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5125 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5125 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5125 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5125 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5125 a=DEBUG:  00000: found valid replication identifier 4
d= p=5125 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5125 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5125 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5125 a=ERROR:  XX000: data stream ended
d= p=5125 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5125) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5139 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5139 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5139 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5139 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5139 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5139 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5139 a=DEBUG:  00000: found valid replication identifier 4
d= p=5139 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5139 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5139 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5139 a=ERROR:  XX000: data stream ended
d= p=5139 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5139) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=5149 a=DEBUG:  00000: autovacuum: processing database "template1"
d= p=5149 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5150 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5150 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5150 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5150 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5150 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5150 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5150 a=DEBUG:  00000: found valid replication identifier 4
d= p=5150 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5150 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5150 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5150 a=ERROR:  XX000: data stream ended
d= p=5150 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5150) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5155 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5155 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5155 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5155 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5155 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5155 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5155 a=DEBUG:  00000: found valid replication identifier 4
d= p=5155 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5155 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5155 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5155 a=ERROR:  XX000: data stream ended
d= p=5155 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5155) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5165 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5165 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5165 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5165 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5165 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5165 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5165 a=DEBUG:  00000: found valid replication identifier 4
d= p=5165 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5165 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5165 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5165 a=ERROR:  XX000: data stream ended
d= p=5165 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5165) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=5169 a=DEBUG:  00000: autovacuum: processing database
"trustpay_null_production"
d= p=5169 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5192 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5192 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5192 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5192 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5192 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5192 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5192 a=DEBUG:  00000: found valid replication identifier 4
d= p=5192 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5192 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5192 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5192 a=ERROR:  XX000: data stream ended
d= p=5192 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5192) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5196 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5196 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5196 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5196 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5196 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5196 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5196 a=DEBUG:  00000: found valid replication identifier 4
d= p=5196 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5196 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5196 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5196 a=ERROR:  XX000: data stream ended
d= p=5196 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5196) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5210 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5210 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5210 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5210 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5210 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5210 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5210 a=DEBUG:  00000: found valid replication identifier 4
d= p=5210 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5210 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5210 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5210 a=ERROR:  XX000: data stream ended
d= p=5210 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5210) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=5218 a=DEBUG:  00000: autovacuum: processing database "db_name"
d= p=5218 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5229 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5229 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5229 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5229 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5229 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5229 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5229 a=DEBUG:  00000: found valid replication identifier 4
d= p=5229 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5229 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5229 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5229 a=ERROR:  XX000: data stream ended
d= p=5229 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5229) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5240 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5240 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5240 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5240 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5240 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5240 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5240 a=DEBUG:  00000: found valid replication identifier 4
d= p=5240 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5240 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5240 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5240 a=ERROR:  XX000: data stream ended
d= p=5240 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5240) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=6700 a=LOG:  00000: starting background worker process "bdr
(6106458869483394081,1,16396,): node3: apply"
d= p=6700 a=LOCATION:  do_start_bgworker, postmaster.c:5354
d= p=5249 a=NOTICE:  00000: version "1.0" of extension "btree_gist" is
already installed
d= p=5249 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5249 a=NOTICE:  00000: version "0.7.1" of extension "bdr" is already
installed
d= p=5249 a=LOCATION:  ExecAlterExtensionStmt, extension.c:2700
d= p=5249 a=DEBUG:  00000: bdr (6106458869483394081,1,16396,): node3: apply
initialized on db_name
d= p=5249 a=LOCATION:  bdr_apply_main, bdr.c:563
d= p=5249 a=DEBUG:  00000: found valid replication identifier 4
d= p=5249 a=LOCATION:  bdr_establish_connection_and_slot, bdr.c:502
d= p=5249 a=INFO:  00000: starting up replication from 4 at 0/0
d= p=5249 a=LOCATION:  bdr_apply_main, bdr.c:614
d= p=5249 a=ERROR:  XX000: data stream ended
d= p=5249 a=LOCATION:  bdr_apply_work, bdr_apply.c:2240
d= p=6700 a=LOG:  00000: worker process: bdr (6106458869483394081,1,16396,):
node3: apply (PID 5249) exited with exit code 1
d= p=6700 a=LOCATION:  LogChildExit, postmaster.c:3287
d= p=5254 a=DEBUG:  00000: autovacuum: processing database "postgres"
d= p=5254 a=LOCATION:  AutoVacWorkerMain, autovacuum.c:1670



--
View this message in context: http://postgresql.nabble.com/BDR-Error-restarted-tp5833139p5835190.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

Hi all,

I have a similar problem.

OS Red Hat Enterprise Linux Server release 6.6.

Postgresql packets were installed from repository as described in
https://wiki.postgresql.org/wiki/BDR_Packages

postgresql-bdr94-bdr-0.8.0beta1-1_2ndQuadrant.el6.x86_64
postgresql-bdr94-contrib-9.4.0_bdr1-1_2ndQuadrant.el6.x86_64
postgresql-bdr94-9.4.0_bdr1-1_2ndQuadrant.el6.x86_64
postgresql-bdr94-libs-9.4.0_bdr1-1_2ndQuadrant.el6.x86_64
postgresql-bdr94-server-9.4.0_bdr1-1_2ndQuadrant.el6.x86_64


There are two servers: 192.168.214.169 (main), 192.168.214.163

On each server run commands:
su - postgres -c "/usr/pgsql-9.4/bin/initdb
--pgdata=/var/lib/pgsql/9.4-bdr/data --auth='trust' --username=test_root"
su - postgres -c "/usr/pgsql-9.4/bin/pg_ctl -l
/var/lib/pgsql/9.4-bdr/data/postgresql-02.log -D
/var/lib/pgsql/9.4-bdr/data -o '-p 5432' -w start"
createdb -p 5432 -U test_root testdb

On the first server I've created a schema and a table in it.

Then configs were updated and postgresql servers were restarted.
iptables is switched off.


//postgrsql.conf [192.168.214.169]

listen_addresses = '192.168.214.169'
port=5432

max_replication_slots = 3
max_wal_senders = 4
wal_level = 'logical'
track_commit_timestamp = on
shared_preload_libraries = 'bdr'
max_worker_processes = 10

bdr.connections = 'second'
bdr.second_dsn = 'host=192.168.214.163 port=5432 dbname=testdb
user=test_root'

// pg_hba.conf [192.168.214.169]

host    all             test_root        192.168.214.163/32      trust
host    replication     test_root        192.168.214.163/32      trust
+ uncomment existing replication lines


//postgrsql.conf [192.168.214.163]

listen_addresses = '192.168.214.163'
port=5432

max_replication_slots = 3
max_wal_senders = 4
wal_level = 'logical'
track_commit_timestamp = on
shared_preload_libraries = 'bdr'
max_worker_processes = 10

bdr.connections = 'first'
bdr.first_dsn = 'host=192.168.214.169 port=5432 dbname=testdb
user=test_root'
bdr.first_init_replica = on
bdr.first_replica_local_dsn = 'port=5432 dbname=testdb user=test_root'

// pg_hba.conf [192.168.214.163]

host    all             test_root        192.168.214.169/32      trust
host    replication     test_root        192.168.214.169/32      trust
+ uncomment existing replication lines


Replication works only on that created table and only from the first to the
second server.
When I try to create one more table, the sql command 'CREATE TABLE ...'
hangs.

The followng errors are added each second in postgresql logs:

// 192.168.214.169

LOG:  starting background worker process "bdr
(6109689945671396351,1,16384,): second: apply"
NOTICE:  version "1.0" of extension "btree_gist" is already installed
CONTEXT:  slot "bdr_16384_6109689945671396351_1_16384__", output plugin
"bdr", in the startup callback
NOTICE:  version "0.8.0.6" of extension "bdr" is already installed
CONTEXT:  slot "bdr_16384_6109689945671396351_1_16384__", output plugin
"bdr", in the startup callback
ERROR:  data stream ended
LOG:  worker process: bdr (6109689945671396351,1,16384,): second: apply (PID
13549) exited with exit code 1

// 192.168.214.163

ERROR:  bdr output plugin: slot creation rejected, bdr.bdr_nodes entry for
local node (sysid=6109702913719211129, timelineid=1, dboid=16384):
status='i', bdr still starting up: applying initial dump of remote node
HINT:  Monitor pg_stat_activity and the logs, wait until the node has caught
up
CONTEXT:  slot "bdr_16384_6109689945671396351_1_16384__", output plugin
"bdr", in the startup callback
LOG:  could not receive data from client: Connection reset by peer

In attachment - debug logs of postgresql.

How to fix it?

first_postgres.log
<http://postgresql.nabble.com/file/n5836407/first_postgres.log>
second_postgres.log
<http://postgresql.nabble.com/file/n5836407/second_postgres.log>




--
View this message in context: http://postgresql.nabble.com/BDR-Error-restarted-tp5833139p5836407.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.