Thread: Recovery Standby Node
Hi,
My standby node is not getting sync with master node. I am performing following steps:1. $ rm -rf /opt/pg/data/9.4/*
2. $ pg_basebackup -h <master-node-ip> -p 1523 -U pg -D /opt/pg/data/9.4 -X stream -P
3. rm -rf /opt/pg/tmp/PGSQL.lock
4. pcs resource cleanup msPostgresql //msPostgresql is resource name
Following are the log:
pgsql(pgsql)[5679]: 2018/02/12_02:38:25 INFO: PostgreSQL is down
pgsql(pgsql)[5679]: 2018/02/12_02:38:26 INFO: PostgreSQL is down
Feb 12 02:38:27 [5562] zjedmsdp20v lrmd: warning: child_timeout_callback: pgsql_start_0 process (PID 5679) timed out
Feb 12 02:38:27 [5562] zjedmsdp20v lrmd: warning: operation_finished: pgsql_start_0:5679 - timed out after 60000ms
Feb 12 02:38:27 [5562] zjedmsdp20v lrmd: info: log_finished: finished - rsc:pgsql action:start call_id:11 pid:5679 exit-code:1 exec-time:60001ms queue-time:0ms
Feb 12 02:38:27 [5565] zjedmsdp20v crmd: info: services_os_action_execute: Managed pgsql_meta-data_0 process 6302 exited with rc=0
Feb 12 02:38:27 [5565] zjedmsdp20v crmd: error: process_lrm_event: Operation pgsql_start_0: Timed Out (node=zjedmsdp20v, call=11, timeout=60000ms)
Feb 12 02:38:27 [5560] zjedmsdp20v cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/12)
Feb 12 02:38:27 [5560] zjedmsdp20v cib: info: cib_perform_op: Diff: --- 0.122.31 2
Feb 12 02:38:27 [5560] zjedmsdp20v cib: info: cib_perform_op: Diff: +++ 0.122.32 (null)
Feb 12 02:38:27 [5560] zjedmsdp20v cib: info: cib_perform_op: + /cib: @num_updates=32
Feb 12 02:38:27 [5560] zjedmsdp20v cib: info: cib_perform_op: + /cib/status/node_state[@id='zjedmsdp20v']/lrm[@id='zjedmsdp20v']/lrm_resources/lrm_resource[@id='pgsql']/lrm
_rsc_op[@id='pgsql_last_0']: @operation_key=pgsql_start_0, @operation=start, @transition-key=15:7444:0:8c793138-d4e4-4780-88f2-2da6808c25ca, @transition-magic=2:1;15:7444:0:8c793138-d4e4-478
0-88f2-2da6808c25ca, @call-id=11, @rc-code=1, @op-status=2, @exec-time=60001
pgsql(pgsql)[5679]: 2018/02/12_02:38:25 INFO: PostgreSQL is down
pgsql(pgsql)[5679]: 2018/02/12_02:38:26 INFO: PostgreSQL is down
Feb 12 02:38:27 [5562] zjedmsdp20v lrmd: warning: child_timeout_callback: pgsql_start_0 process (PID 5679) timed out
Feb 12 02:38:27 [5562] zjedmsdp20v lrmd: warning: operation_finished: pgsql_start_0:5679 - timed out after 60000ms
Feb 12 02:38:27 [5562] zjedmsdp20v lrmd: info: log_finished: finished - rsc:pgsql action:start call_id:11 pid:5679 exit-code:1 exec-time:60001ms queue-time:0ms
Feb 12 02:38:27 [5565] zjedmsdp20v crmd: info: services_os_action_execute: Managed pgsql_meta-data_0 process 6302 exited with rc=0
Feb 12 02:38:27 [5565] zjedmsdp20v crmd: error: process_lrm_event: Operation pgsql_start_0: Timed Out (node=zjedmsdp20v, call=11, timeout=60000ms)
Feb 12 02:38:27 [5560] zjedmsdp20v cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/12)
Feb 12 02:38:27 [5560] zjedmsdp20v cib: info: cib_perform_op: Diff: --- 0.122.31 2
Feb 12 02:38:27 [5560] zjedmsdp20v cib: info: cib_perform_op: Diff: +++ 0.122.32 (null)
Feb 12 02:38:27 [5560] zjedmsdp20v cib: info: cib_perform_op: + /cib: @num_updates=32
Feb 12 02:38:27 [5560] zjedmsdp20v cib: info: cib_perform_op: + /cib/status/node_state[@id='zjedmsdp20v']/lrm[@id='zjedmsdp20v']/lrm_resources/lrm_resource[@id='pgsql']/lrm
_rsc_op[@id='pgsql_last_0']: @operation_key=pgsql_start_0, @operation=start, @transition-key=15:7444:0:8c793138-d4e4-4780-88f2-2da6808c25ca, @transition-magic=2:1;15:7444:0:8c793138-d4e4-478
0-88f2-2da6808c25ca, @call-id=11, @rc-code=1, @op-status=2, @exec-time=60001
And output of crm_mod -Afr -1 is:
Last updated: Mon Feb 12 02:39:00 2018
Last change: Mon Feb 12 02:31:45 2018
Stack: cman
Current DC: zjedmsdp19v - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured
3 Resources configured
Online: [ zjedmsdp19v zjedmsdp20v ]
Full list of resources:
vip-master (ocf::heartbeat:IPaddr2): Started zjedmsdp19v
Master/Slave Set: msPostgresql [pgsql]
Masters: [ zjedmsdp19v ]
Stopped: [ zjedmsdp20v ]
Node Attributes:
* Node zjedmsdp19v:
+ master-pgsql : 1000
+ pgsql-data-status : LATEST
+ pgsql-master-baseline : 0000002054000090
+ pgsql-status : PRI
* Node zjedmsdp20v:
+ master-pgsql : -INFINITY
+ pgsql-data-status : DISCONNECT
+ pgsql-status : STOP
Migration summary:
* Node zjedmsdp19v:
* Node zjedmsdp20v:
pgsql: migration-threshold=1 fail-count=1000000 last-failure='Mon Feb 12 02:38:27 2018'
Failed actions:
pgsql_start_0 on zjedmsdp20v 'unknown error' (1): call=11, status=Timed Out, last-rc-change='Mon Feb 12 02:37:27 2018', queued=0ms, exec=60001ms
Last updated: Mon Feb 12 02:39:00 2018
Last change: Mon Feb 12 02:31:45 2018
Stack: cman
Current DC: zjedmsdp19v - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured
3 Resources configured
Online: [ zjedmsdp19v zjedmsdp20v ]
Full list of resources:
vip-master (ocf::heartbeat:IPaddr2): Started zjedmsdp19v
Master/Slave Set: msPostgresql [pgsql]
Masters: [ zjedmsdp19v ]
Stopped: [ zjedmsdp20v ]
Node Attributes:
* Node zjedmsdp19v:
+ master-pgsql : 1000
+ pgsql-data-status : LATEST
+ pgsql-master-baseline : 0000002054000090
+ pgsql-status : PRI
* Node zjedmsdp20v:
+ master-pgsql : -INFINITY
+ pgsql-data-status : DISCONNECT
+ pgsql-status : STOP
Migration summary:
* Node zjedmsdp19v:
* Node zjedmsdp20v:
pgsql: migration-threshold=1 fail-count=1000000 last-failure='Mon Feb 12 02:38:27 2018'
Failed actions:
pgsql_start_0 on zjedmsdp20v 'unknown error' (1): call=11, status=Timed Out, last-rc-change='Mon Feb 12 02:37:27 2018', queued=0ms, exec=60001ms
I shall thankful any guidance or support on this
Regards,
Chander