Re: very high replay_lag on 3-node cluster - Mailing list pgsql-general

From Jehan-Guillaume (ioguix) de Rorthais
Subject Re: very high replay_lag on 3-node cluster
Date
Msg-id 20190722112754.512b4840@firost
Whole thread Raw
In response to Re: very high replay_lag on 3-node cluster  (Tiemen Ruiten <t.ruiten@tech-lab.io>)
Responses Re: very high replay_lag on 3-node cluster  (Tiemen Ruiten <t.ruiten@tech-lab.io>)
List pgsql-general
Hi,

On Mon, 22 Jul 2019 11:05:57 +0200
Tiemen Ruiten <t.ruiten@tech-lab.io> wrote:
[...]
> > Now to my current issue: I took the advice to add more monitoring on
> > replay lag (using pg_last_xact_replay_timestamp) and things are not looking
> > good. Last night replication lagged by almost 6 hours on one of the
> > nodes[3], but eventually caught up. As you can see in that screenshot,
> > ph-sql-03 is consistently slower to replay than ph-sql-05 (ph-sql-04 is the
> > current master) and there happen to be different SSD's in ph-sql-03
> > (Crucial MX300 vs Crucial MX500 in the other two), which makes me think
> > this is IO related.

Such a difference is quite surprising. Moreover, I suppose you have some
caching in front of disks (either RAID or SAN?). Could you describe your disk
stack with more details?

Do you have any detailed metrics about disks and network IO to share?

The network is the same for both nodes?



pgsql-general by date:

Previous
From: Tiemen Ruiten
Date:
Subject: Re: very high replay_lag on 3-node cluster
Next
From: Tiemen Ruiten
Date:
Subject: Re: very high replay_lag on 3-node cluster