Thread: [GENERAL] Chained slaves smaller?

Hi,

I have SR set up in a couple of datacenters, where there’s a master in DC_A with 2 slaves, and a 3rd slave off that
masterin DC_ B. Also, in DC_B I have 2 slaves chained off the “local master”. Our main database is ~551GB in DC_A and
onthe replica in B that is subscribed to the real master. However, on one of the chained slaves in DC_B that database
isonly 484GB. The only thing different about this smaller slave is that it was created by taking a basebackup from the
“localmaster” in DC_B rather than sucking it over the WAN from the true master in DC_A.  

This makes no sense to me since I thought SR replicas are bit for bit copies, so I’m somewhat concerned. Any ideas how
thiscould be? 
—
Jon Erdman
Postgres Zealot

Jon Erdman <postgresql@thewickedtribe.net> writes:

> Hi,
>
> I have SR set up in a couple of datacenters, where there’s a master in
> DC_A with 2 slaves, and a 3rd slave off that master in DC_ B. Also, in
> DC_B I have 2 slaves chained off the “local master”. Our main database
> is ~551GB in DC_A and on the replica in B that is subscribed to the
> real master. However, on one of the chained slaves in DC_B that
> database is only 484GB. The only thing different about this smaller
> slave is that it was created by taking a basebackup from the “local
> master” in DC_B rather than sucking it over the WAN from the true
> master in DC_A.

Hey Jon.

They should be pretty near identical unless there are unlogged tables on
master but in such a case all your streamers should be undersized to
account for this, not just a single one.

I'd troubleshoot by...

* Question the size measurement method, df, du etc.

* Re-do the streamer and/or build an additional one using same method to
  see if reproduces.

* Determine where the descrepancy is on a per-directory level.

I had a system once that had 500G of orphaned heap/index data files
after a hard crash that went away finally after a pg_upgrade.

If your master and all streamers except that newest one are likewise
bloated, this is one possible explanation.

Not sure how plausible though.

In other words,  that extra data was valid, visible and legit at time of
creation of all other streamers but  now somehow stayed behind after
something $interesting happened... and pg_basebackup ignores it.

Pls let us know what you find.

HTH


> This makes no sense to me since I thought SR replicas are bit for bit copies, so I’m somewhat concerned. Any ideas
howthis could be? 
> —
> Jon Erdman
> Postgres Zealot

--
Jerry Sievers
Postgres DBA/Development Consulting
e: postgres.consulting@comcast.net
p: 312.241.7800

Nevermind. Turns out it was on the wrong timeline and replication was broken. It was smaller because it was 77 days
behind.(facepalm) 

> On Jun 23, 2017, at 2:40 PM, Jon Erdman <postgresql@thewickedtribe.net> wrote:
>
> Hi,
>
> I have SR set up in a couple of datacenters, where there’s a master in DC_A with 2 slaves, and a 3rd slave off that
masterin DC_ B. Also, in DC_B I have 2 slaves chained off the “local master”. Our main database is ~551GB in DC_A and
onthe replica in B that is subscribed to the real master. However, on one of the chained slaves in DC_B that database
isonly 484GB. The only thing different about this smaller slave is that it was created by taking a basebackup from the
“localmaster” in DC_B rather than sucking it over the WAN from the true master in DC_A.  
>
> This makes no sense to me since I thought SR replicas are bit for bit copies, so I’m somewhat concerned. Any ideas
howthis could be? 
> —
> Jon Erdman
> Postgres Zealot