Thread: Slave claims requested WAL segment already removed - but it wasn't
Hi- We have a master (pg 9.4.4 on Ubuntu 14.10) and a slave (pg 9.4.8 on Centos 7). During a period of heavy use, the slave began complaining that the “requested WAL segment xxxxxx has already been removed”. But the WAL segment was still on the master. The issue was resolved by manually copying the pg_xlog directory over to the slave. I don’t see any errors on the master log file, or any other messages on the slave’s. What happened? How can this be prevented in the future? Thanks, Jeff
On 06/01/2016 06:43 PM, Jeff Beck wrote: > Hi- > We have a master (pg 9.4.4 on Ubuntu 14.10) and a slave (pg 9.4.8 on > Centos 7). During a period of heavy use, the slave began complaining > that the “requested WAL segment xxxxxx has already been removed”. But > the WAL segment was still on the master. The issue was resolved by > manually copying the pg_xlog directory over to the slave. What is your setup?: 1) Streaming 2) WAL archiving 3) Both 4) Do you have more then one standby? > > I don’t see any errors on the master log file, or any other messages on > the slave’s. What happened? How can this be prevented in the future? > > Thanks, > Jeff > > > > -- Adrian Klaver adrian.klaver@aklaver.com
On Thu, Jun 2, 2016 at 11:43 AM, Jeff Beck <becked@fastmail.fm> wrote:
Hi-
We have a master (pg 9.4.4 on Ubuntu 14.10) and a slave (pg 9.4.8 on
Centos 7). During a period of heavy use, the slave began complaining
that the “requested WAL segment xxxxxx has already been removed”. But
the WAL segment was still on the master. The issue was resolved by
manually copying the pg_xlog directory over to the slave.
I don’t see any errors on the master log file, or any other messages on
the slave’s. What happened? How can this be prevented in the future?
If you are using streaming replication, then, enabling replication slots will ensure all the WALs which are not replayed to slave will be retained at master.
Having said that, your situation seems a bit strange to me. Slave cannot complain about an existing WAL segment at master, can you just give a bit more details about your replication architecture ?
Slave is unable to read the required WAL segment for some reason ? Do you have this situation happening regularly ?
Regards,
Venkata B N
Fujitsu Australia
On 2 June 2016 at 02:43, Jeff Beck <becked@fastmail.fm> wrote: > Hi- > We have a master (pg 9.4.4 on Ubuntu 14.10) and a slave (pg 9.4.8 on > Centos 7). During a period of heavy use, the slave began complaining > that the “requested WAL segment xxxxxx has already been removed”. But > the WAL segment was still on the master. The issue was resolved by > manually copying the pg_xlog directory over to the slave. > > I don’t see any errors on the master log file, or any other messages on > the slave’s. What happened? How can this be prevented in the future? I've seen this once. In my case there was a downstream slave of the slave that I'd forgotten about, and the log entries on the slave were the result of slave-of-slave asking slave for a WAL file that was not present on slave.
Thanks! This seems to be exactly what the issue was. On Fri, Jun 3, 2016, at 03:36 AM, Nick Cleaton wrote: > On 2 June 2016 at 02:43, Jeff Beck <becked@fastmail.fm> wrote: > > Hi- > > We have a master (pg 9.4.4 on Ubuntu 14.10) and a slave (pg 9.4.8 on > > Centos 7). During a period of heavy use, the slave began complaining > > that the “requested WAL segment xxxxxx has already been removed”. But > > the WAL segment was still on the master. The issue was resolved by > > manually copying the pg_xlog directory over to the slave. > > > > I don’t see any errors on the master log file, or any other messages on > > the slave’s. What happened? How can this be prevented in the future? > > I've seen this once. In my case there was a downstream slave of the > slave that I'd forgotten about, and the log entries on the slave were > the result of slave-of-slave asking slave for a WAL file that was not > present on slave. > > > -- > Sent via pgsql-general mailing list (pgsql-general@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-general