Thread: Start Walreceiver completely before shut down it on standby server.
Start Walreceiver completely before shut down it on standby server.
The walreceiver will be shut down, when read an invalid record in the
WAL streaming from master.And then, we retry from archive/pg_wal again.
After that, we start walreceiver in RequestXLogStreaming(), and read
record from the WAL streaming. But before walreceiver starts, we read
data from file which be streamed over and present in pg_wal by last
time, because of walrcv->receivedUpto > RecPtr and the wal is actually
flush on disk. Now, we read the invalid record again, what the next to
do? Shut down the walreceiver and do it again.
So, we always read the invalid record, starting the walreceiver and make
it down before it starts completely.
This code fix it by set the walrcv->receivedUpto to the starting point,
we can read nothing before the walreceiver starts and streaming.
The walreceiver will be shut down, when read an invalid record in the
WAL streaming from master.And then, we retry from archive/pg_wal again.
After that, we start walreceiver in RequestXLogStreaming(), and read
record from the WAL streaming. But before walreceiver starts, we read
data from file which be streamed over and present in pg_wal by last
time, because of walrcv->receivedUpto > RecPtr and the wal is actually
flush on disk. Now, we read the invalid record again, what the next to
do? Shut down the walreceiver and do it again.
So, we always read the invalid record, starting the walreceiver and make
it down before it starts completely.
This code fix it by set the walrcv->receivedUpto to the starting point,
we can read nothing before the walreceiver starts and streaming.
Attachment
On Tue, Dec 10, 2019 at 3:06 AM jiankang liu <liujk1994@gmail.com> wrote:
Start Walreceiver completely before shut down it on standby server.
The walreceiver will be shut down, when read an invalid record in the
WAL streaming from master.And then, we retry from archive/pg_wal again.
After that, we start walreceiver in RequestXLogStreaming(), and read
record from the WAL streaming. But before walreceiver starts, we read
data from file which be streamed over and present in pg_wal by last
time, because of walrcv->receivedUpto > RecPtr and the wal is actually
flush on disk. Now, we read the invalid record again, what the next to
do? Shut down the walreceiver and do it again.
I am missing something here, if walrcv->receivedUpto > RecPtr, why are we getting / reading invalid record?
At Tue, 10 Dec 2019 10:40:53 -0800, Ashwin Agrawal <aagrawal@pivotal.io> wrote in > On Tue, Dec 10, 2019 at 3:06 AM jiankang liu <liujk1994@gmail.com> wrote: > > > Start Walreceiver completely before shut down it on standby server. > > > > The walreceiver will be shut down, when read an invalid record in the > > WAL streaming from master.And then, we retry from archive/pg_wal again. > > > > After that, we start walreceiver in RequestXLogStreaming(), and read > > record from the WAL streaming. But before walreceiver starts, we read > > data from file which be streamed over and present in pg_wal by last > > time, because of walrcv->receivedUpto > RecPtr and the wal is actually > > flush on disk. Now, we read the invalid record again, what the next to > > do? Shut down the walreceiver and do it again. > > > > I am missing something here, if walrcv->receivedUpto > RecPtr, why are we > getting / reading invalid record? I bet on that the standby is connecting to a wrong master. For example, something like happens when the master has been reinitalized from a backup and experienced another history, then the standby was initialized from the reborn master but the stale archive files on the standby are left alone. Anyway that cannot happen on correctly running replication set and what to do in the case is starting from a new basebackup of the master, making sure to erase stale archive files if any. About the proposed fix, it doesn't seem to cause start process to rewind WAL to that LSN. Even if that happens, it leads to no better than a broken database. regards. -- Kyotaro Horiguchi NTT Open Source Software Center
I'm sorry I did not say it clearly.
During my use of PG, I encountered such errors "incorrect resource manager data checksum in record at 0/5013730",
it will keeps printing this error message and never stops on standby server，at same time, the walreceier process is lost.
In a few months, we encountered this situation twice, each time testing more than 200 connections to read/write, after 2 or 3 days of continuous operation.
Maybe disk problem, but also the line is operating system problems, leading off the data disk error occurred.
It will print this error never stops, and the walreceiver is lost, if we do nothing.
Just restart standby server of PG, it only print this error message once, and then connect to master. Everthing is OK.
Why we neet to restart my server, it can not fixs that problem online? Why the walreceiver is lost?
The record has been flushed to disk by the walreceiver, and the the startup process always read record and apply it. When it reads an invalid record, it will shut down the walreceiver by signal SIGTERM. Then, it will read from ARCHIVE/PG_WAL, just read files from pg_wal. Read an invalid record again or read the file end(the read len is not equals XLOG_BLCKSZ), the startup process will starts the walreceiver by RequestXLogStreaming() and switch to read from XLOG_FROM_STREAM.
In RequestXLogStreaming(), set the walrcv->receiveStart = recptr, the walreceiver will get the WAL from master start at walrcv->receiveStart. So we can read the new data which streaming from master by the walreceiver this time, instead of the wrong data on disk. It should not print the error message never stop and the walreceiver should not be lost after we read an invalid record. But the fact is not work.
What happened?
The previous step, the startup process starts the walreceiver, and switch to read from XLOG_FROM_STREAM. Then, check the walreceiver is active before we read it, even the postmaster does not start the walreceiver, but the walrcv->walRcvState == STARTING, we think the walreceiver is active, and ready to read.
Now, begin to read data if new data has arrived. How to check it?
If the Recptr, which is pointer we read, is lower than walrcv->receivedUpto, we can read the data, even if the walreceiver does not start completely and the data is OLD which has invalid reccord.
Read it, and read an invalid reccord again, just stop the walreceiver again(the walreceiver does not start completely, it has not pid, just set walrcv->walRcvState = WALRCV_STOPPED). When the walreceiver starts, running into WalReceiverMain(), check the walrcv->walRcvState == WALRCV_STOPPED, myself has been shut down by others, just exit. So the walreceiver starts, exit again and again.
The startup process next to do is, starts the walreceiver, read data(read the invalid record), shut down the walreceiver, also agiain and again.
Why restart standby server of PG will be OK?
The startup process begin to REDO, reads an invalid record, prints the error message, and starts the walreceiver by RequestXLogStreaming() and switch to read from XLOG_FROM_STREAM. This is first time to start the walreceiver, set walrcv->receivedUpto = walrcv->receiveStart = recptr.
The startup process ready to read new data, but RecPtr >= walrcv->receivedUpto, wait the walreceiver get WAL from master.
So that, we get the WAL from master instead of the WAL on disk, by restart standby server of PG.
By my fix, ervery time we start the walreceiver, the startup process will wait for new data instead of read OLD data, Just like restart standby server.
So, we can fix the problem online and the walreceiver will not be lost.
Kyotaro Horiguchi <horikyota.ntt@gmail.com> 于2019年12月11日周三 下午1:38写道：
At Tue, 10 Dec 2019 10:40:53 -0800, Ashwin Agrawal <aagrawal@pivotal.io> wrote in
> On Tue, Dec 10, 2019 at 3:06 AM jiankang liu <liujk1994@gmail.com> wrote:
>
> > Start Walreceiver completely before shut down it on standby server.
> >
> > The walreceiver will be shut down, when read an invalid record in the
> > WAL streaming from master.And then, we retry from archive/pg_wal again.
> >
> > After that, we start walreceiver in RequestXLogStreaming(), and read
> > record from the WAL streaming. But before walreceiver starts, we read
> > data from file which be streamed over and present in pg_wal by last
> > time, because of walrcv->receivedUpto > RecPtr and the wal is actually
> > flush on disk. Now, we read the invalid record again, what the next to
> > do? Shut down the walreceiver and do it again.
> >
>
> I am missing something here, if walrcv->receivedUpto > RecPtr, why are we
> getting / reading invalid record?
I bet on that the standby is connecting to a wrong master. For
example, something like happens when the master has been reinitalized
from a backup and experienced another history, then the standby was
initialized from the reborn master but the stale archive files on the
standby are left alone.
Anyway that cannot happen on correctly running replication set and
what to do in the case is starting from a new basebackup of the
master, making sure to erase stale archive files if any.
About the proposed fix, it doesn't seem to cause start process to
rewind WAL to that LSN. Even if that happens, it leads to no better
than a broken database.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Wed, Dec 11, 2019 at 04:06:26PM +0800, jiankang liu wrote: > During my use of PG, I encountered such errors "incorrect resource manager > data checksum in record at 0/5013730", > it will keeps printing this error message and never stops on standby > server，at same time, the walreceier process is lost. > Maybe disk problem, but also the line is operating system problems, leading > off the data disk error occurred. An incorrect resource manager checksum points to the checksum of a record to be busted, because the full record has been read from disk, but it is failing a basic sanity check. That's basically a sign of corruption. Do you have data checksums enabled? If your WAL records are broken, you have unfortunately good chances of having problems in other areas of your data folder :( At this stage, it would be wiser to run diagnostics on your server, and be very careful with your chosen steps. Here are guidelines on the wiki: https://wiki.postgresql.org/wiki/Corruption -- Michael