Thread: BUG #13822: Slave terminated - WAL contains references to invalid page
BUG #13822: Slave terminated - WAL contains references to invalid page
From
marek.petr@tieto.com
Date:
The following bug has been logged on the website: Bug reference: 13822 Logged by: Marek Petr Email address: marek.petr@tieto.com PostgreSQL version: 9.4.5 Operating system: RHEL6_x86_64 Description: Several days after one in-place and one out of place upgrade from 9.3 to 9.4 version following event occured on both environments: 2015-12-15 22:35:18 CET @ WARNING: page 4119662 of relation base/16422/18134 is uninitialized 2015-12-15 22:35:18 CET @ CONTEXT: xlog redo visible: rel 1663/16422/18134; blk 4119662 2015-12-15 22:35:18 CET @ PANIC: WAL contains references to invalid pages 2015-12-15 22:35:18 CET @ CONTEXT: xlog redo visible: rel 1663/16422/18134; blk 4119662 2015-12-15 22:35:18 CET @ LOG: startup process (PID 22269) was terminated by signal 6: Aborted 2015-12-15 22:35:18 CET @ LOG: terminating any other active server process Once it was TOAST and another regular table.
Re: BUG #13822: Slave terminated - WAL contains references to invalid page
From
Michael Paquier
Date:
On Thu, Dec 17, 2015 at 9:50 PM, <marek.petr@tieto.com> wrote: > Several days after one in-place and one out of place upgrade from 9.3 to 9.4 > version following event occured on both environments: > > 2015-12-15 22:35:18 CET @ WARNING: page 4119662 of relation base/16422/18134 > is uninitialized > 2015-12-15 22:35:18 CET @ CONTEXT: xlog redo visible: rel 1663/16422/18134; > blk 4119662 > 2015-12-15 22:35:18 CET @ PANIC: WAL contains references to invalid pages > 2015-12-15 22:35:18 CET @ CONTEXT: xlog redo visible: rel 1663/16422/18134; > blk 4119662 > 2015-12-15 22:35:18 CET @ LOG: startup process (PID 22269) was terminated by > signal 6: Aborted > 2015-12-15 22:35:18 CET @ LOG: terminating any other active server process > > Once it was TOAST and another regular table. This is the indication of some data corruption, page 4119662 referring to at least a size of 31GB, but this so less information it is hard to guess what could happen. Is 31GB more or less the size of this relation? If you deploy a slave from a fresh base backup, do you still see the error? That's unlikely so if it is the second time you are seeing this problem, but it may be a problem of corruption within the WAL segments themselves. -- Michael
SGVsbG8sDQoNClllcywgIHRoYXQgcmVsYXRpb24geW91J3JlIGFza2luZyBoYXMgY3VycmVudGx5 IDM5R0IuIFNsYXZlIHJlYnVpbGQgaGVscGVkIGJ1dCBpbiBvdGhlciBlbnZpcm9ubWVudCBpdCBv Y2N1cmVkIGFnYWluIGV2ZW4gYWZ0ZXIgdGhlIHJlYnVpbGQuDQpGb2xsb3dpbmcgdHdvIG9jY3Vy ZW5jZXMgYXJlIGZyb20gZGlmZmVyZW50IFBvc3RncmVTUUwgbWFzdGVyL3NsYXZlIHN5c3RlbSAo dXNpbmcgc3RyZWFtaW5nIHJlcGxpY2F0aW9uKSBydW5uaW5nIGF0IGRpZmZlcmVudCBtYWNoaW5l czoNCg0KMjAxNS0xMi0xNSAxMzowNTozOSBDRVQgQCAgV0FSTklORzogIHBhZ2UgNDMzMzI3NSBv ZiByZWxhdGlvbiBiYXNlLzE2NDIyLzE3MjMwIGlzIHVuaW5pdGlhbGl6ZWQNCjIwMTUtMTItMTUg MTM6MDU6MzkgQ0VUIEAgIENPTlRFWFQ6ICB4bG9nIHJlZG8gdmlzaWJsZTogcmVsIDE2NjMvMTY0 MjIvMTcyMzA7IGJsayA0MzMzMjc1DQoyMDE1LTEyLTE1IDEzOjA1OjM5IENFVCBAICBQQU5JQzog IFdBTCBjb250YWlucyByZWZlcmVuY2VzIHRvIGludmFsaWQgcGFnZXMNCjIwMTUtMTItMTUgMTM6 MDU6MzkgQ0VUIEAgIENPTlRFWFQ6ICB4bG9nIHJlZG8gdmlzaWJsZTogcmVsIDE2NjMvMTY0MjIv MTcyMzA7IGJsayA0MzMzMjc1DQoyMDE1LTEyLTE1IDEzOjA1OjM5IENFVCBAICBMT0c6ICBzdGFy dHVwIHByb2Nlc3MgKFBJRCA4OTYzKSB3YXMgdGVybWluYXRlZCBieSBzaWduYWwgNjogQWJvcnRl ZA0KMjAxNS0xMi0xNSAxMzowNTozOSBDRVQgQCAgTE9HOiAgdGVybWluYXRpbmcgYW55IG90aGVy IGFjdGl2ZSBzZXJ2ZXIgcHJvY2Vzc2VzDQoNCjIwMTUtMTItMjIgMDA6MjU6MTEgQ0VUIEAgIFdB Uk5JTkc6ICBwYWdlIDcxNTY2IG9mIHJlbGF0aW9uIGJhc2UvMTY0MjIvMjMyNTMgaXMgdW5pbml0 aWFsaXplZA0KMjAxNS0xMi0yMiAwMDoyNToxMSBDRVQgQCAgQ09OVEVYVDogIHhsb2cgcmVkbyB2 aXNpYmxlOiByZWwgMTY2My8xNjQyMi8yMzI1MzsgYmxrIDcxNTY2DQoyMDE1LTEyLTIyIDAwOjI1 OjExIENFVCBAICBQQU5JQzogIFdBTCBjb250YWlucyByZWZlcmVuY2VzIHRvIGludmFsaWQgcGFn ZXMNCjIwMTUtMTItMjIgMDA6MjU6MTEgQ0VUIEAgIENPTlRFWFQ6ICB4bG9nIHJlZG8gdmlzaWJs ZTogcmVsIDE2NjMvMTY0MjIvMjMyNTM7IGJsayA3MTU2Ng0KMjAxNS0xMi0yMiAwMDoyNToxMiBD RVQgQCAgTE9HOiAgc3RhcnR1cCBwcm9jZXNzIChQSUQgMjQ0MzQpIHdhcyB0ZXJtaW5hdGVkIGJ5 IHNpZ25hbCA2OiBBYm9ydGVkDQoyMDE1LTEyLTIyIDAwOjI1OjEyIENFVCBAICBMT0c6ICB0ZXJt aW5hdGluZyBhbnkgb3RoZXIgYWN0aXZlIHNlcnZlciBwcm9jZXNzZXMNCg0Kc2VsZWN0IHJlbG5h bWUgZnJvbSBwZ19jbGFzcyB3aGVyZSByZWxmaWxlbm9kZSBpbiAoJzE3MjMwJywnMjMyNTMnKTsN CiAgICByZWxuYW1lICAgICANCi0tLS0tLS0tLS0tLS0tLS0NCiBwZ190b2FzdF8xNzIyNQ0KIHBn X3RvYXN0XzIzMjQ2DQooMiByb3dzKQ0KDQpGaXJzdCB0b2FzdCdzIHJlbGF0aW9uIGhhcyAzNEdC LCBzZWNvbmQgMjQ1MiBNQi4NCg0KSXMgaXQgcG9zc2libGUgdG8gZ2V0IG1vcmUgaW5mbyBmcm9t IHNvbWUgZGVlcGVyIGxvZ2dpbmcgZm9yIHRoZSBjYXNlIGl0IHdpbGwgb2NjdXIgYWdhaW4/DQoN ClJlZ2FyZHMNCk1hcmVrDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBNaWNo YWVsIFBhcXVpZXIgW21haWx0bzptaWNoYWVsLnBhcXVpZXJAZ21haWwuY29tXSANClNlbnQ6IEZy aWRheSwgRGVjZW1iZXIgMTgsIDIwMTUgNjowNSBBTQ0KVG86IFBldHIgTWFyZWsgPE1hcmVrLlBl dHJAdGlldG8uY29tPg0KQ2M6IFBvc3RncmVTUUwgbWFpbGluZyBsaXN0cyA8cGdzcWwtYnVnc0Bw b3N0Z3Jlc3FsLm9yZz4NClN1YmplY3Q6IFJlOiBbQlVHU10gQlVHICMxMzgyMjogU2xhdmUgdGVy bWluYXRlZCAtIFdBTCBjb250YWlucyByZWZlcmVuY2VzIHRvIGludmFsaWQgcGFnZQ0KDQpPbiBU aHUsIERlYyAxNywgMjAxNSBhdCA5OjUwIFBNLCAgPG1hcmVrLnBldHJAdGlldG8uY29tPiB3cm90 ZToNCj4gU2V2ZXJhbCBkYXlzIGFmdGVyIG9uZSBpbi1wbGFjZSBhbmQgb25lIG91dCBvZiBwbGFj ZSB1cGdyYWRlIGZyb20gOS4zIA0KPiB0byA5LjQgdmVyc2lvbiBmb2xsb3dpbmcgZXZlbnQgb2Nj dXJlZCBvbiBib3RoICBlbnZpcm9ubWVudHM6DQo+DQo+IDIwMTUtMTItMTUgMjI6MzU6MTggQ0VU IEAgV0FSTklORzogcGFnZSA0MTE5NjYyIG9mIHJlbGF0aW9uIA0KPiBiYXNlLzE2NDIyLzE4MTM0 IGlzIHVuaW5pdGlhbGl6ZWQNCj4gMjAxNS0xMi0xNSAyMjozNToxOCBDRVQgQCBDT05URVhUOiB4 bG9nIHJlZG8gdmlzaWJsZTogcmVsIA0KPiAxNjYzLzE2NDIyLzE4MTM0OyBibGsgNDExOTY2Mg0K PiAyMDE1LTEyLTE1IDIyOjM1OjE4IENFVCBAIFBBTklDOiBXQUwgY29udGFpbnMgcmVmZXJlbmNl cyB0byBpbnZhbGlkIA0KPiBwYWdlcw0KPiAyMDE1LTEyLTE1IDIyOjM1OjE4IENFVCBAIENPTlRF WFQ6IHhsb2cgcmVkbyB2aXNpYmxlOiByZWwgDQo+IDE2NjMvMTY0MjIvMTgxMzQ7IGJsayA0MTE5 NjYyDQo+IDIwMTUtMTItMTUgMjI6MzU6MTggQ0VUIEAgTE9HOiBzdGFydHVwIHByb2Nlc3MgKFBJ RCAyMjI2OSkgd2FzIA0KPiB0ZXJtaW5hdGVkIGJ5IHNpZ25hbCA2OiBBYm9ydGVkDQo+IDIwMTUt MTItMTUgMjI6MzU6MTggQ0VUIEAgTE9HOiB0ZXJtaW5hdGluZyBhbnkgb3RoZXIgYWN0aXZlIHNl cnZlciANCj4gcHJvY2Vzcw0KPg0KPiBPbmNlIGl0IHdhcyBUT0FTVCBhbmQgYW5vdGhlciByZWd1 bGFyIHRhYmxlLg0KDQpUaGlzIGlzIHRoZSBpbmRpY2F0aW9uIG9mIHNvbWUgZGF0YSBjb3JydXB0 aW9uLCBwYWdlIDQxMTk2NjIgcmVmZXJyaW5nIHRvIGF0IGxlYXN0IGEgc2l6ZSBvZiAzMUdCLCBi dXQgdGhpcyBzbyBsZXNzIGluZm9ybWF0aW9uIGl0IGlzIGhhcmQgdG8gZ3Vlc3Mgd2hhdCBjb3Vs ZCBoYXBwZW4uIElzIDMxR0IgbW9yZSBvciBsZXNzIHRoZSBzaXplIG9mIHRoaXMgcmVsYXRpb24/ IElmIHlvdSBkZXBsb3kgYSBzbGF2ZSBmcm9tIGEgZnJlc2ggYmFzZSBiYWNrdXAsIGRvIHlvdSBz dGlsbCBzZWUgdGhlIGVycm9yPyBUaGF0J3MgdW5saWtlbHkgc28gaWYgaXQgaXMgdGhlIHNlY29u ZCB0aW1lIHlvdSBhcmUgc2VlaW5nIHRoaXMgcHJvYmxlbSwgYnV0IGl0IG1heSBiZSBhIHByb2Js ZW0gb2YgY29ycnVwdGlvbiB3aXRoaW4gdGhlIFdBTCBzZWdtZW50cyB0aGVtc2VsdmVzLg0KLS0N Ck1pY2hhZWwNCg==
Re: BUG #13822: Slave terminated - WAL contains references to invalid page
From
Michael Paquier
Date:
On Tue, Dec 22, 2015 at 9:05 PM, <Marek.Petr@tieto.com> wrote:
> 2015-12-22 00:25:11 CET @ WARNING: page 71566 of relation base/16422/23253 is uninitialized
> 2015-12-22 00:25:11 CET @ CONTEXT: xlog redo visible: rel 1663/16422/23253; blk 71566
> 2015-12-22 00:25:11 CET @ PANIC: WAL contains references to invalid pages
> 2015-12-22 00:25:11 CET @ CONTEXT: xlog redo visible: rel 1663/16422/23253; blk 71566
> 2015-12-22 00:25:12 CET @ LOG: startup process (PID 24434) was terminated by signal 6: Aborted
> 2015-12-22 00:25:12 CET @ LOG: terminating any other active server processes
Looking more closely at that, this is the code path of the redo
routine for XLOG_HEAP2_VISIBLE. I have been looking at the area of the
code around visibilitymap_set to try to see if there could be a race
condition with another backend extending the relation and causing the
page to be uninitialized but have not found anything yet. 9.4 has been
out for some time, and this is the first report of this kind for this
redo routine. Still, you have been able to reproduce the problem
twice, so this has the smell of a bug... Others, opinions?
Did you rebuild a new slave and let the master running, and perhaps
some data corruption is coming from it? What's the state of the same
pages on the master? Are they zero'ed?
Also, are you using any parameter with a value different than the
default. I don't know fsync, full_page_writes...
> select relname from pg_class where relfilenode in ('17230','23253');
> relname
> ----------------
> pg_toast_17225
> pg_toast_23246
> (2 rows)
>
> First toast's relation has 34GB, second 2452 MB.
> Is it possible to get more info from some deeper logging for the case it will occur again?
I am not sure to understand what you are looking for here. You could
make the logs more verbose but this would bloat your log partition...
--
Michael
VHJpZWQgdG8gdXNlIHBhZ2VpbnNwZWN0IG1vZHVsZSBmb3IgYWZmZWN0ZWQgcGFnZXMgZnJvbSBs YXN0IHR3byBvY2N1cmVuY2VzOg0KDQoyMDE1LTEyLTE1IDEzOjA1OjM5IENFVCBAICBXQVJOSU5H OiAgcGFnZSA0MzMzMjc1IG9mIHJlbGF0aW9uIGJhc2UvMTY0MjIvMTcyMzAgaXMgdW5pbml0aWFs aXplZA0KMjAxNS0xMi0yMiAwMDoyNToxMSBDRVQgQCAgV0FSTklORzogIHBhZ2UgNzE1NjYgb2Yg cmVsYXRpb24gYmFzZS8xNjQyMi8yMzI1MyBpcyB1bmluaXRpYWxpemVkDQoNCkZvbGxvd2luZyBv dXRwdXRzIGFyZSB0aGUgc2FtZSBmb3IgbWFzdGVyIGFuZCBzbGF2ZToNCg0KIyBzZWxlY3QgcmVs bmFtZSBmcm9tIHBnX2NsYXNzIHdoZXJlIHJlbGZpbGVub2RlIGluICgnMTcyMzAnLCcyMzI1Mycp Ow0KICAgIHJlbG5hbWUgICAgIA0KLS0tLS0tLS0tLS0tLS0tLQ0KIHBnX3RvYXN0XzE3MjI1DQog cGdfdG9hc3RfMjMyNDYNCigyIHJvd3MpDQoNCiMgU0VMRUNUICogRlJPTSBwYWdlX2hlYWRlcihn ZXRfcmF3X3BhZ2UoJ3BnX3RvYXN0LnBnX3RvYXN0XzIzMjQ2JywgNzE1NjYpKTsNCiAgICAgbHNu ICAgICAgfCBjaGVja3N1bSB8IGZsYWdzIHwgbG93ZXIgfCB1cHBlciB8IHNwZWNpYWwgfCBwYWdl c2l6ZSB8IHZlcnNpb24gfCBwcnVuZV94aWQgDQotLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tKy0t LS0tLS0rLS0tLS0tLSstLS0tLS0tKy0tLS0tLS0tLSstLS0tLS0tLS0tKy0tLS0tLS0tLSstLS0t LS0tLS0tLQ0KIDE4MS8xQkY1OTNBOCB8ICAgICAgICAwIHwgICAgIDQgfCAgICA0NCB8ICAgNjAw IHwgICAgODE5MiB8ICAgICA4MTkyIHwgICAgICAgNCB8ICAgICAgICAgMA0KKDEgcm93KQ0KDQoj IFNFTEVDVCAqIEZST00gcGFnZV9oZWFkZXIoZ2V0X3Jhd19wYWdlKCdwZ190b2FzdC5wZ190b2Fz dF8xNzIyNScsIDQzMzMyNzUpKTsNCiAgICAgbHNuICAgICAgfCBjaGVja3N1bSB8IGZsYWdzIHwg bG93ZXIgfCB1cHBlciB8IHNwZWNpYWwgfCBwYWdlc2l6ZSB8IHZlcnNpb24gfCBwcnVuZV94aWQg DQotLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tKy0tLS0tLS0rLS0tLS0tLSstLS0tLS0tKy0tLS0t LS0tLSstLS0tLS0tLS0tKy0tLS0tLS0tLSstLS0tLS0tLS0tLQ0KIDE3Mi8zMEJFQ0I1OCB8ICAg ICAgICAwIHwgICAgIDQgfCAgICA0MCB8ICAgIDY0IHwgICAgODE5MiB8ICAgICA4MTkyIHwgICAg ICAgNCB8ICAgICAgICAgMA0KKDEgcm93KQ0KDQojIFNFTEVDVCAqIEZST00gaGVhcF9wYWdlX2l0 ZW1zKGdldF9yYXdfcGFnZSgncGdfdG9hc3QucGdfdG9hc3RfMjMyNDYnLCA3MTU2NikpOw0KIGxw IHwgbHBfb2ZmIHwgbHBfZmxhZ3MgfCBscF9sZW4gfCAgdF94bWluICAgfCB0X3htYXggfCB0X2Zp ZWxkMyB8ICB0X2N0aWQgICB8IHRfaW5mb21hc2syIHwgdF9pbmZvbWFzayB8IHRfaG9mZiB8IHRf Yml0cyB8IHRfb2lkIA0KLS0tLSstLS0tLS0tLSstLS0tLS0tLS0tKy0tLS0tLS0tKy0tLS0tLS0t LS0tKy0tLS0tLS0tKy0tLS0tLS0tLS0rLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLSstLS0tLS0t LS0tLS0rLS0tLS0tLS0rLS0tLS0tLS0rLS0tLS0tLQ0KICAxIHwgICA2MTYwIHwgICAgICAgIDEg fCAgIDIwMzIgfCAzNjQ4OTQ4MjEgfCAgICAgIDAgfCAgICAgICAgMSB8ICg3MTU2NiwxKSB8ICAg ICAgICAgICAzIHwgICAgICAgMjMwNiB8ICAgICAyNCB8ICAgICAgICB8ICAgICAgDQogIDIgfCAg IDUxNjAgfCAgICAgICAgMSB8ICAgIDk5NCB8IDM2NDg5NDgyMSB8ICAgICAgMCB8ICAgICAgICAx IHwgKDcxNTY2LDIpIHwgICAgICAgICAgIDMgfCAgICAgICAyMzA2IHwgICAgIDI0IHwgICAgICAg IHwgICAgICANCiAgMyB8ICAgMzEyOCB8ICAgICAgICAxIHwgICAyMDMyIHwgMzY0ODk0OTM0IHwg ICAgICAwIHwgICAgICAgIDEgfCAoNzE1NjYsMykgfCAgICAgICAgICAgMyB8ICAgICAgIDIzMDYg fCAgICAgMjQgfCAgICAgICAgfCAgICAgIA0KICA0IHwgICAyNjMyIHwgICAgICAgIDEgfCAgICA0 OTYgfCAzNjQ4OTQ5MzQgfCAgICAgIDAgfCAgICAgICAgMSB8ICg3MTU2Niw0KSB8ICAgICAgICAg ICAzIHwgICAgICAgMjMwNiB8ICAgICAyNCB8ICAgICAgICB8ICAgICAgDQogIDUgfCAgICA2MDAg fCAgICAgICAgMSB8ICAgMjAzMiB8IDM2NDg5NTI0MSB8ICAgICAgMCB8ICAgICAgICAxIHwgKDcx NTY2LDUpIHwgICAgICAgICAgIDMgfCAgICAgICAyMzA2IHwgICAgIDI0IHwgICAgICAgIHwgICAg ICANCig1IHJvd3MpDQoNCiMgU0VMRUNUICogRlJPTSBoZWFwX3BhZ2VfaXRlbXMoZ2V0X3Jhd19w YWdlKCdwZ190b2FzdC5wZ190b2FzdF8xNzIyNScsIDQzMzMyNzUpKTsNCiBscCB8IGxwX29mZiB8 IGxwX2ZsYWdzIHwgbHBfbGVuIHwgIHRfeG1pbiAgIHwgdF94bWF4IHwgdF9maWVsZDMgfCAgIHRf Y3RpZCAgICB8IHRfaW5mb21hc2syIHwgdF9pbmZvbWFzayB8IHRfaG9mZiB8IHRfYml0cyB8IHRf b2lkIA0KLS0tLSstLS0tLS0tLSstLS0tLS0tLS0tKy0tLS0tLS0tKy0tLS0tLS0tLS0tKy0tLS0t LS0tKy0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLSst LS0tLS0tLSstLS0tLS0tLSstLS0tLS0tDQogIDEgfCAgIDYxNjAgfCAgICAgICAgMSB8ICAgMjAz MiB8IDM2MDIyNDIzOCB8ICAgICAgMCB8ICAgICAgICAxIHwgKDQzMzMyNzUsMSkgfCAgICAgICAg ICAgMyB8ICAgICAgIDIzMDYgfCAgICAgMjQgfCAgICAgICAgfCAgICAgIA0KICAyIHwgICA0MTI4 IHwgICAgICAgIDEgfCAgIDIwMzIgfCAzNjAyMjQyMzggfCAgICAgIDAgfCAgICAgICAgMSB8ICg0 MzMzMjc1LDIpIHwgICAgICAgICAgIDMgfCAgICAgICAyMzA2IHwgICAgIDI0IHwgICAgICAgIHwg ICAgICANCiAgMyB8ICAgMjA5NiB8ICAgICAgICAxIHwgICAyMDMyIHwgMzYwMjI0MjM4IHwgICAg ICAwIHwgICAgICAgIDEgfCAoNDMzMzI3NSwzKSB8ICAgICAgICAgICAzIHwgICAgICAgMjMwNiB8 ICAgICAyNCB8ICAgICAgICB8ICAgICAgDQogIDQgfCAgICAgNjQgfCAgICAgICAgMSB8ICAgMjAz MiB8IDM2MDIyNDIzOCB8ICAgICAgMCB8ICAgICAgICAxIHwgKDQzMzMyNzUsNCkgfCAgICAgICAg ICAgMyB8ICAgICAgIDIzMDYgfCAgICAgMjQgfCAgICAgICAgfCAgICAgIA0KKDQgcm93cykNCg0K DQpOb24tZGVmYXVsdCBwYXJzOg0KDQpsaXN0ZW5fYWRkcmVzc2VzID0gJyonDQptYXhfY29ubmVj dGlvbnMgPSA2MDANCnNzbCA9IG9uDQpzc2xfY2VydF9maWxlID0gJ3h4eC5jcnQnICAgICAgICAg ICANCnNzbF9rZXlfZmlsZSA9ICd4eHgua2V5JyAgICAgICAgICAgIA0Kc3NsX2NhX2ZpbGUgPSAn eHh4LmNydCcNCnNoYXJlZF9idWZmZXJzID0gNEdCDQp3b3JrX21lbSA9IDY5OTBrQg0KbWFpbnRl bmFuY2Vfd29ya19tZW0gPSA4MTlNQg0Kc2hhcmVkX3ByZWxvYWRfbGlicmFyaWVzID0gJ3BnX3N0 YXRfc3RhdGVtZW50cycNCnBnX3N0YXRfc3RhdGVtZW50cy5tYXggPSAxMDAwMA0KcGdfc3RhdF9z dGF0ZW1lbnRzLnRyYWNrID0gYWxsDQp3YWxfbGV2ZWwgPSBob3Rfc3RhbmRieQ0KY2hlY2twb2lu dF9zZWdtZW50cyA9IDMyDQpjaGVja3BvaW50X2NvbXBsZXRpb25fdGFyZ2V0ID0gMC45DQphcmNo aXZlX21vZGUgPSBvbg0KYXJjaGl2ZV9jb21tYW5kID0gJ3JzeW5jIC1hICVwIHh4eEB4eHg6L2Rh dGEvd2FsX2FyY2gvJWYnDQptYXhfd2FsX3NlbmRlcnMgPSA1DQp3YWxfa2VlcF9zZWdtZW50cyA9 IDY0DQpob3Rfc3RhbmRieSA9IG9uDQplZmZlY3RpdmVfY2FjaGVfc2l6ZSA9IDEyR0INCg0KDQpT bGF2ZSByZWJ1aWxkZWQgYW5kIGl0J3MgcnVubmluZyBhbG1vc3QgYSB3ZWVrIGZvciBub3cuDQoN ClJlZ2FyZHMNCk1hcmVrDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBNaWNo YWVsIFBhcXVpZXIgW21haWx0bzptaWNoYWVsLnBhcXVpZXJAZ21haWwuY29tXSANClNlbnQ6IFNh dHVyZGF5LCBEZWNlbWJlciAyNiwgMjAxNSAyOjE1IFBNDQpUbzogUGV0ciBNYXJlayA8TWFyZWsu UGV0ckB0aWV0by5jb20+DQpDYzogUG9zdGdyZVNRTCBtYWlsaW5nIGxpc3RzIDxwZ3NxbC1idWdz QHBvc3RncmVzcWwub3JnPg0KU3ViamVjdDogUmU6IFtCVUdTXSBCVUcgIzEzODIyOiBTbGF2ZSB0 ZXJtaW5hdGVkIC0gV0FMIGNvbnRhaW5zIHJlZmVyZW5jZXMgdG8gaW52YWxpZCBwYWdlDQoNCk9u IFR1ZSwgRGVjIDIyLCAyMDE1IGF0IDk6MDUgUE0sICA8TWFyZWsuUGV0ckB0aWV0by5jb20+IHdy b3RlOg0KPiAyMDE1LTEyLTIyIDAwOjI1OjExIENFVCBAICBXQVJOSU5HOiAgcGFnZSA3MTU2NiBv ZiByZWxhdGlvbiANCj4gYmFzZS8xNjQyMi8yMzI1MyBpcyB1bmluaXRpYWxpemVkDQo+IDIwMTUt MTItMjIgMDA6MjU6MTEgQ0VUIEAgIENPTlRFWFQ6ICB4bG9nIHJlZG8gdmlzaWJsZTogcmVsIA0K PiAxNjYzLzE2NDIyLzIzMjUzOyBibGsgNzE1NjYNCj4gMjAxNS0xMi0yMiAwMDoyNToxMSBDRVQg QCAgUEFOSUM6ICBXQUwgY29udGFpbnMgcmVmZXJlbmNlcyB0byBpbnZhbGlkIA0KPiBwYWdlcw0K PiAyMDE1LTEyLTIyIDAwOjI1OjExIENFVCBAICBDT05URVhUOiAgeGxvZyByZWRvIHZpc2libGU6 IHJlbCANCj4gMTY2My8xNjQyMi8yMzI1MzsgYmxrIDcxNTY2DQo+IDIwMTUtMTItMjIgMDA6MjU6 MTIgQ0VUIEAgIExPRzogIHN0YXJ0dXAgcHJvY2VzcyAoUElEIDI0NDM0KSB3YXMgDQo+IHRlcm1p bmF0ZWQgYnkgc2lnbmFsIDY6IEFib3J0ZWQNCj4gMjAxNS0xMi0yMiAwMDoyNToxMiBDRVQgQCAg TE9HOiAgdGVybWluYXRpbmcgYW55IG90aGVyIGFjdGl2ZSBzZXJ2ZXIgDQo+IHByb2Nlc3Nlcw0K DQpMb29raW5nIG1vcmUgY2xvc2VseSBhdCB0aGF0LCB0aGlzIGlzIHRoZSBjb2RlIHBhdGggb2Yg dGhlIHJlZG8gcm91dGluZSBmb3IgWExPR19IRUFQMl9WSVNJQkxFLiBJIGhhdmUgYmVlbiBsb29r aW5nIGF0IHRoZSBhcmVhIG9mIHRoZSBjb2RlIGFyb3VuZCB2aXNpYmlsaXR5bWFwX3NldCB0byB0 cnkgdG8gc2VlIGlmIHRoZXJlIGNvdWxkIGJlIGEgcmFjZSBjb25kaXRpb24gd2l0aCBhbm90aGVy IGJhY2tlbmQgZXh0ZW5kaW5nIHRoZSByZWxhdGlvbiBhbmQgY2F1c2luZyB0aGUgcGFnZSB0byBi ZSB1bmluaXRpYWxpemVkIGJ1dCBoYXZlIG5vdCBmb3VuZCBhbnl0aGluZyB5ZXQuIDkuNCBoYXMg YmVlbiBvdXQgZm9yIHNvbWUgdGltZSwgYW5kIHRoaXMgaXMgdGhlIGZpcnN0IHJlcG9ydCBvZiB0 aGlzIGtpbmQgZm9yIHRoaXMgcmVkbyByb3V0aW5lLiBTdGlsbCwgeW91IGhhdmUgYmVlbiBhYmxl IHRvIHJlcHJvZHVjZSB0aGUgcHJvYmxlbSB0d2ljZSwgc28gdGhpcyBoYXMgdGhlIHNtZWxsIG9m IGEgYnVnLi4uIE90aGVycywgb3BpbmlvbnM/DQoNCkRpZCB5b3UgcmVidWlsZCBhIG5ldyBzbGF2 ZSBhbmQgbGV0IHRoZSBtYXN0ZXIgcnVubmluZywgYW5kIHBlcmhhcHMgc29tZSBkYXRhIGNvcnJ1 cHRpb24gaXMgY29taW5nIGZyb20gaXQ/IFdoYXQncyB0aGUgc3RhdGUgb2YgdGhlIHNhbWUgcGFn ZXMgb24gdGhlIG1hc3Rlcj8gQXJlIHRoZXkgemVybydlZD8NCg0KQWxzbywgYXJlIHlvdSB1c2lu ZyBhbnkgcGFyYW1ldGVyIHdpdGggYSB2YWx1ZSBkaWZmZXJlbnQgdGhhbiB0aGUgZGVmYXVsdC4g SSBkb24ndCBrbm93IGZzeW5jLCBmdWxsX3BhZ2Vfd3JpdGVzLi4uDQoNCj4gc2VsZWN0IHJlbG5h bWUgZnJvbSBwZ19jbGFzcyB3aGVyZSByZWxmaWxlbm9kZSBpbiAoJzE3MjMwJywnMjMyNTMnKTsN Cj4gICAgIHJlbG5hbWUNCj4gLS0tLS0tLS0tLS0tLS0tLQ0KPiAgcGdfdG9hc3RfMTcyMjUNCj4g IHBnX3RvYXN0XzIzMjQ2DQo+ICgyIHJvd3MpDQo+DQo+IEZpcnN0IHRvYXN0J3MgcmVsYXRpb24g aGFzIDM0R0IsIHNlY29uZCAyNDUyIE1CLg0KPiBJcyBpdCBwb3NzaWJsZSB0byBnZXQgbW9yZSBp bmZvIGZyb20gc29tZSBkZWVwZXIgbG9nZ2luZyBmb3IgdGhlIGNhc2UgaXQgd2lsbCBvY2N1ciBh Z2Fpbj8NCg0KSSBhbSBub3Qgc3VyZSB0byB1bmRlcnN0YW5kIHdoYXQgeW91IGFyZSBsb29raW5n IGZvciBoZXJlLiBZb3UgY291bGQgbWFrZSB0aGUgbG9ncyBtb3JlIHZlcmJvc2UgYnV0IHRoaXMg d291bGQgYmxvYXQgeW91ciBsb2cgcGFydGl0aW9uLi4uDQotLQ0KTWljaGFlbA0K
Re: BUG #13822: Slave terminated - WAL contains references to invalid page
From
Michael Paquier
Date:
On Mon, Dec 28, 2015 at 8:50 PM, <Marek.Petr@tieto.com> wrote: > Tried to use pageinspect module for affected pages from last two occurences: > > 2015-12-15 13:05:39 CET @ WARNING: page 4333275 of relation base/16422/17230 is uninitialized > 2015-12-22 00:25:11 CET @ WARNING: page 71566 of relation base/16422/23253 is uninitialized > > Following outputs are the same for master and slave: > > [results] Hm, OK. > Non-default pars: > > [params] There is nothing fishy here. > Slave rebuilded and it's running almost a week for now. Hm. Has this slave replayed the same WAL records as the slave that has failed previously? Or did it use a fresher base backup? If that's the latter the problem would have been fixed by itself for those two relation pages as they would have been correctly created by the base backup and not the WAL replay. Perhaps that's too late, but could it be possible to scan the WAL segments you have and see if there is record referring to those pages being initialized or not? You would need to find a record like that: [insert|update|multi-insert|hot_update](init) rel %u/%u/%u; tid %u/%u tid is t_ctid referred in those upper results you just sent. And this record should normally be present before the ones that caused the PANIC setting the visibility map bit. If that's not the case, it may be possible that there is actually a bug if the page is not found as being initialized properly first. At least we are sure that the corruption is not coming from the master. -- Michael
SSB1c2VkIGZyZXNoIGJhc2UgYmFja3VwIGZvciBzbGF2ZSBhZnRlciBib3RoIGNyYXNoZXMuDQpB bHNvIHRyaWVkIHRvIHNjYW4gYXJjaGl2ZWQgd2FscyBzZXZlcmFsIGhvdXJzIGJlZm9yZSBsYXN0 IGNyYXNoIGFuZCBmb3VuZCBvbmx5IHRoZSBmb2xsb3dpbmcgZm9yIHN0cmluZyA3MTU2NjoNCg0K cm1ncjogSGVhcDIgICAgICAgbGVuIChyZWMvdG90KTogICAgIDIwLyAgICA1MiwgdHg6ICAgICAg ICAgIDAsIGxzbjogMTg3Lzk4Nzg1OUQ4LCBwcmV2IDE4Ny85ODc4NTlBMCwgYmtwOiAwMDAwLCBk ZXNjOiB2aXNpYmxlOiByZWwgMTY2My8xNjQyMi8xNzIxNjsgYmxrIDcxNTY2DQpybWdyOiBIZWFw MiAgICAgICBsZW4gKHJlYy90b3QpOiAgICAgMjAvICAgIDUyLCB0eDogICAgICAgICAgMCwgbHNu OiAxODcvOUNDNTkwMjAsIHByZXYgMTg3LzlDQzU4RkU4LCBia3A6IDAwMDAsIGRlc2M6IHZpc2li bGU6IHJlbCAxNjYzLzE2NDIyLzE3MjIwOyBibGsgNzE1NjYNCnJtZ3I6IEhlYXAyICAgICAgIGxl biAocmVjL3RvdCk6ICAgICAyMC8gICAgNTIsIHR4OiAgICAgICAgICAwLCBsc246IDE4Ny85RTM1 NkQ5OCwgcHJldiAxODcvOUUzNTZENjAsIGJrcDogMDAwMCwgZGVzYzogdmlzaWJsZTogcmVsIDE2 NjMvMTY0MjIvMjMyNTM7IGJsayA3MTU2Ng0KDQpSZWdhcmRzDQpNYXJlaw0KDQotLS0tLU9yaWdp bmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogTWljaGFlbCBQYXF1aWVyIFttYWlsdG86bWljaGFlbC5w YXF1aWVyQGdtYWlsLmNvbV0gDQpTZW50OiBNb25kYXksIERlY2VtYmVyIDI4LCAyMDE1IDQ6Mjkg UE0NClRvOiBQZXRyIE1hcmVrIDxNYXJlay5QZXRyQHRpZXRvLmNvbT4NCkNjOiBQb3N0Z3JlU1FM IG1haWxpbmcgbGlzdHMgPHBnc3FsLWJ1Z3NAcG9zdGdyZXNxbC5vcmc+DQpTdWJqZWN0OiBSZTog W0JVR1NdIEJVRyAjMTM4MjI6IFNsYXZlIHRlcm1pbmF0ZWQgLSBXQUwgY29udGFpbnMgcmVmZXJl bmNlcyB0byBpbnZhbGlkIHBhZ2UNCg0KT24gTW9uLCBEZWMgMjgsIDIwMTUgYXQgODo1MCBQTSwg IDxNYXJlay5QZXRyQHRpZXRvLmNvbT4gd3JvdGU6DQo+IFRyaWVkIHRvIHVzZSBwYWdlaW5zcGVj dCBtb2R1bGUgZm9yIGFmZmVjdGVkIHBhZ2VzIGZyb20gbGFzdCB0d28gb2NjdXJlbmNlczoNCj4N Cj4gMjAxNS0xMi0xNSAxMzowNTozOSBDRVQgQCAgV0FSTklORzogIHBhZ2UgNDMzMzI3NSBvZiBy ZWxhdGlvbiANCj4gYmFzZS8xNjQyMi8xNzIzMCBpcyB1bmluaXRpYWxpemVkDQo+IDIwMTUtMTIt MjIgMDA6MjU6MTEgQ0VUIEAgIFdBUk5JTkc6ICBwYWdlIDcxNTY2IG9mIHJlbGF0aW9uIA0KPiBi YXNlLzE2NDIyLzIzMjUzIGlzIHVuaW5pdGlhbGl6ZWQNCj4NCj4gRm9sbG93aW5nIG91dHB1dHMg YXJlIHRoZSBzYW1lIGZvciBtYXN0ZXIgYW5kIHNsYXZlOg0KPg0KPiBbcmVzdWx0c10NCg0KSG0s IE9LLg0KDQo+IE5vbi1kZWZhdWx0IHBhcnM6DQo+DQo+IFtwYXJhbXNdDQoNClRoZXJlIGlzIG5v dGhpbmcgZmlzaHkgaGVyZS4NCg0KPiBTbGF2ZSByZWJ1aWxkZWQgYW5kIGl0J3MgcnVubmluZyBh bG1vc3QgYSB3ZWVrIGZvciBub3cuDQoNCkhtLiBIYXMgdGhpcyBzbGF2ZSByZXBsYXllZCB0aGUg c2FtZSBXQUwgcmVjb3JkcyBhcyB0aGUgc2xhdmUgdGhhdCBoYXMgZmFpbGVkIHByZXZpb3VzbHk/ IE9yIGRpZCBpdCB1c2UgYSBmcmVzaGVyIGJhc2UgYmFja3VwPyBJZiB0aGF0J3MgdGhlIGxhdHRl ciB0aGUgcHJvYmxlbSB3b3VsZCBoYXZlIGJlZW4gZml4ZWQgYnkgaXRzZWxmIGZvciB0aG9zZSB0 d28gcmVsYXRpb24gcGFnZXMgYXMgdGhleSB3b3VsZCBoYXZlIGJlZW4gY29ycmVjdGx5IGNyZWF0 ZWQgYnkgdGhlIEFuZCBhbmQgbm90IHRoZSBXQUwgcmVwbGF5LiBQZXJoYXBzIHRoYXQncyB0b28g bGF0ZSwgYnV0IGNvdWxkIGl0IGJlIHBvc3NpYmxlIHRvIHNjYW4gdGhlIFdBTCBzZWdtZW50cyB5 b3UgaGF2ZSBhbmQgc2VlIGlmIHRoZXJlIGlzIHJlY29yZCByZWZlcnJpbmcgdG8gdGhvc2UgcGFn ZXMgYmVpbmcgaW5pdGlhbGl6ZWQgb3Igbm90PyBZb3Ugd291bGQgbmVlZCB0byBmaW5kIGEgcmVj b3JkIGxpa2UgdGhhdDoNCltpbnNlcnR8dXBkYXRlfG11bHRpLWluc2VydHxob3RfdXBkYXRlXShp bml0KSAgcmVsICV1LyV1LyV1OyB0aWQgJXUvJXUgdGlkIGlzIHRfY3RpZCByZWZlcnJlZCBpbiB0 aG9zZSB1cHBlciByZXN1bHRzIHlvdSBqdXN0IHNlbnQuIEFuZCB0aGlzIHJlY29yZCBzaG91bGQg bm9ybWFsbHkgYmUgcHJlc2VudCBiZWZvcmUgdGhlIG9uZXMgdGhhdCBjYXVzZWQgdGhlIFBBTklD IHNldHRpbmcgdGhlIHZpc2liaWxpdHkgbWFwIGJpdC4gSWYgdGhhdCdzIG5vdCB0aGUgY2FzZSwg aXQgbWF5IGJlIHBvc3NpYmxlIHRoYXQgdGhlcmUgaXMgYWN0dWFsbHkgYSBidWcgaWYgdGhlIHBh Z2UgaXMgbm90IGZvdW5kIGFzIGJlaW5nIGluaXRpYWxpemVkIHByb3Blcmx5IGZpcnN0LiBBdCBs ZWFzdCB3ZSBhcmUgc3VyZSB0aGF0IHRoZSBjb3JydXB0aW9uIGlzIG5vdCBjb21pbmcgZnJvbSB0 aGUgbWFzdGVyLg0KLS0NCk1pY2hhZWwNCg==
Hi, On 2015-12-29 15:51:32 +0000, Marek.Petr@tieto.com wrote: > I used fresh base backup for slave after both crashes. > Also tried to scan archived wals several hours before last crash and found only the following for string 71566: > > rmgr: Heap2 len (rec/tot): 20/ 52, tx: 0, lsn: 187/987859D8, prev 187/987859A0, bkp: 0000, desc:visible: rel 1663/16422/17216; blk 71566 > rmgr: Heap2 len (rec/tot): 20/ 52, tx: 0, lsn: 187/9CC59020, prev 187/9CC58FE8, bkp: 0000, desc:visible: rel 1663/16422/17220; blk 71566 > rmgr: Heap2 len (rec/tot): 20/ 52, tx: 0, lsn: 187/9E356D98, prev 187/9E356D60, bkp: 0000, desc:visible: rel 1663/16422/23253; blk 71566 Could you detail how you take base backups and how you set them up as new replicas? Master's pg_controldata, the crashed standby's and the standby's backup_label.old would be helpful. Greetings, Andres Freund
U2FtZSBpc3N1ZSBzaW11bGF0ZWQgdmlhIFBJVFIgZnJvbSBiYXNlIGJhY2t1cCB1cCB0byA1IG1p bnV0ZXMgYWZ0ZXIgdGhlIGNyYXNoOg0KDQoyMDE1LTEyLTMwIDA4OjMxOjE5IENFVCBAICBMT0c6 ICBkYXRhYmFzZSBzeXN0ZW0gd2FzIGludGVycnVwdGVkOyBsYXN0IGtub3duIHVwIGF0IDIwMTUt MTItMjEgMjA6MTU6MDIgQ0VUDQoyMDE1LTEyLTMwIDA4OjMxOjE5IENFVCBAICBMT0c6ICBzdGFy dGluZyBwb2ludC1pbi10aW1lIHJlY292ZXJ5IHRvIDIwMTUtMTItMjIgMDA6MzA6MDArMDENCjIw MTUtMTItMzAgMDg6MzE6MTkgQ0VUIEAgIExPRzogIGRhdGFiYXNlIHN5c3RlbSB3YXMgbm90IHBy b3Blcmx5IHNodXQgZG93bjsgYXV0b21hdGljIHJlY292ZXJ5IGluIHByb2dyZXNzDQoyMDE1LTEy LTMwIDA4OjMxOjE5IENFVCBAICBMT0c6ICByZWRvIHN0YXJ0cyBhdCAxODYvRDcyNjA1NDANCjIw MTUtMTItMzAgMDg6MzE6MTkgQ0VUIEAgIExPRzogIGNvbnNpc3RlbnQgcmVjb3Zlcnkgc3RhdGUg cmVhY2hlZCBhdCAxODYvRDgwMDAwMDANCjIwMTUtMTItMzAgMDg6MzE6MTkgQ0VUIEAgIExPRzog IGRhdGFiYXNlIHN5c3RlbSBpcyByZWFkeSB0byBhY2NlcHQgcmVhZCBvbmx5IGNvbm5lY3Rpb25z DQoyMDE1LTEyLTMwIDA4OjMxOjE5IENFVCBAICBMT0c6ICByZXN0b3JlZCBsb2cgZmlsZSAiMDAw MDAwMDEwMDAwMDE4NjAwMDAwMEQ4IiBmcm9tIGFyY2hpdmUNCjIwMTUtMTItMzAgMDg6MzE6MjAg Q0VUIEAgIExPRzogIHJlc3RvcmVkIGxvZyBmaWxlICIwMDAwMDAwMTAwMDAwMTg2MDAwMDAwRDki IGZyb20gYXJjaGl2ZQ0KLy4uLi8NCjIwMTUtMTItMzAgMDg6MzI6NDggQ0VUIEAgIExPRzogIHJl c3RvcmVkIGxvZyBmaWxlICIwMDAwMDAwMTAwMDAwMTg3MDAwMDAwOUQiIGZyb20gYXJjaGl2ZQ0K MjAxNS0xMi0zMCAwODozMjo1MCBDRVQgQCAgTE9HOiAgcmVzdG9yZWQgbG9nIGZpbGUgIjAwMDAw MDAxMDAwMDAxODcwMDAwMDA5RSIgZnJvbSBhcmNoaXZlDQoyMDE1LTEyLTMwIDA4OjMyOjUxIENF VCBAICBXQVJOSU5HOiAgcGFnZSA3MzE3MiBvZiByZWxhdGlvbiBiYXNlLzE2NDIyLzIzMjUzIGlz IHVuaW5pdGlhbGl6ZWQNCjIwMTUtMTItMzAgMDg6MzI6NTEgQ0VUIEAgIENPTlRFWFQ6ICB4bG9n IHJlZG8gdmlzaWJsZTogcmVsIDE2NjMvMTY0MjIvMjMyNTM7IGJsayA3MzE3Mg0KMjAxNS0xMi0z MCAwODozMjo1MSBDRVQgQCAgUEFOSUM6ICBXQUwgY29udGFpbnMgcmVmZXJlbmNlcyB0byBpbnZh bGlkIHBhZ2VzDQoyMDE1LTEyLTMwIDA4OjMyOjUxIENFVCBAICBDT05URVhUOiAgeGxvZyByZWRv IHZpc2libGU6IHJlbCAxNjYzLzE2NDIyLzIzMjUzOyBibGsgNzMxNzINCjIwMTUtMTItMzAgMDg6 MzI6NTEgQ0VUIEAgIExPRzogIHN0YXJ0dXAgcHJvY2VzcyAoUElEIDI5MzI0KSB3YXMgdGVybWlu YXRlZCBieSBzaWduYWwgNjogQWJvcnRlZA0KMjAxNS0xMi0zMCAwODozMjo1MSBDRVQgQCAgTE9H OiAgdGVybWluYXRpbmcgYW55IG90aGVyIGFjdGl2ZSBzZXJ2ZXIgcHJvY2Vzc2VzDQoNClJlZ2Fy ZHMNCk1hcmVrDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBQZXRyIE1hcmVr IA0KU2VudDogVHVlc2RheSwgRGVjZW1iZXIgMjksIDIwMTUgNDo1MiBQTQ0KVG86ICdNaWNoYWVs IFBhcXVpZXInIDxtaWNoYWVsLnBhcXVpZXJAZ21haWwuY29tPg0KQ2M6IFBvc3RncmVTUUwgbWFp bGluZyBsaXN0cyA8cGdzcWwtYnVnc0Bwb3N0Z3Jlc3FsLm9yZz4NClN1YmplY3Q6IFJFOiBbQlVH U10gQlVHICMxMzgyMjogU2xhdmUgdGVybWluYXRlZCAtIFdBTCBjb250YWlucyByZWZlcmVuY2Vz IHRvIGludmFsaWQgcGFnZQ0KDQpJIHVzZWQgZnJlc2ggYmFzZSBiYWNrdXAgZm9yIHNsYXZlIGFm dGVyIGJvdGggY3Jhc2hlcy4NCkFsc28gdHJpZWQgdG8gc2NhbiBhcmNoaXZlZCB3YWxzIHNldmVy YWwgaG91cnMgYmVmb3JlIGxhc3QgY3Jhc2ggYW5kIGZvdW5kIG9ubHkgdGhlIGZvbGxvd2luZyBm b3Igc3RyaW5nIDcxNTY2Og0KDQpybWdyOiBIZWFwMiAgICAgICBsZW4gKHJlYy90b3QpOiAgICAg MjAvICAgIDUyLCB0eDogICAgICAgICAgMCwgbHNuOiAxODcvOTg3ODU5RDgsIHByZXYgMTg3Lzk4 Nzg1OUEwLCBia3A6IDAwMDAsIGRlc2M6IHZpc2libGU6IHJlbCAxNjYzLzE2NDIyLzE3MjE2OyBi bGsgNzE1NjYNCnJtZ3I6IEhlYXAyICAgICAgIGxlbiAocmVjL3RvdCk6ICAgICAyMC8gICAgNTIs IHR4OiAgICAgICAgICAwLCBsc246IDE4Ny85Q0M1OTAyMCwgcHJldiAxODcvOUNDNThGRTgsIGJr cDogMDAwMCwgZGVzYzogdmlzaWJsZTogcmVsIDE2NjMvMTY0MjIvMTcyMjA7IGJsayA3MTU2Ng0K cm1ncjogSGVhcDIgICAgICAgbGVuIChyZWMvdG90KTogICAgIDIwLyAgICA1MiwgdHg6ICAgICAg ICAgIDAsIGxzbjogMTg3LzlFMzU2RDk4LCBwcmV2IDE4Ny85RTM1NkQ2MCwgYmtwOiAwMDAwLCBk ZXNjOiB2aXNpYmxlOiByZWwgMTY2My8xNjQyMi8yMzI1MzsgYmxrIDcxNTY2DQoNClJlZ2FyZHMN Ck1hcmVrDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBNaWNoYWVsIFBhcXVp ZXIgW21haWx0bzptaWNoYWVsLnBhcXVpZXJAZ21haWwuY29tXSANClNlbnQ6IE1vbmRheSwgRGVj ZW1iZXIgMjgsIDIwMTUgNDoyOSBQTQ0KVG86IFBldHIgTWFyZWsgPE1hcmVrLlBldHJAdGlldG8u Y29tPg0KQ2M6IFBvc3RncmVTUUwgbWFpbGluZyBsaXN0cyA8cGdzcWwtYnVnc0Bwb3N0Z3Jlc3Fs Lm9yZz4NClN1YmplY3Q6IFJlOiBbQlVHU10gQlVHICMxMzgyMjogU2xhdmUgdGVybWluYXRlZCAt IFdBTCBjb250YWlucyByZWZlcmVuY2VzIHRvIGludmFsaWQgcGFnZQ0KDQpPbiBNb24sIERlYyAy OCwgMjAxNSBhdCA4OjUwIFBNLCAgPE1hcmVrLlBldHJAdGlldG8uY29tPiB3cm90ZToNCj4gVHJp ZWQgdG8gdXNlIHBhZ2VpbnNwZWN0IG1vZHVsZSBmb3IgYWZmZWN0ZWQgcGFnZXMgZnJvbSBsYXN0 IHR3byBvY2N1cmVuY2VzOg0KPg0KPiAyMDE1LTEyLTE1IDEzOjA1OjM5IENFVCBAICBXQVJOSU5H OiAgcGFnZSA0MzMzMjc1IG9mIHJlbGF0aW9uIA0KPiBiYXNlLzE2NDIyLzE3MjMwIGlzIHVuaW5p dGlhbGl6ZWQNCj4gMjAxNS0xMi0yMiAwMDoyNToxMSBDRVQgQCAgV0FSTklORzogIHBhZ2UgNzE1 NjYgb2YgcmVsYXRpb24gDQo+IGJhc2UvMTY0MjIvMjMyNTMgaXMgdW5pbml0aWFsaXplZA0KPg0K PiBGb2xsb3dpbmcgb3V0cHV0cyBhcmUgdGhlIHNhbWUgZm9yIG1hc3RlciBhbmQgc2xhdmU6DQo+ DQo+IFtyZXN1bHRzXQ0KDQpIbSwgT0suDQoNCj4gTm9uLWRlZmF1bHQgcGFyczoNCj4NCj4gW3Bh cmFtc10NCg0KVGhlcmUgaXMgbm90aGluZyBmaXNoeSBoZXJlLg0KDQo+IFNsYXZlIHJlYnVpbGRl ZCBhbmQgaXQncyBydW5uaW5nIGFsbW9zdCBhIHdlZWsgZm9yIG5vdy4NCg0KSG0uIEhhcyB0aGlz IHNsYXZlIHJlcGxheWVkIHRoZSBzYW1lIFdBTCByZWNvcmRzIGFzIHRoZSBzbGF2ZSB0aGF0IGhh cyBmYWlsZWQgcHJldmlvdXNseT8gT3IgZGlkIGl0IHVzZSBhIGZyZXNoZXIgYmFzZSBiYWNrdXA/ IElmIHRoYXQncyB0aGUgbGF0dGVyIHRoZSBwcm9ibGVtIHdvdWxkIGhhdmUgYmVlbiBmaXhlZCBi eSBpdHNlbGYgZm9yIHRob3NlIHR3byByZWxhdGlvbiBwYWdlcyBhcyB0aGV5IHdvdWxkIGhhdmUg YmVlbiBjb3JyZWN0bHkgY3JlYXRlZCBieSB0aGUgQW5kIGFuZCBub3QgdGhlIFdBTCByZXBsYXku IFBlcmhhcHMgdGhhdCdzIHRvbyBsYXRlLCBidXQgY291bGQgaXQgYmUgcG9zc2libGUgdG8gc2Nh biB0aGUgV0FMIHNlZ21lbnRzIHlvdSBoYXZlIGFuZCBzZWUgaWYgdGhlcmUgaXMgcmVjb3JkIHJl ZmVycmluZyB0byB0aG9zZSBwYWdlcyBiZWluZyBpbml0aWFsaXplZCBvciBub3Q/IFlvdSB3b3Vs ZCBuZWVkIHRvIGZpbmQgYSByZWNvcmQgbGlrZSB0aGF0Og0KW2luc2VydHx1cGRhdGV8bXVsdGkt aW5zZXJ0fGhvdF91cGRhdGVdKGluaXQpICByZWwgJXUvJXUvJXU7IHRpZCAldS8ldSB0aWQgaXMg dF9jdGlkIHJlZmVycmVkIGluIHRob3NlIHVwcGVyIHJlc3VsdHMgeW91IGp1c3Qgc2VudC4gQW5k IHRoaXMgcmVjb3JkIHNob3VsZCBub3JtYWxseSBiZSBwcmVzZW50IGJlZm9yZSB0aGUgb25lcyB0 aGF0IGNhdXNlZCB0aGUgUEFOSUMgc2V0dGluZyB0aGUgdmlzaWJpbGl0eSBtYXAgYml0LiBJZiB0 aGF0J3Mgbm90IHRoZSBjYXNlLCBpdCBtYXkgYmUgcG9zc2libGUgdGhhdCB0aGVyZSBpcyBhY3R1 YWxseSBhIGJ1ZyBpZiB0aGUgcGFnZSBpcyBub3QgZm91bmQgYXMgYmVpbmcgaW5pdGlhbGl6ZWQg cHJvcGVybHkgZmlyc3QuIEF0IGxlYXN0IHdlIGFyZSBzdXJlIHRoYXQgdGhlIGNvcnJ1cHRpb24g aXMgbm90IGNvbWluZyBmcm9tIHRoZSBtYXN0ZXIuDQotLQ0KTWljaGFlbA0K
Hi,=0A= =0A= I created new replica (original one crashed 2015-12-15 13:05:39 CET) via: = =0A= =0A= $ pg_basebackup -h <master_IP> -D $PGDATA -R -P -U replication --xlog-metho= d=3Dstream=0A= $ cat $PGDATA/recovery.conf =0A= standby_mode =3D 'on'=0A= primary_conninfo =3D 'user=3Dreplication password=3D*** host=3D<master_IP> = port=3D5432 sslmode=3Dprefer sslcompression=3D1 krbsrvname=3Dpostgres'=0A= # service postgresql-9.4 start=0A= =0A= This new replica crashed 2015-12-22 00:25:11 CET.=0A= =0A= Ordinary base backups are done by:=0A= =0A= /usr/pgsql-9.4/bin/pg_basebackup -h <master_IP> -D <local_pg_basebackup_dir= > -Ft -z -v -Xf -U postgres=0A= =0A= =0A= Running Master pg_controldata:=0A= =0A= pg_control version number: 942=0A= Catalog version number: 201409291=0A= Database system identifier: 6225208935473175762=0A= Database cluster state: in production=0A= pg_control last modified: Wed 30 Dec 2015 08:00:02 PM CET=0A= Latest checkpoint location: 191/2E08DC90=0A= Prior checkpoint location: 191/2E0829E0=0A= Latest checkpoint's REDO location: 191/2E08DC58=0A= Latest checkpoint's REDO WAL file: 00000001000001910000002E=0A= Latest checkpoint's TimeLineID: 1=0A= Latest checkpoint's PrevTimeLineID: 1=0A= Latest checkpoint's full_page_writes: on=0A= Latest checkpoint's NextXID: 0/370154188=0A= Latest checkpoint's NextOID: 4803404=0A= Latest checkpoint's NextMultiXactId: 161565=0A= Latest checkpoint's NextMultiOffset: 410484=0A= Latest checkpoint's oldestXID: 187316348=0A= Latest checkpoint's oldestXID's DB: 16422=0A= Latest checkpoint's oldestActiveXID: 370154188=0A= Latest checkpoint's oldestMultiXid: 1=0A= Latest checkpoint's oldestMulti's DB: 16422=0A= Time of latest checkpoint: Wed 30 Dec 2015 08:00:01 PM CET=0A= Fake LSN counter for unlogged rels: 0/1=0A= Minimum recovery ending location: 0/0=0A= Min recovery ending loc's timeline: 0=0A= Backup start location: 0/0=0A= Backup end location: 0/0=0A= End-of-backup record required: no=0A= Current wal_level setting: hot_standby=0A= Current wal_log_hints setting: off=0A= Current max_connections setting: 600=0A= Current max_worker_processes setting: 8=0A= Current max_prepared_xacts setting: 0=0A= Current max_locks_per_xact setting: 64=0A= Maximum data alignment: 8=0A= Database block size: 8192=0A= Blocks per segment of large relation: 131072=0A= WAL block size: 8192=0A= Bytes per WAL segment: 16777216=0A= Maximum length of identifiers: 64=0A= Maximum columns in an index: 32=0A= Maximum size of a TOAST chunk: 1996=0A= Size of a large-object chunk: 2048=0A= Date/time type storage: 64-bit integers=0A= Float4 argument passing: by value=0A= Float8 argument passing: by value=0A= Data page checksum version: 0=0A= =0A= =0A= Running Slave pg_controldata:=0A= =0A= pg_control version number: 942=0A= Catalog version number: 201409291=0A= Database system identifier: 6225208935473175762=0A= Database cluster state: in archive recovery=0A= pg_control last modified: Wed 30 Dec 2015 08:00:40 PM CET=0A= Latest checkpoint location: 191/2E08DC90=0A= Prior checkpoint location: 191/2E0829E0=0A= Latest checkpoint's REDO location: 191/2E08DC58=0A= Latest checkpoint's REDO WAL file: 00000001000001910000002E=0A= Latest checkpoint's TimeLineID: 1=0A= Latest checkpoint's PrevTimeLineID: 1=0A= Latest checkpoint's full_page_writes: on=0A= Latest checkpoint's NextXID: 0/370154188=0A= Latest checkpoint's NextOID: 4803404=0A= Latest checkpoint's NextMultiXactId: 161565=0A= Latest checkpoint's NextMultiOffset: 410484=0A= Latest checkpoint's oldestXID: 187316348=0A= Latest checkpoint's oldestXID's DB: 16422=0A= Latest checkpoint's oldestActiveXID: 370154188=0A= Latest checkpoint's oldestMultiXid: 1=0A= Latest checkpoint's oldestMulti's DB: 16422=0A= Time of latest checkpoint: Wed 30 Dec 2015 08:00:01 PM CET=0A= Fake LSN counter for unlogged rels: 0/1=0A= Minimum recovery ending location: 191/2E093C68=0A= Min recovery ending loc's timeline: 1=0A= Backup start location: 0/0=0A= Backup end location: 0/0=0A= End-of-backup record required: no=0A= Current wal_level setting: hot_standby=0A= Current wal_log_hints setting: off=0A= Current max_connections setting: 600=0A= Current max_worker_processes setting: 8=0A= Current max_prepared_xacts setting: 0=0A= Current max_locks_per_xact setting: 64=0A= Maximum data alignment: 8=0A= Database block size: 8192=0A= Blocks per segment of large relation: 131072=0A= WAL block size: 8192=0A= Bytes per WAL segment: 16777216=0A= Maximum length of identifiers: 64=0A= Maximum columns in an index: 32=0A= Maximum size of a TOAST chunk: 1996=0A= Size of a large-object chunk: 2048=0A= Date/time type storage: 64-bit integers=0A= Float4 argument passing: by value=0A= Float8 argument passing: by value=0A= Data page checksum version: 0=0A= =0A= =0A= Slave pg_controldata after second PITR attempt:=0A= =0A= pg_control version number: 942=0A= Catalog version number: 201409291=0A= Database system identifier: 6225208935473175762=0A= Database cluster state: shut down in recovery=0A= pg_control last modified: Wed 30 Dec 2015 08:53:51 PM CET=0A= Latest checkpoint location: 187/9DA56658=0A= Prior checkpoint location: 186/D7000060=0A= Latest checkpoint's REDO location: 187/9D647E30=0A= Latest checkpoint's REDO WAL file: 00000001000001870000009D=0A= Latest checkpoint's TimeLineID: 1=0A= Latest checkpoint's PrevTimeLineID: 1=0A= Latest checkpoint's full_page_writes: on=0A= Latest checkpoint's NextXID: 0/367008737=0A= Latest checkpoint's NextOID: 4721484=0A= Latest checkpoint's NextMultiXactId: 160851=0A= Latest checkpoint's NextMultiOffset: 409011=0A= Latest checkpoint's oldestXID: 187316348=0A= Latest checkpoint's oldestXID's DB: 16422=0A= Latest checkpoint's oldestActiveXID: 367008737=0A= Latest checkpoint's oldestMultiXid: 1=0A= Latest checkpoint's oldestMulti's DB: 16422=0A= Time of latest checkpoint: Tue 22 Dec 2015 12:21:40 AM CET=0A= Fake LSN counter for unlogged rels: 0/1=0A= Minimum recovery ending location: 187/9FECB9A0=0A= Min recovery ending loc's timeline: 1=0A= Backup start location: 0/0=0A= Backup end location: 0/0=0A= End-of-backup record required: no=0A= Current wal_level setting: hot_standby=0A= Current wal_log_hints setting: off=0A= Current max_connections setting: 600=0A= Current max_worker_processes setting: 8=0A= Current max_prepared_xacts setting: 0=0A= Current max_locks_per_xact setting: 64=0A= Maximum data alignment: 8=0A= Database block size: 8192=0A= Blocks per segment of large relation: 131072=0A= WAL block size: 8192=0A= Bytes per WAL segment: 16777216=0A= Maximum length of identifiers: 64=0A= Maximum columns in an index: 32=0A= Maximum size of a TOAST chunk: 1996=0A= Size of a large-object chunk: 2048=0A= Date/time type storage: 64-bit integers=0A= Float4 argument passing: by value=0A= Float8 argument passing: by value=0A= Data page checksum version: 0=0A= =0A= =0A= # cat backup_label.old =0A= START WAL LOCATION: 186/D7000028 (file 0000000100000186000000D7)=0A= CHECKPOINT LOCATION: 186/D7000060=0A= BACKUP METHOD: streamed=0A= BACKUP FROM: master=0A= START TIME: 2015-12-21 19:00:03 CET=0A= LABEL: pg_basebackup base backup=0A= =0A= =0A= Slave's pg_controldata above after PITR has been created after second recov= ery attempt.=0A= To be honest I don't understand why this second PITR finished without PANIC= when previous attempt from the same base backup and archived wals failed.= =0A= See logs details below:=0A= =0A= First (crashed) PITR:=0A= =0A= 2015-12-30 08:31:19 CET @ LOG: database system was interrupted; last know= n up at 2015-12-21 20:15:02 CET=0A= 2015-12-30 08:31:19 CET @ LOG: starting point-in-time recovery to 2015-12= -22 00:30:00+01=0A= 2015-12-30 08:31:19 CET @ LOG: database system was not properly shut down= ; automatic recovery in progress=0A= 2015-12-30 08:31:19 CET @ LOG: redo starts at 186/D7260540=0A= 2015-12-30 08:31:19 CET @ LOG: consistent recovery state reached at 186/D= 8000000=0A= 2015-12-30 08:31:19 CET @ LOG: database system is ready to accept read on= ly connections=0A= 2015-12-30 08:31:19 CET @ LOG: restored log file "0000000100000186000000D= 8" from archive=0A= 2015-12-30 08:31:20 CET @ LOG: restored log file "0000000100000186000000D= 9" from archive=0A= /.../=0A= 2015-12-30 08:32:48 CET @ LOG: restored log file "00000001000001870000009= D" from archive=0A= 2015-12-30 08:32:50 CET @ LOG: restored log file "00000001000001870000009= E" from archive=0A= 2015-12-30 08:32:51 CET @ WARNING: page 73172 of relation base/16422/2325= 3 is uninitialized=0A= 2015-12-30 08:32:51 CET @ CONTEXT: xlog redo visible: rel 1663/16422/2325= 3; blk 73172=0A= 2015-12-30 08:32:51 CET @ PANIC: WAL contains references to invalid pages= =0A= 2015-12-30 08:32:51 CET @ CONTEXT: xlog redo visible: rel 1663/16422/2325= 3; blk 73172=0A= 2015-12-30 08:32:51 CET @ LOG: startup process (PID 29324) was terminated= by signal 6: Aborted=0A= 2015-12-30 08:32:51 CET @ LOG: terminating any other active server proces= ses=0A= =0A= For this one unfortunatelly I don't have pg_controldata.=0A= =0A= Second one:=0A= =0A= 2015-12-30 20:42:39 CET @ LOG: database system was interrupted; last know= n up at 2015-12-21 20:15:02 CET=0A= 2015-12-30 20:42:39 CET @ LOG: starting point-in-time recovery to 2015-12= -22 00:30:00+01=0A= 2015-12-30 20:42:39 CET @ LOG: restored log file "0000000100000186000000D= 7" from archive=0A= 2015-12-30 20:42:39 CET @ LOG: redo starts at 186/D7000028=0A= 2015-12-30 20:42:39 CET @ LOG: consistent recovery state reached at 186/D= 7265F80=0A= 2015-12-30 20:42:39 CET @ LOG: database system is ready to accept read on= ly connections=0A= 2015-12-30 20:42:40 CET @ LOG: restored log file "0000000100000186000000D= 8" from archive=0A= 2015-12-30 20:42:40 CET @ LOG: restored log file "0000000100000186000000D= 9" from archive=0A= /.../=0A= 2015-12-30 20:44:03 CET @ LOG: restored log file "00000001000001870000009= D" from archive=0A= 2015-12-30 20:44:05 CET @ LOG: restored log file "00000001000001870000009= E" from archive=0A= 2015-12-30 20:44:08 CET @ LOG: restored log file "00000001000001870000009= F" from archive=0A= 2015-12-30 20:44:10 CET @ LOG: recovery stopping before commit of transac= tion 367008747, time 2015-12-22 00:30:01.876776+01=0A= 2015-12-30 20:44:10 CET @ LOG: recovery has paused=0A= 2015-12-30 20:44:10 CET @ HINT: Execute pg_xlog_replay_resume() to contin= ue.=0A= =0A= Regards=0A= Marek=0A= ________________________________________=0A= From: Andres Freund <andres@anarazel.de>=0A= Sent: Wednesday, December 30, 2015 12:49 PM=0A= To: Petr Marek=0A= Cc: michael.paquier@gmail.com; pgsql-bugs@postgresql.org=0A= Subject: Re: [BUGS] BUG #13822: Slave terminated - WAL contains references = to invalid page=0A= =0A= Hi,=0A= =0A= On 2015-12-29 15:51:32 +0000, Marek.Petr@tieto.com wrote:=0A= > I used fresh base backup for slave after both crashes.=0A= > Also tried to scan archived wals several hours before last crash and foun= d only the following for string 71566:=0A= >=0A= > rmgr: Heap2 len (rec/tot): 20/ 52, tx: 0, lsn: 187/= 987859D8, prev 187/987859A0, bkp: 0000, desc: visible: rel 1663/16422/17216= ; blk 71566=0A= > rmgr: Heap2 len (rec/tot): 20/ 52, tx: 0, lsn: 187/= 9CC59020, prev 187/9CC58FE8, bkp: 0000, desc: visible: rel 1663/16422/17220= ; blk 71566=0A= > rmgr: Heap2 len (rec/tot): 20/ 52, tx: 0, lsn: 187/= 9E356D98, prev 187/9E356D60, bkp: 0000, desc: visible: rel 1663/16422/23253= ; blk 71566=0A= =0A= Could you detail how you take base backups and how you set them up as=0A= new replicas? Master's pg_controldata, the crashed standby's and the=0A= standby's backup_label.old would be helpful.=0A= =0A= Greetings,=0A= =0A= Andres Freund=
Re: BUG #13822: Slave terminated - WAL contains references to invalid page
From
Michael Paquier
Date:
On Thu, Dec 31, 2015 at 5:20 AM, <Marek.Petr@tieto.com> wrote: > To be honest I don't understand why this second PITR finished without PANIC when previous attempt from the same base backupand archived wals failed. I have no clue here either as the first PITR failed while the second completed correctly, replaying the same content. It seems that this relation block got corrupted in some way because of an issue with memory and/or disk, or presumably in the WAL segment itself. I would discard the possibility of a bug in the WAL replay based on this evidence... Is there actually something different in the way you ran the replay in both cases? Like on different servers? -- Michael
Tm8uIEJvdGggY2FzZXMgcmFuIGluIHRoZSBzYW1lIHdheSBhbmQgc2FtZSBtYWNoaW5lcy4NCg0K UmVnYXJkcw0KTWFyZWsNCg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IE1pY2hh ZWwgUGFxdWllciBbbWFpbHRvOm1pY2hhZWwucGFxdWllckBnbWFpbC5jb21dIA0KU2VudDogVHVl c2RheSwgSmFudWFyeSAwNSwgMjAxNiA4OjUzIEFNDQpUbzogUGV0ciBNYXJlayA8TWFyZWsuUGV0 ckB0aWV0by5jb20+DQpDYzogQW5kcmVzIEZyZXVuZCA8YW5kcmVzQGFuYXJhemVsLmRlPjsgUG9z dGdyZVNRTCBtYWlsaW5nIGxpc3RzIDxwZ3NxbC1idWdzQHBvc3RncmVzcWwub3JnPg0KU3ViamVj dDogUmU6IFtCVUdTXSBCVUcgIzEzODIyOiBTbGF2ZSB0ZXJtaW5hdGVkIC0gV0FMIGNvbnRhaW5z IHJlZmVyZW5jZXMgdG8gaW52YWxpZCBwYWdlDQoNCk9uIFRodSwgRGVjIDMxLCAyMDE1IGF0IDU6 MjAgQU0sICA8TWFyZWsuUGV0ckB0aWV0by5jb20+IHdyb3RlOg0KPiBUbyBiZSBob25lc3QgSSBk b24ndCB1bmRlcnN0YW5kIHdoeSB0aGlzIHNlY29uZCBQSVRSIGZpbmlzaGVkIHdpdGhvdXQgUEFO SUMgd2hlbiBwcmV2aW91cyBhdHRlbXB0IGZyb20gdGhlIHNhbWUgYmFzZSBiYWNrdXAgYW5kIGFy Y2hpdmVkIHdhbHMgZmFpbGVkLg0KDQpJIGhhdmUgbm8gY2x1ZSBoZXJlIGVpdGhlciBhcyB0aGUg Zmlyc3QgUElUUiBmYWlsZWQgd2hpbGUgdGhlIHNlY29uZCBjb21wbGV0ZWQgY29ycmVjdGx5LCBy ZXBsYXlpbmcgdGhlIHNhbWUgY29udGVudC4gSXQgc2VlbXMgdGhhdCB0aGlzIHJlbGF0aW9uIGJs b2NrIGdvdCBjb3JydXB0ZWQgaW4gc29tZSB3YXkgYmVjYXVzZSBvZiBhbiBpc3N1ZSB3aXRoIG1l bW9yeSBhbmQvb3IgZGlzaywgb3IgcHJlc3VtYWJseSBpbiB0aGUgV0FMIHNlZ21lbnQgaXRzZWxm LiBJIHdvdWxkIGRpc2NhcmQgdGhlIHBvc3NpYmlsaXR5IG9mIGEgYnVnIGluIHRoZSBXQUwgcmVw bGF5IGJhc2VkIG9uIHRoaXMgZXZpZGVuY2UuLi4gSXMgdGhlcmUgYWN0dWFsbHkgc29tZXRoaW5n IGRpZmZlcmVudCBpbiB0aGUgd2F5IHlvdSByYW4gdGhlIHJlcGxheSBpbiBib3RoIGNhc2VzPyBM aWtlIG9uIGRpZmZlcmVudCBzZXJ2ZXJzPw0KLS0NCk1pY2hhZWwNCg==