Re: 9.6.11- could not truncate directory "pg_serial": apparent wraparound - Mailing list pgsql-general

From Pavel Suderevsky
Subject Re: 9.6.11- could not truncate directory "pg_serial": apparent wraparound
Date
Msg-id CAEBTBzt24tEsCkupOCxUvy=hnvMnZ59aXECoMAOyAO6iFmhTBg@mail.gmail.com
Whole thread Raw
In response to 9.6.11- could not truncate directory "pg_serial": apparent wraparound  (Pavel Suderevsky <psuderevsky@gmail.com>)
Responses Re: 9.6.11- could not truncate directory "pg_serial": apparent wraparound  (Thomas Munro <thomas.munro@gmail.com>)
List pgsql-general
Guys, still need your help.

Previous night:
2019-04-05 00:35:04 UTC  LOG:  could not truncate directory "pg_serial": apparent wraparound
2019-04-05 00:40:04 UTC  LOG:  could not truncate directory "pg_serial": apparent wraparound
(2 checkpoints)

It turned that I have some problem with performance related to predicate locking on this platform.
A lot of long prepared statements with the SerializableXactHashLock and predicate_lock_manager wait_events followed by high CPU usage happened during 00:30 and 00:45. During this period there were 55k pred locks granted at max and 30k in average. Probably because of high CPU usage some statements were spending a lot of time in bind/parse steps.

Probably if you advise me what could cause "pg_serial": apparent wraparound messages I would have more chances to handle all the performance issues.
 
Thank you!
--
Pavel Suderevsky

пн, 11 мар. 2019 г. в 19:09, Pavel Suderevsky <psuderevsky@gmail.com>:
Hi,

PG: 9.6.11
OS: CentOS
Env: AWS EC2

I've faced the following exceptions in PostgreSQL server log:
> could not truncate directory "pg_serial": apparent wraparound
Sometimes it repeats every 5 min and the longest period was 40 min.

In fact, I can't find any suspicious events happening that periods. 
pg_wait_sampling didn't catch any events, no long queries (more than 60s), Autovacuum workers or transactions in "idle in transaction" state were in action at this time.

>You should not see the errors you are reporting nor
>the warning I mentioned unless a serializable transaction remains
>active long enough for about 1 billion transaction IDs to be
>consumed.

Database age now is just 18.5 millions of transactions.

Server has two standbys (sync and async), hot_standby_feedback is off.

Please advice what I can do to find a reason of these exceptions.

pgsql-general by date:

Previous
From: rihad
Date:
Subject: SIGTERM/SIGINT master/slave behavior
Next
From: senor
Date:
Subject: pg_upgrade --jobs