Re: Occasional lengthy locking causing stalling on commit - Mailing list pgsql-general

From Ben Hoskings
Subject Re: Occasional lengthy locking causing stalling on commit
Date
Msg-id CACTv1AKav4bHjYVEzffrzLh0S1YiqftiM4J7VDfWnvSQHi_RVQ@mail.gmail.com
Whole thread Raw
In response to Re: Occasional lengthy locking causing stalling on commit  (Ben Hoskings <ben@hoskings.net>)
List pgsql-general
G'day all,

A quick follow-up on this issue for interest's sake. The stalling we
were seeing turned out to be a Cloud SQL issue and not related to our
listen/notify usage.

Cloud SQL has an automatic storage increase process that resizes the
underlying disk as required to account for cluster growth. As it turns
out that process occasionally causes I/O to stall for a brief window.
https://cloud.google.com/sql/docs/postgres/instance-settings#automatic-storage-increase-2ndgen

The workaround supplied by Google is to manually provision slack
storage in larger increments to prevent the more frequent automatic
increases, which happen 25GB at a time on a large cluster.

We didn't make the connection because disk resize events are not
visible in any logs; Google Support found the issue by correlating the
timestamps of our observed outages with their internal logs.

Hopefully this is useful for someone else. Thanks again for your help
Tom - your advice on listen/notify locking on commit was very useful
despite not being the cause in this case.

Cheers
Ben

On Mon, 1 Feb 2021 at 12:33, Ben Hoskings <ben@hoskings.net> wrote:
>
> On Mon, 1 Feb 2021 at 10:33, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >
> > One thing that just occurred to me is that you might find it
> > interesting to keep tabs on what's in the $PGDATA/pg_notify
> > directory.  Do the performance burps correspond to transitory
> > peaks in the amount of data there?  Or (grasping at straws here...)
> > wraparound of the file names back to 0000?
>
> We don't have filesystem access on Cloud SQL - the downside of the
> managed route :)
>
> It sounds like it might be time to bump the pg13 upgrade up the TODO list.
>
> Cheers
> Ben



pgsql-general by date:

Previous
From: goldgraeber-werbetechnik@t-online.de
Date:
Subject: Re: unicode match normal forms
Next
From: Hannes Kühtreiber
Date:
Subject: Logical Replication: SELECT pg_catalog.set_config Statement