Thread: Materialized view refreshing problem

Materialized view refreshing problem

From
Hellen Jiang
Date:

Hi All,

 

We have one business critical materialized view( let’s called materialized_view_1) which contains around 16M records.

This materialized view is refreshed CONCURRENTLY every 1 minute, and got refreshed normally (without CONCURRENTLY) every 6 hours.

 

But from time and time, we found that it took forever to refresh materialized view( normally or concurrently), and there is no response to query from this materialized view.

In this case, we have to create new materialized view(materialized_view_1_new) with exactly the same definition, and the new materialized view(materialized_view_1_new) works well and the refresh finishes in 15 seconds.

And after about 1 months, we have to do this again: I mean use freshly new materialized view to replace the old one.

 

It looks like there are too much garbage in the old materialized view as time going.

Any idea how we can fix this materialized view refreshing problem?

 

Thanks

 

Hellen Jiang

Re: Materialized view refreshing problem

From
Tom Lane
Date:
Hellen Jiang <hjiang@federatedwireless.com> writes:
> But from time and time, we found that it took forever to refresh materialized view( normally or concurrently), and
thereis no response to query from this materialized view. 

That sounds like a locking problem.  Have you looked into pg_locks
while this is happening, to see what may be waiting on what?

            regards, tom lane



Re: Materialized view refreshing problem

From
Hellen Jiang
Date:

When the materialized view was refreshing concurrently, I saw the pg_lock like this: there was no response within 30 seconds when the api calls this query, and we got api time out.

sasanalytics=> select relation::regclass, * from pg_locks where not granted;

                     relation                      | locktype | database |  relation  | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction |  pid  |     mode      | granted | fastpath 

---------------------------------------------------+----------+----------+------------+------+-------+------------+---------------+---------+-------+----------+--------------------+-------+---------------+---------+----------

 mvw_cbsd_status_grants_lessinfo_active_softmarker | relation |    16401 | 1886512426 |      |       |            |               |         |       |          | 236/858            | 17332 | ExclusiveLock | f       | f

(1 row)

 

When the materialized view was refreshing normally( without concurrently), I saw the following pg_locks: there was no response within 30 seconds when the api calls this query, and we got api time out.

sasanalytics=> select relation::regclass, * from pg_locks where not granted;

                     relation                      | locktype | database |  relation  | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction |  pid  |        mode         | granted | fastpath 

---------------------------------------------------+----------+----------+------------+------+-------+------------+---------------+---------+-------+----------+--------------------+-------+---------------------+---------+----------

 mvw_cbsd_status_grants_lessinfo_active_softmarker | relation |    16401 | 1886512426 |      |       |            |               |         |       |          | 245/1163           | 15932 | AccessExclusiveLock | f       | f

(1 row)

 

 

From: Tom Lane <tgl@sss.pgh.pa.us>
Date: Thursday, August 24, 2023 at 3:02 PM
To: Hellen Jiang <hjiang@federatedwireless.com>
Cc: pgsql-general@lists.postgresql.org <pgsql-general@lists.postgresql.org>
Subject: Re: Materialized view refreshing problem

Hellen Jiang <hjiang@federatedwireless.com> writes:
> But from time and time, we found that it took forever to refresh materialized view( normally or concurrently), and there is no response to query from this materialized view.

That sounds like a locking problem.  Have you looked into pg_locks
while this is happening, to see what may be waiting on what?

                        regards, tom lane

Re: Materialized view refreshing problem

From
Tom Lane
Date:
Hellen Jiang <hjiang@federatedwireless.com> writes:
> When the materialized view was refreshing concurrently, I saw the pg_lock like this: there was no response within 30
secondswhen the api calls this query, and we got api time out. 

Well, that seems to confirm my idea that the REFRESH is blocked behind
somebody else's lock.  But you need to look to see which session is
holding a lock on mvw_cbsd_status_grants_lessinfo_active_softmarker
and what it's doing.

            regards, tom lane