Thread: [HACKERS] Moving relation extension locks out of heavyweight lock manager

[HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

11 May 2017, 06:39:03

Hi all,

Currently, the relation extension lock is implemented using
heavyweight lock manager and almost functions (except for
brin_page_cleanup) using LockRelationForExntesion use it with
ExclusiveLock mode. But actually it doesn't need multiple lock modes
or deadlock detection or any of the other functionality that the
heavyweight lock manager provides. I think It's enough to use
something like LWLock. So I'd like to propose to change relation
extension lock management so that it works using LWLock instead.

Attached draft patch makes relation extension locks uses LWLock rather
than heavyweight lock manager, using by shared hash table storing
information of the relation extension lock. The basic idea is that we
add hash table in shared memory for relation extension locks and each
hash entry is LWLock struct. Whenever the process wants to acquire
relation extension locks, it searches appropriate LWLock entry in hash
table and acquire it. The process can remove a hash entry when
unlocking it if nobody is holding and waiting it.

This work would be helpful not only for existing workload but also
future works like some parallel utility commands, which is discussed
on other threads[1]. At least for parallel vacuum, this feature helps
to solve issue that the implementation of parallel vacuum has.

I ran pgbench for 10 min three times(scale factor is 5000), here is a
performance measurement result.

clients TPS(HEAD) TPS(Patched)
4 2092.612 2031.277
8 3153.732 3046.789
16 4562.072 4625.419
32 6439.391 6479.526
64 7767.364 7779.636
100 7917.173 7906.567

* 16 core Xeon E5620 2.4GHz
* 32 GB RAM
* ioDrive

In current implementation, it seems there is no performance degradation so far.
Please give me feedback.

[1]
* Block level parallel vacuum WIP
<https://www.postgresql.org/message-id/CAD21AoD1xAqp4zK-Vi1cuY3feq2oO8HcpJiz32UDUfe0BE31Xw%40mail.gmail.com>
* CREATE TABLE with parallel workers, 10.0?
<https://www.postgresql.org/message-id/CAFBoRzeoDdjbPV4riCE%2B2ApV%2BY8nV4HDepYUGftm5SuKWna3rQ%40mail.gmail.com>
* utility commands benefiting from parallel plan
<https://www.postgresql.org/message-id/CAJrrPGcY3SZa40vU%2BR8d8dunXp9JRcFyjmPn2RF9_4cxjHd7uA%40mail.gmail.com>

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Moving_extension_lock_out_of_heavyweight_lock_v1.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

12 May 2017, 07:54:48

On Wed, May 10, 2017 at 8:39 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> Currently, the relation extension lock is implemented using
> heavyweight lock manager and almost functions (except for
> brin_page_cleanup) using LockRelationForExntesion use it with
> ExclusiveLock mode. But actually it doesn't need multiple lock modes
> or deadlock detection or any of the other functionality that the
> heavyweight lock manager provides. I think It's enough to use
> something like LWLock. So I'd like to propose to change relation
> extension lock management so that it works using LWLock instead.

That's not a good idea because it'll make the code that executes while
holding that lock noninterruptible.

Possibly something based on condition variables would work better.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Tom Lane

Date:

12 May 2017, 09:44:04

Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, May 10, 2017 at 8:39 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> ... I'd like to propose to change relation
>> extension lock management so that it works using LWLock instead.

> That's not a good idea because it'll make the code that executes while
> holding that lock noninterruptible.

Is that really a problem?  We typically only hold it over one kernel call,
which ought to be noninterruptible anyway.  Also, the CheckpointLock is
held for far longer, and we've not heard complaints about that one.

I'm slightly suspicious of the claim that we don't need deadlock
detection.  There are places that e.g. touch FSM while holding this
lock.  It might be all right but it needs close review, not just an
assertion that it's not a problem.
        regards, tom lane

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

13 May 2017, 17:19:52

On Thu, May 11, 2017 at 6:09 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> This work would be helpful not only for existing workload but also
> future works like some parallel utility commands, which is discussed
> on other threads[1]. At least for parallel vacuum, this feature helps
> to solve issue that the implementation of parallel vacuum has.
>
> I ran pgbench for 10 min three times(scale factor is 5000), here is a
> performance measurement result.
>
> clients   TPS(HEAD)   TPS(Patched)
> 4           2092.612       2031.277
> 8           3153.732       3046.789
> 16         4562.072       4625.419
> 32         6439.391       6479.526
> 64         7767.364       7779.636
> 100       7917.173       7906.567
>
> * 16 core Xeon E5620 2.4GHz
> * 32 GB RAM
> * ioDrive
>
> In current implementation, it seems there is no performance degradation so far.
>

I think it is good to check pgbench, but we should do tests of the
bulk load as this lock is stressed during such a workload.  Some of
the tests we have done when we have improved the performance of bulk
load can be found in an e-mail [1].

[1] -
https://www.postgresql.org/message-id/CAFiTN-tkX6gs-jL8VrPxg6OG9VUAKnObUq7r7pWQqASzdF5OwA%40mail.gmail.com

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

13 May 2017, 17:27:45

On Fri, May 12, 2017 at 9:14 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Wed, May 10, 2017 at 8:39 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> ... I'd like to propose to change relation
>>> extension lock management so that it works using LWLock instead.
>
>> That's not a good idea because it'll make the code that executes while
>> holding that lock noninterruptible.
>
> Is that really a problem?  We typically only hold it over one kernel call,
> which ought to be noninterruptible anyway.
>

During parallel bulk load operations, I think we hold it over multiple
kernel calls.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

16 May 2017, 14:40:05

On Sat, May 13, 2017 at 8:19 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Thu, May 11, 2017 at 6:09 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> This work would be helpful not only for existing workload but also
>> future works like some parallel utility commands, which is discussed
>> on other threads[1]. At least for parallel vacuum, this feature helps
>> to solve issue that the implementation of parallel vacuum has.
>>
>> I ran pgbench for 10 min three times(scale factor is 5000), here is a
>> performance measurement result.
>>
>> clients   TPS(HEAD)   TPS(Patched)
>> 4           2092.612       2031.277
>> 8           3153.732       3046.789
>> 16         4562.072       4625.419
>> 32         6439.391       6479.526
>> 64         7767.364       7779.636
>> 100       7917.173       7906.567
>>
>> * 16 core Xeon E5620 2.4GHz
>> * 32 GB RAM
>> * ioDrive
>>
>> In current implementation, it seems there is no performance degradation so far.
>>
>
> I think it is good to check pgbench, but we should do tests of the
> bulk load as this lock is stressed during such a workload.  Some of
> the tests we have done when we have improved the performance of bulk
> load can be found in an e-mail [1].
>

Thank you for sharing.

I've measured using two test scripts attached on that thread. Here is result.

* Copy test script
Client    HEAD     Patched
4          452.60     455.53
8          561.74     561.09
16        592.50     592.21
32        602.53     599.53
64        605.01     606.42

* Insert test script
Client    HEAD     Patched
4          159.04     158.44
8          169.41     169.69
16        177.11     178.14
32        182.14     181.99
64        182.11     182.73

It seems there is no performance degradation so far.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

16 May 2017, 22:30:03

On Sat, May 13, 2017 at 7:27 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Fri, May 12, 2017 at 9:14 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:
>>> On Wed, May 10, 2017 at 8:39 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>>> ... I'd like to propose to change relation
>>>> extension lock management so that it works using LWLock instead.
>>
>>> That's not a good idea because it'll make the code that executes while
>>> holding that lock noninterruptible.
>>
>> Is that really a problem?  We typically only hold it over one kernel call,
>> which ought to be noninterruptible anyway.
>
> During parallel bulk load operations, I think we hold it over multiple
> kernel calls.

We do.  Also, RelationGetNumberOfBlocks() is not necessarily only one
kernel call, no?  Nor is vm_extend.

Also, it's not just the backend doing the filesystem operation that's
non-interruptible, but also any waiters, right?

Maybe this isn't a big problem, but it does seem to be that it would
be better to avoid it if we can.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

19 May 2017, 08:12:23

On Wed, May 17, 2017 at 1:30 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sat, May 13, 2017 at 7:27 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> On Fri, May 12, 2017 at 9:14 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> Robert Haas <robertmhaas@gmail.com> writes:
>>>> On Wed, May 10, 2017 at 8:39 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>>>> ... I'd like to propose to change relation
>>>>> extension lock management so that it works using LWLock instead.
>>>
>>>> That's not a good idea because it'll make the code that executes while
>>>> holding that lock noninterruptible.
>>>
>>> Is that really a problem?  We typically only hold it over one kernel call,
>>> which ought to be noninterruptible anyway.
>>
>> During parallel bulk load operations, I think we hold it over multiple
>> kernel calls.
>
> We do.  Also, RelationGetNumberOfBlocks() is not necessarily only one
> kernel call, no?  Nor is vm_extend.

Yeah, these functions could call more than one kernel calls while
holding extension lock.

> Also, it's not just the backend doing the filesystem operation that's
> non-interruptible, but also any waiters, right?
>
> Maybe this isn't a big problem, but it does seem to be that it would
> be better to avoid it if we can.
>

I agree to change it to be interruptible for more safety.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

21 June 2017, 21:03:51

On Fri, May 19, 2017 at 11:12 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> On Wed, May 17, 2017 at 1:30 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Sat, May 13, 2017 at 7:27 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> On Fri, May 12, 2017 at 9:14 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>> Robert Haas <robertmhaas@gmail.com> writes:
>>>>> On Wed, May 10, 2017 at 8:39 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>>>>> ... I'd like to propose to change relation
>>>>>> extension lock management so that it works using LWLock instead.
>>>>
>>>>> That's not a good idea because it'll make the code that executes while
>>>>> holding that lock noninterruptible.
>>>>
>>>> Is that really a problem?  We typically only hold it over one kernel call,
>>>> which ought to be noninterruptible anyway.
>>>
>>> During parallel bulk load operations, I think we hold it over multiple
>>> kernel calls.
>>
>> We do.  Also, RelationGetNumberOfBlocks() is not necessarily only one
>> kernel call, no?  Nor is vm_extend.
>
> Yeah, these functions could call more than one kernel calls while
> holding extension lock.
>
>> Also, it's not just the backend doing the filesystem operation that's
>> non-interruptible, but also any waiters, right?
>>
>> Maybe this isn't a big problem, but it does seem to be that it would
>> be better to avoid it if we can.
>>
>
> I agree to change it to be interruptible for more safety.
>

Attached updated version patch. To use the lock mechanism similar to
LWLock but interrupt-able, I introduced new lock manager for extension
lock. A lot of code especially locking and unlocking, is inspired by
LWLock but it uses the condition variables to wait for acquiring lock.
Other part is not changed from previous patch. This is still a PoC
patch, lacks documentation. The following is the measurement result
with test script same as I used before.

* Copy test script
     HEAD    Patched
4    436.6   436.1
8    561.8   561.8
16   580.7   579.4
32   588.5   597.0
64   596.1   599.0

* Insert test script
     HEAD    Patched
4    156.5   156.0
8    167.0   167.9
16   176.2   175.6
32   181.1   181.0
64   181.5   183.0

Since I replaced heavyweight lock with lightweight lock I expected the
performance slightly improves from HEAD but it was almost same result.
I'll continue to look at more detail.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Moving_extension_lock_out_of_heavyweight_lock_v2.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

16 August 2017, 08:13:41

On Thu, Jun 22, 2017 at 12:03 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> On Fri, May 19, 2017 at 11:12 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> On Wed, May 17, 2017 at 1:30 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>>> On Sat, May 13, 2017 at 7:27 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>>> On Fri, May 12, 2017 at 9:14 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>>> Robert Haas <robertmhaas@gmail.com> writes:
>>>>>> On Wed, May 10, 2017 at 8:39 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>>>>>> ... I'd like to propose to change relation
>>>>>>> extension lock management so that it works using LWLock instead.
>>>>>
>>>>>> That's not a good idea because it'll make the code that executes while
>>>>>> holding that lock noninterruptible.
>>>>>
>>>>> Is that really a problem?  We typically only hold it over one kernel call,
>>>>> which ought to be noninterruptible anyway.
>>>>
>>>> During parallel bulk load operations, I think we hold it over multiple
>>>> kernel calls.
>>>
>>> We do.  Also, RelationGetNumberOfBlocks() is not necessarily only one
>>> kernel call, no?  Nor is vm_extend.
>>
>> Yeah, these functions could call more than one kernel calls while
>> holding extension lock.
>>
>>> Also, it's not just the backend doing the filesystem operation that's
>>> non-interruptible, but also any waiters, right?
>>>
>>> Maybe this isn't a big problem, but it does seem to be that it would
>>> be better to avoid it if we can.
>>>
>>
>> I agree to change it to be interruptible for more safety.
>>
>
> Attached updated version patch. To use the lock mechanism similar to
> LWLock but interrupt-able, I introduced new lock manager for extension
> lock. A lot of code especially locking and unlocking, is inspired by
> LWLock but it uses the condition variables to wait for acquiring lock.
> Other part is not changed from previous patch. This is still a PoC
> patch, lacks documentation. The following is the measurement result
> with test script same as I used before.
>
> * Copy test script
>      HEAD    Patched
> 4    436.6   436.1
> 8    561.8   561.8
> 16   580.7   579.4
> 32   588.5   597.0
> 64   596.1   599.0
>
> * Insert test script
>      HEAD    Patched
> 4    156.5   156.0
> 8    167.0   167.9
> 16   176.2   175.6
> 32   181.1   181.0
> 64   181.5   183.0
>
> Since I replaced heavyweight lock with lightweight lock I expected the
> performance slightly improves from HEAD but it was almost same result.
> I'll continue to look at more detail.
>

The previous patch conflicts with current HEAD, I rebased the patch to
current HEAD.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Moving_extension_lock_out_of_heavyweight_lock_v3.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Thomas Munro

Date:

08 September 2017, 04:24:11

On Wed, Aug 16, 2017 at 2:13 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> The previous patch conflicts with current HEAD, I rebased the patch to
> current HEAD.

Hi Masahiko-san,

FYI this doesn't build anymore.  I think it's just because the wait
event enumerators were re-alphabetised in pgstat.h:

../../../../src/include/pgstat.h:820:2: error: redeclaration of
enumerator ‘WAIT_EVENT_LOGICAL_SYNC_DATA’ WAIT_EVENT_LOGICAL_SYNC_DATA, ^
../../../../src/include/pgstat.h:806:2: note: previous definition of
‘WAIT_EVENT_LOGICAL_SYNC_DATA’ was here WAIT_EVENT_LOGICAL_SYNC_DATA, ^
../../../../src/include/pgstat.h:821:2: error: redeclaration of
enumerator ‘WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE’ WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, ^
../../../../src/include/pgstat.h:807:2: note: previous definition of
‘WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE’ was here WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE, ^

--
Thomas Munro
http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Thomas Munro

Date:

08 September 2017, 05:25:54

On Fri, Sep 8, 2017 at 10:24 AM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> On Wed, Aug 16, 2017 at 2:13 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> The previous patch conflicts with current HEAD, I rebased the patch to
>> current HEAD.
>
> Hi Masahiko-san,

Hi Sawada-san,

I have just learned from a colleague who is knowledgeable about
Japanese customs and kind enough to correct me that the appropriate
term of address for our colleagues in Japan on this mailing list is
<lastname>-san.  I was confused about that -- apologies for my
clumsiness.

-- 
Thomas Munro
http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

08 September 2017, 07:07:07

On Fri, Sep 8, 2017 at 8:25 AM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> On Fri, Sep 8, 2017 at 10:24 AM, Thomas Munro
> <thomas.munro@enterprisedb.com> wrote:
>> On Wed, Aug 16, 2017 at 2:13 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> The previous patch conflicts with current HEAD, I rebased the patch to
>>> current HEAD.
>>
>> Hi Masahiko-san,
>
> Hi Sawada-san,
>
> I have just learned from a colleague who is knowledgeable about
> Japanese customs and kind enough to correct me that the appropriate
> term of address for our colleagues in Japan on this mailing list is
> <lastname>-san.  I was confused about that -- apologies for my
> clumsiness.

Don't worry about it, either is ok. In Japan there is a custom of
writing <lastname>-san but <firstname>-san is also not incorrect :-)
(also I think it's hard to distinguish between last name and first
name of Japanese name).

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

08 September 2017, 08:32:15

On Fri, Sep 8, 2017 at 7:24 AM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> On Wed, Aug 16, 2017 at 2:13 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> The previous patch conflicts with current HEAD, I rebased the patch to
>> current HEAD.
>
> Hi Masahiko-san,
>
> FYI this doesn't build anymore.  I think it's just because the wait
> event enumerators were re-alphabetised in pgstat.h:
>
> ../../../../src/include/pgstat.h:820:2: error: redeclaration of
> enumerator ‘WAIT_EVENT_LOGICAL_SYNC_DATA’
>   WAIT_EVENT_LOGICAL_SYNC_DATA,
>   ^
> ../../../../src/include/pgstat.h:806:2: note: previous definition of
> ‘WAIT_EVENT_LOGICAL_SYNC_DATA’ was here
>   WAIT_EVENT_LOGICAL_SYNC_DATA,
>   ^
> ../../../../src/include/pgstat.h:821:2: error: redeclaration of
> enumerator ‘WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE’
>   WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE,
>   ^
> ../../../../src/include/pgstat.h:807:2: note: previous definition of
> ‘WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE’ was here
>   WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE,
>   ^
>

Thank you for the information! Attached rebased patch.

--
Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Moving_extension_lock_out_of_heavyweight_lock_v4.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

26 October 2017, 16:36:37

On Fri, Sep 8, 2017 at 4:32 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> On Fri, Sep 8, 2017 at 7:24 AM, Thomas Munro
> <thomas.munro@enterprisedb.com> wrote:
>> On Wed, Aug 16, 2017 at 2:13 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> The previous patch conflicts with current HEAD, I rebased the patch to
>>> current HEAD.
>>
>> Hi Masahiko-san,
>>
>> FYI this doesn't build anymore.  I think it's just because the wait
>> event enumerators were re-alphabetised in pgstat.h:
>>
>> ../../../../src/include/pgstat.h:820:2: error: redeclaration of
>> enumerator ‘WAIT_EVENT_LOGICAL_SYNC_DATA’
>>   WAIT_EVENT_LOGICAL_SYNC_DATA,
>>   ^
>> ../../../../src/include/pgstat.h:806:2: note: previous definition of
>> ‘WAIT_EVENT_LOGICAL_SYNC_DATA’ was here
>>   WAIT_EVENT_LOGICAL_SYNC_DATA,
>>   ^
>> ../../../../src/include/pgstat.h:821:2: error: redeclaration of
>> enumerator ‘WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE’
>>   WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE,
>>   ^
>> ../../../../src/include/pgstat.h:807:2: note: previous definition of
>> ‘WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE’ was here
>>   WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE,
>>   ^
>>
>
> Thank you for the information! Attached rebased patch.
>

Since the previous patch conflicts with current HEAD, I attached the
updated patch for next CF.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Attachment

Moving_extension_lock_out_of_heavyweight_lock_v5.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

26 October 2017, 21:03:45

On Thu, Oct 26, 2017 at 12:36 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> Since the previous patch conflicts with current HEAD, I attached the
> updated patch for next CF.

I think we should back up here and ask ourselves a couple of questions:

1. What are we trying to accomplish here?

2. Is this the best way to accomplish it?

To the first question, the problem as I understand it as follows:
Heavyweight locks don't conflict between members of a parallel group.
However, this is wrong for LOCKTAG_RELATION_EXTENSION, LOCKTAG_PAGE,
LOCKTAG_TUPLE, and LOCKTAG_SPECULATIVE_TOKEN.  Currently, those cases
don't arise, because parallel operations are strictly read-only
(except for inserts by the leader into a just-created table, when only
one member of the group can be taking the lock anyway).  However, once
we allow writes, they become possible, so some solution is needed.

To the second question, there are a couple of ways we could fix this.
First, we could continue to allow these locks to be taken in the
heavyweight lock manager, but make them conflict even between members
of the same lock group.  This is, however, complicated.  A significant
problem (or so I think) is that the deadlock detector logic, which is
already quite hard to test, will become even more complicated, since
wait edges between members of a lock group need to exist at some times
and not other times.  Moreover, to the best of my knowledge, the
increased complexity would have no benefit, because it doesn't look to
me like we ever take any other heavyweight lock while holding one of
these four kinds of locks.  Therefore, no deadlock can occur: if we're
waiting for one of these locks, the process that holds it is not
waiting for any other heavyweight lock.  This gives rise to a second
idea: move these locks out of the heavyweight lock manager and handle
them with separate code that does not have deadlock detection and
doesn't need as many lock modes.  I think that idea is basically
sound, although it's possibly not the only sound idea.

However, that makes me wonder whether we shouldn't be a bit more
aggressive with this patch: why JUST relation extension locks?  Why
not all four types of locks listed above?  Actually, tuple locks are a
bit sticky, because they have four lock modes.  The other three kinds
are very similar -- all you can do is "take it" (implicitly, in
exclusive mode), "try to take it" (again, implicitly, in exclusive
mode), or "wait for it to be released" (i.e. share lock and then
release).  Another idea is to try to handle those three types and
leave the tuple locking problem for another day.

I suggest that a good thing to do more or less immediately, regardless
of when this patch ends up being ready, would be to insert an
insertion that LockAcquire() is never called while holding a lock of
one of these types.  If that assertion ever fails, then the whole
theory that these lock types don't need deadlock detection is wrong,
and we'd like to find out about that sooner or later.

On the details of the patch, it appears that RelExtLockAcquire()
executes the wait-for-lock code with the partition lock held, and then
continues to hold the partition lock for the entire time that the
relation extension lock is held.  That not only makes all code that
runs while holding the lock non-interruptible but makes a lot of the
rest of this code pointless.  How is any of this atomics code going to
be reached by more than one process at the same time if the entire
bucket is exclusive-locked?  I would guess that the concurrency is not
very good here for the same reason.  Of course, just releasing the
bucket lock wouldn't be right either, because then ext_lock might go
away while we've got a pointer to it, which wouldn't be good.  I think
you could make this work if each lock had both a locker count and a
pin count, and the object can only be removed when the pin_count is 0.
So the lock algorithm would look like this:

- Acquire the partition LWLock.
- Find the item of interest, creating it if necessary.  If out of
memory for more elements, sweep through the table and reclaim
0-pin-count entries, then retry.
- Increment the pin count.
- Attempt to acquire the lock atomically; if we succeed, release the
partition lock and return.
- If this was a conditional-acquire, then decrement the pin count,
release the partition lock and return.
- Release the partition lock.
- Sleep on the condition variable until we manage to atomically
acquire the lock.

The unlock algorithm would just decrement the pin count and, if the
resulting value is non-zero, broadcast on the condition variable.

Although I think this will work, I'm not sure this is actually a great
algorithm.  Every lock acquisition has to take and release the
partition lock, use at least two more atomic ops (to take the pin and
the lock), and search a hash table.  I don't think that's going to be
staggeringly fast.  Maybe it's OK.  It's not that much worse, possibly
not any worse, than what the main lock manager does now.  However,
especially if we implement a solution specific to relation locks, it
seems like it would be better if we could somehow optimize based on
the facts that (1) many relation locks will not conflict and (2) it's
very common for the same backend to take and release the same
extension lock over and over again.  I don't have a specific proposal
right now.

Whatever we end up with, I think we should write some kind of a test
harness to benchmark the number of acquire/release cycles per second
that we can do with the current relation extension lock system vs. the
proposed new system.  Ideally, we'd be faster, since we're proposing a
more specialized mechanism.  But at least we should not be slower.
pgbench isn't a good test because the relation extension lock will
barely be taken let alone contended; we need to check something like
parallel copies into the same table to see any effect.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

30 October 2017, 12:17:59

On Fri, Oct 27, 2017 at 12:03 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Oct 26, 2017 at 12:36 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> Since the previous patch conflicts with current HEAD, I attached the
>> updated patch for next CF.
>
> I think we should back up here and ask ourselves a couple of questions:

Thank you for summarizing of the purpose and discussion of this patch.

> 1. What are we trying to accomplish here?
>
> 2. Is this the best way to accomplish it?
>
> To the first question, the problem as I understand it as follows:
> Heavyweight locks don't conflict between members of a parallel group.
> However, this is wrong for LOCKTAG_RELATION_EXTENSION, LOCKTAG_PAGE,
> LOCKTAG_TUPLE, and LOCKTAG_SPECULATIVE_TOKEN.  Currently, those cases
> don't arise, because parallel operations are strictly read-only
> (except for inserts by the leader into a just-created table, when only
> one member of the group can be taking the lock anyway).  However, once
> we allow writes, they become possible, so some solution is needed.
>
> To the second question, there are a couple of ways we could fix this.
> First, we could continue to allow these locks to be taken in the
> heavyweight lock manager, but make them conflict even between members
> of the same lock group.  This is, however, complicated.  A significant
> problem (or so I think) is that the deadlock detector logic, which is
> already quite hard to test, will become even more complicated, since
> wait edges between members of a lock group need to exist at some times
> and not other times.  Moreover, to the best of my knowledge, the
> increased complexity would have no benefit, because it doesn't look to
> me like we ever take any other heavyweight lock while holding one of
> these four kinds of locks.  Therefore, no deadlock can occur: if we're
> waiting for one of these locks, the process that holds it is not
> waiting for any other heavyweight lock.  This gives rise to a second
> idea: move these locks out of the heavyweight lock manager and handle
> them with separate code that does not have deadlock detection and
> doesn't need as many lock modes.  I think that idea is basically
> sound, although it's possibly not the only sound idea.

I'm on the same page.

>
> However, that makes me wonder whether we shouldn't be a bit more
> aggressive with this patch: why JUST relation extension locks?  Why
> not all four types of locks listed above?  Actually, tuple locks are a
> bit sticky, because they have four lock modes.  The other three kinds
> are very similar -- all you can do is "take it" (implicitly, in
> exclusive mode), "try to take it" (again, implicitly, in exclusive
> mode), or "wait for it to be released" (i.e. share lock and then
> release).  Another idea is to try to handle those three types and
> leave the tuple locking problem for another day.
>
> I suggest that a good thing to do more or less immediately, regardless
> of when this patch ends up being ready, would be to insert an
> insertion that LockAcquire() is never called while holding a lock of
> one of these types.  If that assertion ever fails, then the whole
> theory that these lock types don't need deadlock detection is wrong,
> and we'd like to find out about that sooner or later.

I understood. I'll check that first. If this direction has no problem
and we changed these three locks so that it uses new lock mechanism,
we'll not be able to use these locks at the same time. Since it also
means that we impose a limitation to the future we should think
carefully about it. We can implement the deadlock detection mechanism
for it again but it doesn't make sense.

>
> On the details of the patch, it appears that RelExtLockAcquire()
> executes the wait-for-lock code with the partition lock held, and then
> continues to hold the partition lock for the entire time that the
> relation extension lock is held.  That not only makes all code that
> runs while holding the lock non-interruptible but makes a lot of the
> rest of this code pointless.  How is any of this atomics code going to
> be reached by more than one process at the same time if the entire
> bucket is exclusive-locked?  I would guess that the concurrency is not
> very good here for the same reason.  Of course, just releasing the
> bucket lock wouldn't be right either, because then ext_lock might go
> away while we've got a pointer to it, which wouldn't be good.  I think
> you could make this work if each lock had both a locker count and a
> pin count, and the object can only be removed when the pin_count is 0.
> So the lock algorithm would look like this:
>
> - Acquire the partition LWLock.
> - Find the item of interest, creating it if necessary.  If out of
> memory for more elements, sweep through the table and reclaim
> 0-pin-count entries, then retry.
> - Increment the pin count.
> - Attempt to acquire the lock atomically; if we succeed, release the
> partition lock and return.
> - If this was a conditional-acquire, then decrement the pin count,
> release the partition lock and return.
> - Release the partition lock.
> - Sleep on the condition variable until we manage to atomically
> acquire the lock.
>
> The unlock algorithm would just decrement the pin count and, if the
> resulting value is non-zero, broadcast on the condition variable.

Thank you for the suggestion!

> Although I think this will work, I'm not sure this is actually a great
> algorithm.  Every lock acquisition has to take and release the
> partition lock, use at least two more atomic ops (to take the pin and
> the lock), and search a hash table.  I don't think that's going to be
> staggeringly fast.  Maybe it's OK.  It's not that much worse, possibly
> not any worse, than what the main lock manager does now.  However,
> especially if we implement a solution specific to relation locks, it
> seems like it would be better if we could somehow optimize based on
> the facts that (1) many relation locks will not conflict and (2) it's
> very common for the same backend to take and release the same
> extension lock over and over again.  I don't have a specific proposal
> right now.

Yeah, we can optimize based on the purpose of the solution. In either
case I should answer the above question first.

>
> Whatever we end up with, I think we should write some kind of a test
> harness to benchmark the number of acquire/release cycles per second
> that we can do with the current relation extension lock system vs. the
> proposed new system.  Ideally, we'd be faster, since we're proposing a
> more specialized mechanism.  But at least we should not be slower.
> pgbench isn't a good test because the relation extension lock will
> barely be taken let alone contended; we need to check something like
> parallel copies into the same table to see any effect.
>

I did a benchmark using a custom script that always updates the
primary key (disabling HOT updates). But parallel copies into the same
tale would also be good. Thank you.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

06 November 2017, 15:42:41

On Mon, Oct 30, 2017 at 3:17 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> On Fri, Oct 27, 2017 at 12:03 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Thu, Oct 26, 2017 at 12:36 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> Since the previous patch conflicts with current HEAD, I attached the
>>> updated patch for next CF.
>>
>> I think we should back up here and ask ourselves a couple of questions:
>
> Thank you for summarizing of the purpose and discussion of this patch.
>
>> 1. What are we trying to accomplish here?
>>
>> 2. Is this the best way to accomplish it?
>>
>> To the first question, the problem as I understand it as follows:
>> Heavyweight locks don't conflict between members of a parallel group.
>> However, this is wrong for LOCKTAG_RELATION_EXTENSION, LOCKTAG_PAGE,
>> LOCKTAG_TUPLE, and LOCKTAG_SPECULATIVE_TOKEN.  Currently, those cases
>> don't arise, because parallel operations are strictly read-only
>> (except for inserts by the leader into a just-created table, when only
>> one member of the group can be taking the lock anyway).  However, once
>> we allow writes, they become possible, so some solution is needed.
>>
>> To the second question, there are a couple of ways we could fix this.
>> First, we could continue to allow these locks to be taken in the
>> heavyweight lock manager, but make them conflict even between members
>> of the same lock group.  This is, however, complicated.  A significant
>> problem (or so I think) is that the deadlock detector logic, which is
>> already quite hard to test, will become even more complicated, since
>> wait edges between members of a lock group need to exist at some times
>> and not other times.  Moreover, to the best of my knowledge, the
>> increased complexity would have no benefit, because it doesn't look to
>> me like we ever take any other heavyweight lock while holding one of
>> these four kinds of locks.  Therefore, no deadlock can occur: if we're
>> waiting for one of these locks, the process that holds it is not
>> waiting for any other heavyweight lock.  This gives rise to a second
>> idea: move these locks out of the heavyweight lock manager and handle
>> them with separate code that does not have deadlock detection and
>> doesn't need as many lock modes.  I think that idea is basically
>> sound, although it's possibly not the only sound idea.
>
> I'm on the same page.
>
>>
>> However, that makes me wonder whether we shouldn't be a bit more
>> aggressive with this patch: why JUST relation extension locks?  Why
>> not all four types of locks listed above?  Actually, tuple locks are a
>> bit sticky, because they have four lock modes.  The other three kinds
>> are very similar -- all you can do is "take it" (implicitly, in
>> exclusive mode), "try to take it" (again, implicitly, in exclusive
>> mode), or "wait for it to be released" (i.e. share lock and then
>> release).  Another idea is to try to handle those three types and
>> leave the tuple locking problem for another day.
>>
>> I suggest that a good thing to do more or less immediately, regardless
>> of when this patch ends up being ready, would be to insert an
>> insertion that LockAcquire() is never called while holding a lock of
>> one of these types.  If that assertion ever fails, then the whole
>> theory that these lock types don't need deadlock detection is wrong,
>> and we'd like to find out about that sooner or later.
>
> I understood. I'll check that first.

I've checked whether LockAcquire is called while holding a lock of one
of four types: LOCKTAG_RELATION_EXTENSION, LOCKTAG_PAGE,
LOCKTAG_TUPLE, and LOCKTAG_SPECULATIVE_TOKEN. To summary, I think that
we cannot move these four lock types together out of heavy-weight
lock, but can move only the relation extension lock with tricks.

Here is detail of the survey.

* LOCKTAG_RELATION_EXTENSION
There is a path that LockRelationForExtension() could be called while
holding another relation extension lock. In brin_getinsertbuffer(), we
acquire a relation extension lock for a index relation and could
initialize a new buffer (brin_initailize_empty_new_buffer()). During
initializing a new buffer, we call RecordPageWithFreeSpace() which
eventually can call fsm_readbuf(rel, addr, true) where the third
argument is "extend". We can process this problem by having the list
(or local hash) of acquired locks and skip acquiring the lock if
already had. For other call paths calling LockRelationForExtension, I
don't see any problem.

* LOCKTAG_PAGE, LOCKTAG_TUPLE, LOCKTAG_SPECULATIVE_INSERTION
There is a path that we can acquire a relation extension lock while
holding these lock.
For LOCKTAG_PAGE, in ginInsertCleanup() we acquire a page lock for the
meta page and process the pending list which could acquire a relation
extension lock for a index relation. For LOCKTAG_TUPLE, in
heap_update() we acquire a tuple lock and could call
RelationGetBufferForTuple(). For LOCKTAG_SPECULATIVE_INSERTION, in
ExecInsert() we acquire a speculative insertion lock and call
heap_insert and ExecInsertIndexTuples(). The operation that is called
while holding each lock type can acquire a relation extension lock.

Also the following is the list of places where we call LockAcquire()
with four lock types (result of git grep "XXX"). I've checked based on
the following list.

* LockRelationForExtension()
contrib/bloom/blutils.c:
LockRelationForExtension(index, ExclusiveLock);
contrib/pgstattuple/pgstattuple.c:
LockRelationForExtension(rel, ExclusiveLock);
src/backend/access/brin/brin_pageops.c:
LockRelationForExtension(idxrel, ShareLock);
src/backend/access/brin/brin_pageops.c:
LockRelationForExtension(irel, ExclusiveLock);
src/backend/access/brin/brin_revmap.c:
LockRelationForExtension(irel, ExclusiveLock);
src/backend/access/gin/ginutil.c:
LockRelationForExtension(index, ExclusiveLock);
src/backend/access/gin/ginvacuum.c:
LockRelationForExtension(index, ExclusiveLock);
src/backend/access/gin/ginvacuum.c:
LockRelationForExtension(index, ExclusiveLock);
src/backend/access/gist/gistutil.c:
LockRelationForExtension(r, ExclusiveLock);
src/backend/access/gist/gistvacuum.c:
LockRelationForExtension(rel, ExclusiveLock);
src/backend/access/gist/gistvacuum.c:
LockRelationForExtension(rel, ExclusiveLock);
src/backend/access/heap/hio.c:
LockRelationForExtension(relation, ExclusiveLock);
src/backend/access/heap/hio.c:
LockRelationForExtension(relation, ExclusiveLock);
src/backend/access/heap/visibilitymap.c:
LockRelationForExtension(rel, ExclusiveLock);
src/backend/access/nbtree/nbtpage.c:
LockRelationForExtension(rel, ExclusiveLock);
src/backend/access/nbtree/nbtree.c:
LockRelationForExtension(rel, ExclusiveLock);
src/backend/access/spgist/spgutils.c:
LockRelationForExtension(index, ExclusiveLock);
src/backend/access/spgist/spgvacuum.c:
LockRelationForExtension(index, ExclusiveLock);
src/backend/commands/vacuumlazy.c:
LockRelationForExtension(onerel, ExclusiveLock);
src/backend/storage/freespace/freespace.c:
LockRelationForExtension(rel, ExclusiveLock);
src/backend/storage/lmgr/lmgr.c:LockRelationForExtension(Relation
relation, LOCKMODE lockmode)

* ConditionalLockRelationForExtension
src/backend/access/heap/hio.c:          else if
(!ConditionalLockRelationForExtension(relation, ExclusiveLock))
src/backend/storage/lmgr/lmgr.c:ConditionalLockRelationForExtension(Relation
relation, LOCKMODE lockmode)

* LockPage
src/backend/access/gin/ginfast.c:               LockPage(index,
GIN_METAPAGE_BLKNO, ExclusiveLock);

* ConditionalLockPage
src/backend/access/gin/ginfast.c:               if
(!ConditionalLockPage(index, GIN_METAPAGE_BLKNO, ExclusiveLock))

* LockTuple
src/backend/access/heap/heapam.c:       LockTuple((rel), (tup),
tupleLockExtraInfo[mode].hwlock)

* ConditionalLockTuple
src/backend/access/heap/heapam.c:       ConditionalLockTuple((rel),
(tup), tupleLockExtraInfo[mode].hwlock)
src/backend/storage/lmgr/lmgr.c:ConditionalLockTuple(Relation
relation, ItemPointer tid, LOCKMODE lockmode)

* SpeculativeInsertionLockAcquire
src/backend/executor/nodeModifyTable.c:                 specToken =
SpeculativeInsertionLockAcquire(GetCurrentTransactionId());

> If this direction has no problem
> and we changed these three locks so that it uses new lock mechanism,
> we'll not be able to use these locks at the same time. Since it also
> means that we impose a limitation to the future we should think
> carefully about it. We can implement the deadlock detection mechanism
> for it again but it doesn't make sense.
>
>>
>> On the details of the patch, it appears that RelExtLockAcquire()
>> executes the wait-for-lock code with the partition lock held, and then
>> continues to hold the partition lock for the entire time that the
>> relation extension lock is held.  That not only makes all code that
>> runs while holding the lock non-interruptible but makes a lot of the
>> rest of this code pointless.  How is any of this atomics code going to
>> be reached by more than one process at the same time if the entire
>> bucket is exclusive-locked?  I would guess that the concurrency is not
>> very good here for the same reason.  Of course, just releasing the
>> bucket lock wouldn't be right either, because then ext_lock might go
>> away while we've got a pointer to it, which wouldn't be good.  I think
>> you could make this work if each lock had both a locker count and a
>> pin count, and the object can only be removed when the pin_count is 0.
>> So the lock algorithm would look like this:
>>
>> - Acquire the partition LWLock.
>> - Find the item of interest, creating it if necessary.  If out of
>> memory for more elements, sweep through the table and reclaim
>> 0-pin-count entries, then retry.
>> - Increment the pin count.
>> - Attempt to acquire the lock atomically; if we succeed, release the
>> partition lock and return.
>> - If this was a conditional-acquire, then decrement the pin count,
>> release the partition lock and return.
>> - Release the partition lock.
>> - Sleep on the condition variable until we manage to atomically
>> acquire the lock.
>>
>> The unlock algorithm would just decrement the pin count and, if the
>> resulting value is non-zero, broadcast on the condition variable.
>
> Thank you for the suggestion!
>
>> Although I think this will work, I'm not sure this is actually a great
>> algorithm.  Every lock acquisition has to take and release the
>> partition lock, use at least two more atomic ops (to take the pin and
>> the lock), and search a hash table.  I don't think that's going to be
>> staggeringly fast.  Maybe it's OK.  It's not that much worse, possibly
>> not any worse, than what the main lock manager does now.  However,
>> especially if we implement a solution specific to relation locks, it
>> seems like it would be better if we could somehow optimize based on
>> the facts that (1) many relation locks will not conflict and (2) it's
>> very common for the same backend to take and release the same
>> extension lock over and over again.  I don't have a specific proposal
>> right now.
>
> Yeah, we can optimize based on the purpose of the solution. In either
> case I should answer the above question first.
>
>>
>> Whatever we end up with, I think we should write some kind of a test
>> harness to benchmark the number of acquire/release cycles per second
>> that we can do with the current relation extension lock system vs. the
>> proposed new system.  Ideally, we'd be faster, since we're proposing a
>> more specialized mechanism.  But at least we should not be slower.
>> pgbench isn't a good test because the relation extension lock will
>> barely be taken let alone contended; we need to check something like
>> parallel copies into the same table to see any effect.
>>
>
> I did a benchmark using a custom script that always updates the
> primary key (disabling HOT updates). But parallel copies into the same
> tale would also be good. Thank you.
>

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

08 November 2017, 02:41:36

On Mon, Nov 6, 2017 at 4:42 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> I suggest that a good thing to do more or less immediately, regardless
>>> of when this patch ends up being ready, would be to insert an
>>> insertion that LockAcquire() is never called while holding a lock of
>>> one of these types.  If that assertion ever fails, then the whole
>>> theory that these lock types don't need deadlock detection is wrong,
>>> and we'd like to find out about that sooner or later.
>>
>> I understood. I'll check that first.
>
> I've checked whether LockAcquire is called while holding a lock of one
> of four types: LOCKTAG_RELATION_EXTENSION, LOCKTAG_PAGE,
> LOCKTAG_TUPLE, and LOCKTAG_SPECULATIVE_TOKEN. To summary, I think that
> we cannot move these four lock types together out of heavy-weight
> lock, but can move only the relation extension lock with tricks.
>
> Here is detail of the survey.

Thanks for these details, but I'm not sure I fully understand.

> * LOCKTAG_RELATION_EXTENSION
> There is a path that LockRelationForExtension() could be called while
> holding another relation extension lock. In brin_getinsertbuffer(), we
> acquire a relation extension lock for a index relation and could
> initialize a new buffer (brin_initailize_empty_new_buffer()). During
> initializing a new buffer, we call RecordPageWithFreeSpace() which
> eventually can call fsm_readbuf(rel, addr, true) where the third
> argument is "extend". We can process this problem by having the list
> (or local hash) of acquired locks and skip acquiring the lock if
> already had. For other call paths calling LockRelationForExtension, I
> don't see any problem.

Does calling fsm_readbuf(rel,addr,true) take some heavyweight lock?

Basically, what matters here in the end is whether we can articulate a
deadlock-proof rule around the order in which these locks are
acquired.  The simplest such rule would be "you can only acquire one
lock of any of these types at a time, and you can't subsequently
acquire a heavyweight lock".  But a more complicated rule would be OK
too, e.g. "you can acquire as many heavyweight locks as you want, and
after that you can optionally acquire one page, tuple, or speculative
token lock, and after that you can acquire a relation extension lock".
The latter rule, although more complex, is still deadlock-proof,
because the heavyweight locks still use the deadlock detector, and the
rest has a consistent order of lock acquisition that precludes one
backend taking A then B while another backend takes B then A.  I'm not
entirely clear whether your survey leads us to a place where we can
articulate such a deadlock-proof rule.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

08 November 2017, 20:40:03

On Wed, Nov 8, 2017 at 5:41 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Nov 6, 2017 at 4:42 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>>> I suggest that a good thing to do more or less immediately, regardless
>>>> of when this patch ends up being ready, would be to insert an
>>>> insertion that LockAcquire() is never called while holding a lock of
>>>> one of these types.  If that assertion ever fails, then the whole
>>>> theory that these lock types don't need deadlock detection is wrong,
>>>> and we'd like to find out about that sooner or later.
>>>
>>> I understood. I'll check that first.
>>
>> I've checked whether LockAcquire is called while holding a lock of one
>> of four types: LOCKTAG_RELATION_EXTENSION, LOCKTAG_PAGE,
>> LOCKTAG_TUPLE, and LOCKTAG_SPECULATIVE_TOKEN. To summary, I think that
>> we cannot move these four lock types together out of heavy-weight
>> lock, but can move only the relation extension lock with tricks.
>>
>> Here is detail of the survey.
>
> Thanks for these details, but I'm not sure I fully understand.
>
>> * LOCKTAG_RELATION_EXTENSION
>> There is a path that LockRelationForExtension() could be called while
>> holding another relation extension lock. In brin_getinsertbuffer(), we
>> acquire a relation extension lock for a index relation and could
>> initialize a new buffer (brin_initailize_empty_new_buffer()). During
>> initializing a new buffer, we call RecordPageWithFreeSpace() which
>> eventually can call fsm_readbuf(rel, addr, true) where the third
>> argument is "extend". We can process this problem by having the list
>> (or local hash) of acquired locks and skip acquiring the lock if
>> already had. For other call paths calling LockRelationForExtension, I
>> don't see any problem.
>
> Does calling fsm_readbuf(rel,addr,true) take some heavyweight lock?

No, I meant fsm_readbuf(rel,addr,true) can acquire a relation
extension lock. So it's not problem.

> Basically, what matters here in the end is whether we can articulate a
> deadlock-proof rule around the order in which these locks are
> acquired.

You're right, my survey was not enough to make a decision.

As far as the acquiring these four lock types goes, there are two call
paths that acquire any type of locks while holding another type of
lock. The one is that acquiring a relation extension lock and then
acquiring a relation extension lock for the same relation again. As
explained before, this can be resolved by remembering the holding lock
(perhaps holding only last one is enough). Another is that acquiring
either a tuple lock, a page lock or a speculative insertion lock and
then acquiring a relation extension lock. In the second case, we try
to acquire these two locks in the same order; acquiring 3 types lock
and then extension lock. So it's not problem if we apply the rule that
is that we disallow to try acquiring these three lock types while
holding any relation extension lock. Also, as far as I surveyed there
is no path to acquire a relation lock while holding other 3 type
locks.

>  The simplest such rule would be "you can only acquire one
> lock of any of these types at a time, and you can't subsequently
> acquire a heavyweight lock".  But a more complicated rule would be OK
> too, e.g. "you can acquire as many heavyweight locks as you want, and
> after that you can optionally acquire one page, tuple, or speculative
> token lock, and after that you can acquire a relation extension lock".
> The latter rule, although more complex, is still deadlock-proof,
> because the heavyweight locks still use the deadlock detector, and the
> rest has a consistent order of lock acquisition that precludes one
> backend taking A then B while another backend takes B then A.  I'm not
> entirely clear whether your survey leads us to a place where we can
> articulate such a deadlock-proof rule.

Speaking of the acquiring these four lock types and heavy weight lock,
there obviously is a call path to acquire any of four lock types while
holding a heavy weight lock. In reverse, there also is a call path
that we acquire a heavy weight lock while holding any of four lock
types. The call path I found is that in heap_delete we acquire a tuple
lock and call XactLockTableWait or MultiXactIdWait which eventually
could acquire LOCKTAG_TRANSACTION in order to wait for the concurrent
transactions finish. But IIUC since these functions acquire the lock
for the concurrent transaction's transaction id, deadlocks doesn't
happen.
However, there might be other similar call paths if I'm missing
something. For example, we do some operations that might acquire any
heavy weight locks other than LOCKTAG_TRANSACTION, while holding a
page lock (in ginInsertCleanup) or holding a specualtive insertion
lock (in nodeModifyTable).

To summary, I think we can put the following rules in order to move
four lock types out of heavy weight lock.

1. Do not acquire either a tuple lock, a page lock or a speculative
insertion lock while holding a extension lock.
2. Do not acquire any heavy weight lock except for LOCKTAG_TRANSACTION
while holding any of these four lock types.

Also I'm concerned that it imposes the rules for developers which is
difficult to check statically. We can put several assertions to source
code but it's hard to test the all possible paths by regression tests.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

09 November 2017, 21:28:24

On Wed, Nov 8, 2017 at 9:40 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> Speaking of the acquiring these four lock types and heavy weight lock,
> there obviously is a call path to acquire any of four lock types while
> holding a heavy weight lock. In reverse, there also is a call path
> that we acquire a heavy weight lock while holding any of four lock
> types. The call path I found is that in heap_delete we acquire a tuple
> lock and call XactLockTableWait or MultiXactIdWait which eventually
> could acquire LOCKTAG_TRANSACTION in order to wait for the concurrent
> transactions finish. But IIUC since these functions acquire the lock
> for the concurrent transaction's transaction id, deadlocks doesn't
> happen.

No, that's not right.  Now that you mention it, I realize that tuple
locks can definitely cause deadlocks.  Example:

setup:
rhaas=# create table foo (a int, b text);
CREATE TABLE
rhaas=# create table bar (a int, b text);
CREATE TABLE
rhaas=# insert into foo values (1, 'hoge');
INSERT 0 1

session 1:
rhaas=# begin;
BEGIN
rhaas=# update foo set b = 'hogehoge' where a = 1;
UPDATE 1

session 2:
rhaas=# begin;
BEGIN
rhaas=# update foo set b = 'quux' where a = 1;

session 3:
rhaas=# begin;
BEGIN
rhaas=# lock bar;
LOCK TABLE
rhaas=# update foo set b = 'blarfle' where a = 1;

back to session 1:
rhaas=# select * from bar;
ERROR:  deadlock detected
LINE 1: select * from bar;                     ^
DETAIL:  Process 88868 waits for AccessShareLock on relation 16391 of
database 16384; blocked by process 88845.
Process 88845 waits for ExclusiveLock on tuple (0,1) of relation 16385
of database 16384; blocked by process 88840.
Process 88840 waits for ShareLock on transaction 1193; blocked by process 88868.
HINT:  See server log for query details.

So what I said before was wrong: we definitely cannot exclude tuple
locks from deadlock detection.  However, we might be able to handle
the problem in another way: introduce a separate, parallel-query
specific mechanism to avoid having two participants try to update
and/or delete the same tuple at the same time - e.g. advertise the
BufferTag + offset within the page in DSM, and if somebody else
already has that same combination advertised, wait until they no
longer do.  That shouldn't ever deadlock, because the other worker
shouldn't be able to find itself waiting for us while it's busy
updating a tuple.

After some further study, speculative insertion locks look problematic
too.  I'm worried about the code path ExecInsert() [taking speculative
insertion locking] -> heap_insert -> heap_prepare_insert ->
toast_insert_or_update -> toast_save_datum ->
heap_open(rel->rd_rel->reltoastrelid, RowExclusiveLock).  That sure
looks like we can end up waiting for a relation lock while holding a
speculative insertion lock, which seems to mean that speculative
insertion locks are subject to at least theoretical deadlock hazards
as well.  Note that even if we were guaranteed to be holding the lock
on the toast relation already at this point, it wouldn't fix the
problem, because we might still have to build or refresh a relcache
entry at this point, which could end up scanning (and thus locking)
system catalogs.  Any syscache lookup can theoretically take a lock,
even though most of the time it doesn't, and thus taking a lock that
has been removed from the deadlock detector (or, say, an lwlock) and
then performing a syscache lookup with it held is not OK.  So I don't
think we can remove speculative insertion locks from the deadlock
detector either.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Tom Lane

Date:

09 November 2017, 21:38:41

Robert Haas <robertmhaas@gmail.com> writes:
> No, that's not right.  Now that you mention it, I realize that tuple
> locks can definitely cause deadlocks.  Example:

Yeah.  Foreign-key-related tuple locks are another rich source of
examples.

> ... So I don't
> think we can remove speculative insertion locks from the deadlock
> detector either.

That scares me too.  I think that relation extension can safely
be transferred to some lower-level mechanism, because what has to
be done while holding the lock is circumscribed and below the level
of database operations (which might need other locks).  These other
ideas seem a lot riskier.

(But see recent conversation where I discouraged Alvaro from holding
extension locks across BRIN summarization activity.  We'll need to look
and make sure that nobody else has had creative ideas like that.)
        regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

14 November 2017, 13:36:09

Thank you for pointing out and comments.

On Fri, Nov 10, 2017 at 12:38 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> No, that's not right.  Now that you mention it, I realize that tuple
>> locks can definitely cause deadlocks.  Example:
>
> Yeah.  Foreign-key-related tuple locks are another rich source of
> examples.
>
>> ... So I don't
>> think we can remove speculative insertion locks from the deadlock
>> detector either.
>
> That scares me too.  I think that relation extension can safely
> be transferred to some lower-level mechanism, because what has to
> be done while holding the lock is circumscribed and below the level
> of database operations (which might need other locks).  These other
> ideas seem a lot riskier.
>
> (But see recent conversation where I discouraged Alvaro from holding
> extension locks across BRIN summarization activity.  We'll need to look
> and make sure that nobody else has had creative ideas like that.)
>

It seems that we should focus on transferring only relation extension
locks as a first step. The page locks would also be safe but it might
require some fundamental changes related to fast insertion, which is
discussed on other thread[1]. Also in this case I think it's better to
focus on relation extension locks so that we can optimize the
lower-level lock mechanism for it.

So I'll update the patch based on the comment I got from Robert before.

[1] https://www.postgresql.org/message-id/CAD21AoBLUSyiYKnTYtSAbC%2BF%3DXDjiaBrOUEGK%2BzUXdQ8owfPKw%40mail.gmail.com

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

21 November 2017, 04:19:30

On Tue, Nov 14, 2017 at 4:36 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> Thank you for pointing out and comments.
>
> On Fri, Nov 10, 2017 at 12:38 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:
>>> No, that's not right.  Now that you mention it, I realize that tuple
>>> locks can definitely cause deadlocks.  Example:
>>
>> Yeah.  Foreign-key-related tuple locks are another rich source of
>> examples.
>>
>>> ... So I don't
>>> think we can remove speculative insertion locks from the deadlock
>>> detector either.
>>
>> That scares me too.  I think that relation extension can safely
>> be transferred to some lower-level mechanism, because what has to
>> be done while holding the lock is circumscribed and below the level
>> of database operations (which might need other locks).  These other
>> ideas seem a lot riskier.
>>
>> (But see recent conversation where I discouraged Alvaro from holding
>> extension locks across BRIN summarization activity.  We'll need to look
>> and make sure that nobody else has had creative ideas like that.)
>>
>
> It seems that we should focus on transferring only relation extension
> locks as a first step. The page locks would also be safe but it might
> require some fundamental changes related to fast insertion, which is
> discussed on other thread[1]. Also in this case I think it's better to
> focus on relation extension locks so that we can optimize the
> lower-level lock mechanism for it.
>
> So I'll update the patch based on the comment I got from Robert before.
>

Attached updated version patch. I've moved only relation extension
locks out of heavy-weight lock as per discussion so far.

I've done a write-heavy benchmark on my laptop; loading 24kB data to
one table using COPY by 1 client, for 10 seconds. The through-put of
patched is 10% better than current HEAD. The result of 5 times is the
following.

----- PATCHED -----
tps = 178.791515 (excluding connections establishing)
tps = 176.522693 (excluding connections establishing)
tps = 168.705442 (excluding connections establishing)
tps = 158.158009 (excluding connections establishing)
tps = 161.145709 (excluding connections establishing)

----- HEAD -----
tps = 147.079803 (excluding connections establishing)
tps = 149.079540 (excluding connections establishing)
tps = 149.082275 (excluding connections establishing)
tps = 148.255376 (excluding connections establishing)
tps = 145.542552 (excluding connections establishing)

Also I've done a micro-benchmark; calling LockRelationForExtension and
UnlockRelationForExtension tightly in order to measure the number of
lock/unlock cycles per second. The result is,
PATCHED = 3.95892e+06 (cycles/sec)
HEAD = 1.15284e+06 (cycles/sec)
The patched is 3 times faster than current HEAD.

Attached updated patch and the function I used for micro-benchmark.
Please review it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, Nov 22, 2017 at 11:32 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> On Wed, Nov 22, 2017 at 5:25 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Mon, Nov 20, 2017 at 5:19 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> Attached updated version patch. I've moved only relation extension
>>> locks out of heavy-weight lock as per discussion so far.
>>>
>>> I've done a write-heavy benchmark on my laptop; loading 24kB data to
>>> one table using COPY by 1 client, for 10 seconds. The through-put of
>>> patched is 10% better than current HEAD. The result of 5 times is the
>>> following.
>>>
>>> ----- PATCHED -----
>>> tps = 178.791515 (excluding connections establishing)
>>> tps = 176.522693 (excluding connections establishing)
>>> tps = 168.705442 (excluding connections establishing)
>>> tps = 158.158009 (excluding connections establishing)
>>> tps = 161.145709 (excluding connections establishing)
>>>
>>> ----- HEAD -----
>>> tps = 147.079803 (excluding connections establishing)
>>> tps = 149.079540 (excluding connections establishing)
>>> tps = 149.082275 (excluding connections establishing)
>>> tps = 148.255376 (excluding connections establishing)
>>> tps = 145.542552 (excluding connections establishing)
>>>
>>> Also I've done a micro-benchmark; calling LockRelationForExtension and
>>> UnlockRelationForExtension tightly in order to measure the number of
>>> lock/unlock cycles per second. The result is,
>>> PATCHED = 3.95892e+06 (cycles/sec)
>>> HEAD = 1.15284e+06 (cycles/sec)
>>> The patched is 3 times faster than current HEAD.
>>>
>>> Attached updated patch and the function I used for micro-benchmark.
>>> Please review it.
>>
>> That's a nice speed-up.
>>
>> How about a preliminary patch that asserts that we never take another
>> heavyweight lock while holding a relation extension lock?
>>
>
> Agreed. Also, since we disallow to holding more than one locks of
> different relations at once I'll add an assertion for it as well.
>
> I think we no longer need to pass the lock level to
> UnloclRelationForExtension(). Now that relation extension lock will be
> simple we can release the lock in the mode that we used to acquire
> like LWLock.
>

Attached latest patch incorporated all comments so far. Please review it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachment

Moving_extension_lock_out_of_heavyweight_lock_v7.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

29 November 2017, 02:33:40

On Sun, Nov 26, 2017 at 9:33 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> Attached latest patch incorporated all comments so far. Please review it.

I think you only need RelExtLockReleaseAllI() where we currently have
LockReleaseAll(DEFAULT_LOCKMETHOD, ...) not where we have
LockReleaseAll(USER_LOCKMETHOD, ...).  That's because relation
extension locks use the default lock method, not USER_LOCKMETHOD.

You need to update the table of wait events in the documentation.
Please be sure to actually build the documentation afterwards and make
sure it looks OK.  Maybe the way event name should be
RelationExtensionLock rather than just RelationExtension; we are not
waiting for the extension itself.

You have a typo/thinko in lmgr/README: confliction is not a word.
Maybe you mean "When conflicts occur, lock waits are implemented using
condition variables."

Instead of having shared and exclusive locks, how about just having
exclusive locks and introducing a new primitive operation that waits
for the lock to be free and returns without acquiring it? That is
essentially what brin_pageops.c is doing by taking and releasing the
shared lock, and it's the only caller that takes anything but an
exclusive lock.  This seems like it would permit a considerable
simplification of the locking mechanism, since there would then be
only two possible states: 1 (locked) and 0 (not locked).

In RelExtLockAcquire, I can't endorse this sort of coding:

+        if (relid == held_relextlock.lock->relid &&
+            lockmode == held_relextlock.mode)
+        {
+            held_relextlock.nLocks++;
+            return true;
+        }
+        else
+            Assert(false);    /* cannot happen */

Either convert the Assert() to an elog(), or change the if-statement
to an Assert() of the same condition.  I'd probably vote for the first
one.  As it is, if that Assert(false) is ever hit, chaos will (maybe)
ensue.  Let's make sure we nip any such problems in the bud.

"successed" is not a good variable name; that's not an English word.

+        /* Could not got the lock, return iff in conditional locking */
+        if (mustwait && conditional)

Comment contradicts code.  The comment is right; the code need not
test mustwait, as that's already been done.

The way this is hooked into the shared-memory initialization stuff
looks strange in a number of ways:

- Apparently, you're making initialize enough space for as many
relation extension locks as the save of the main heavyweight lock
table, but that seems like overkill.  I'm not sure how much space we
actually need for relation extension locks, but I bet it's a lot less
than we need for regular heavyweight locks.
- The error message emitted when you run out of space also claims that
you can fix the issue by raising max_pred_locks_per_transaction, but
that has no effect on the size of the main lock table or this table.
- The changes to LockShmemSize() suppose that the hash table elements
have a size equal to the size of an LWLock, but the actual size is
sizeof(RELEXTLOCK).
- I don't really know why the code for this should be daisy-chained
off of the lock.c code inside of being called from
CreateSharedMemoryAndSemaphores() just like (almost) all of the other
subsystems.

This code ignores the existence of multiple databases; RELEXTLOCK
contains a relid, but no database OID.  That's easy enough to fix, but
it actually causes no problem unless, by bad luck, you have two
relations with the same OID in different databases that are both being
rapidly extended at the same time -- and even then, it's only a
performance problem, not a correctness problem.  In fact, I wonder if
we shouldn't go further: instead of creating these RELEXTLOCK
structures dynamically, let's just have a fixed number of them, say
1024.  When we get a request to take a lock, hash <dboid, reloid> and
take the result modulo 1024; lock the RELEXTLOCK at that offset in the
array.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Michael Paquier

Date:

30 November 2017, 07:52:14

On Wed, Nov 29, 2017 at 5:33 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sun, Nov 26, 2017 at 9:33 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> Attached latest patch incorporated all comments so far. Please review it.
>
> I think you only need RelExtLockReleaseAllI() where we currently have
> LockReleaseAll(DEFAULT_LOCKMETHOD, ...) not where we have
> LockReleaseAll(USER_LOCKMETHOD, ...).  That's because relation
> extension locks use the default lock method, not USER_LOCKMETHOD.

Latest review is fresh. I am moving this to next CF with "waiting on
author" as status.
-- 
Michael

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

30 November 2017, 08:05:16

On Thu, Nov 30, 2017 at 10:52 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Wed, Nov 29, 2017 at 5:33 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Sun, Nov 26, 2017 at 9:33 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> Attached latest patch incorporated all comments so far. Please review it.
>>
>> I think you only need RelExtLockReleaseAllI() where we currently have
>> LockReleaseAll(DEFAULT_LOCKMETHOD, ...) not where we have
>> LockReleaseAll(USER_LOCKMETHOD, ...).  That's because relation
>> extension locks use the default lock method, not USER_LOCKMETHOD.
>
> Latest review is fresh. I am moving this to next CF with "waiting on
> author" as status.

Thank you Michael-san, I'll submit a latest patch.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

30 November 2017, 17:20:32

On Wed, Nov 29, 2017 at 5:33 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sun, Nov 26, 2017 at 9:33 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> Attached latest patch incorporated all comments so far. Please review it.
>
> I think you only need RelExtLockReleaseAllI() where we currently have
> LockReleaseAll(DEFAULT_LOCKMETHOD, ...) not where we have
> LockReleaseAll(USER_LOCKMETHOD, ...).  That's because relation
> extension locks use the default lock method, not USER_LOCKMETHOD.

Fixed.

> You need to update the table of wait events in the documentation.
> Please be sure to actually build the documentation afterwards and make
> sure it looks OK.  Maybe the way event name should be
> RelationExtensionLock rather than just RelationExtension; we are not
> waiting for the extension itself.

Fixed. I added both new wait_event and wait_event_type for relext
lock. Also checked to pass a documentation build.

>
> You have a typo/thinko in lmgr/README: confliction is not a word.
> Maybe you mean "When conflicts occur, lock waits are implemented using
> condition variables."

Fixed.

>
> Instead of having shared and exclusive locks, how about just having
> exclusive locks and introducing a new primitive operation that waits
> for the lock to be free and returns without acquiring it? That is
> essentially what brin_pageops.c is doing by taking and releasing the
> shared lock, and it's the only caller that takes anything but an
> exclusive lock.  This seems like it would permit a considerable
> simplification of the locking mechanism, since there would then be
> only two possible states: 1 (locked) and 0 (not locked).

I think it's a good idea. With this change, the concurrency of
executing brin_page_cleanup() would get decreased. But since
brin_page_cleanup() is called only during vacuum so far it's no
problem. I think we can process the code in vacuumlazy.c in the same
manner as well. I've changed the patch so that it has only exclusive
locks and introduces WaitForRelationExtensionLockToBeFree() function
to wait for the the lock to be free.

Also, now that we got rid of shared locks, I gathered lock state and
pin count into a atomic uint32.

> In RelExtLockAcquire, I can't endorse this sort of coding:
>
> +        if (relid == held_relextlock.lock->relid &&
> +            lockmode == held_relextlock.mode)
> +        {
> +            held_relextlock.nLocks++;
> +            return true;
> +        }
> +        else
> +            Assert(false);    /* cannot happen */
>
> Either convert the Assert() to an elog(), or change the if-statement
> to an Assert() of the same condition.  I'd probably vote for the first
> one.  As it is, if that Assert(false) is ever hit, chaos will (maybe)
> ensue.  Let's make sure we nip any such problems in the bud.

Agreed, fixed.

>
> "successed" is not a good variable name; that's not an English word.

Fixed.

> +        /* Could not got the lock, return iff in conditional locking */
> +        if (mustwait && conditional)
>
> Comment contradicts code.  The comment is right; the code need not
> test mustwait, as that's already been done.

Fixed.

> The way this is hooked into the shared-memory initialization stuff
> looks strange in a number of ways:
>
> - Apparently, you're making initialize enough space for as many
> relation extension locks as the save of the main heavyweight lock
> table, but that seems like overkill.  I'm not sure how much space we
> actually need for relation extension locks, but I bet it's a lot less
> than we need for regular heavyweight locks.

Agreed. The maximum of the number of relext locks is the number of
relations on a database cluster, it's not relevant with the number of
clients. Currently NLOCKENTS() counts the number of locks including
relation extension lock. One idea is to introduce a new GUC to control
the memory size, although the total memory size for locks will get
increased. Probably we can make it behave similar to
max_pred_locks_per_relation. Or, in order to not change total memory
size for lock even after moved it out of heavyweight lock, we can
divide NLOCKENTS() into heavyweight lock and relation extension lock
(for example, 80% for heavyweight locks and 20% relation extension
locks). But the latter would make parameter tuning hard. I'd vote for
the first one to keep it simple. Any ideas? This part is not fixed in
the patch yet.

> - The error message emitted when you run out of space also claims that
> you can fix the issue by raising max_pred_locks_per_transaction, but
> that has no effect on the size of the main lock table or this table.

Fixed.

> - The changes to LockShmemSize() suppose that the hash table elements
> have a size equal to the size of an LWLock, but the actual size is
> sizeof(RELEXTLOCK).

Fixed.

> - I don't really know why the code for this should be daisy-chained
> off of the lock.c code inside of being called from
> CreateSharedMemoryAndSemaphores() just like (almost) all of the other
> subsystems.

Fixed.

>
> This code ignores the existence of multiple databases; RELEXTLOCK
> contains a relid, but no database OID.  That's easy enough to fix, but
> it actually causes no problem unless, by bad luck, you have two
> relations with the same OID in different databases that are both being
> rapidly extended at the same time -- and even then, it's only a
> performance problem, not a correctness problem.  In fact, I wonder if
> we shouldn't go further: instead of creating these RELEXTLOCK
> structures dynamically, let's just have a fixed number of them, say
> 1024.  When we get a request to take a lock, hash <dboid, reloid> and
> take the result modulo 1024; lock the RELEXTLOCK at that offset in the
> array.
>

Attached the latest patch incorporated comments except for the fix of
the memory size for relext lock.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachment

Moving_extension_lock_out_of_heavyweight_lock_v8.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

01 December 2017, 00:04:25

On Thu, Nov 30, 2017 at 6:20 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> This code ignores the existence of multiple databases; RELEXTLOCK
>> contains a relid, but no database OID.  That's easy enough to fix, but
>> it actually causes no problem unless, by bad luck, you have two
>> relations with the same OID in different databases that are both being
>> rapidly extended at the same time -- and even then, it's only a
>> performance problem, not a correctness problem.  In fact, I wonder if
>> we shouldn't go further: instead of creating these RELEXTLOCK
>> structures dynamically, let's just have a fixed number of them, say
>> 1024.  When we get a request to take a lock, hash <dboid, reloid> and
>> take the result modulo 1024; lock the RELEXTLOCK at that offset in the
>> array.
>
> Attached the latest patch incorporated comments except for the fix of
> the memory size for relext lock.

It doesn't do anything about the comment of mine quoted above.  Since
it's only possible to hold one relation extension lock at a time, we
don't really need the hash table here at all. We can just have an
array of 1024 or so locks and map every <db,relid> pair on to one of
them by hashing.  The worst thing we'll get it some false contention,
but that doesn't seem awful, and it would permit considerable further
simplification of this code -- and maybe make it faster in the
process, because we'd no longer need the hash table, or the pin count,
or the extra LWLocks that protect the hash table.  All we would have
is atomic operations manipulating the lock state, which seems like it
would be quite a lot faster and simpler.

BTW, I think RelExtLockReleaseAll is broken because it shouldn't
HOLD_INTERRUPTS(); I also think it's kind of silly to loop here when
we know we can only hold one lock.  Maybe RelExtLockRelease can take
bool force and do if (force) held_relextlock.nLocks = 0; else
held_relextlock.nLocks--.  Or, better yet, have the caller adjust that
value and then only call RelExtLockRelease() if we needed to release
the lock in shared memory.  That avoids needless branching.  On a
related note, is there any point in having both held_relextlock.nLocks
and num_held_relextlocks?

I think RelationExtensionLock should be a new type of IPC wait event,
rather than a whole new category.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

01 December 2017, 21:14:54

On Fri, Dec 1, 2017 at 10:26 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> On Fri, Dec 1, 2017 at 3:04 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Thu, Nov 30, 2017 at 6:20 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>>> This code ignores the existence of multiple databases; RELEXTLOCK
>>>> contains a relid, but no database OID.  That's easy enough to fix, but
>>>> it actually causes no problem unless, by bad luck, you have two
>>>> relations with the same OID in different databases that are both being
>>>> rapidly extended at the same time -- and even then, it's only a
>>>> performance problem, not a correctness problem.  In fact, I wonder if
>>>> we shouldn't go further: instead of creating these RELEXTLOCK
>>>> structures dynamically, let's just have a fixed number of them, say
>>>> 1024.  When we get a request to take a lock, hash <dboid, reloid> and
>>>> take the result modulo 1024; lock the RELEXTLOCK at that offset in the
>>>> array.
>>>
>>> Attached the latest patch incorporated comments except for the fix of
>>> the memory size for relext lock.
>>
>> It doesn't do anything about the comment of mine quoted above.
>
> Sorry I'd missed the comment.
>
>>   Since it's only possible to hold one relation extension lock at a time, we
>> don't really need the hash table here at all. We can just have an
>> array of 1024 or so locks and map every <db,relid> pair on to one of
>> them by hashing.  The worst thing we'll get it some false contention,
>> but that doesn't seem awful, and it would permit considerable further
>> simplification of this code -- and maybe make it faster in the
>> process, because we'd no longer need the hash table, or the pin count,
>> or the extra LWLocks that protect the hash table.  All we would have
>> is atomic operations manipulating the lock state, which seems like it
>> would be quite a lot faster and simpler.
>
> Agreed. With this change, we will have an array of the struct that has
> lock state and cv. The lock state has the wait count as well as the
> status of lock.
>
>> BTW, I think RelExtLockReleaseAll is broken because it shouldn't
>> HOLD_INTERRUPTS(); I also think it's kind of silly to loop here when
>> we know we can only hold one lock.  Maybe RelExtLockRelease can take
>> bool force and do if (force) held_relextlock.nLocks = 0; else
>> held_relextlock.nLocks--.  Or, better yet, have the caller adjust that
>> value and then only call RelExtLockRelease() if we needed to release
>> the lock in shared memory.  That avoids needless branching.
>
> Agreed. I'd vote for the latter.
>
>>  On a
>> related note, is there any point in having both held_relextlock.nLocks
>> and num_held_relextlocks?
>
> num_held_relextlocks is actually unnecessary, will be removed.
>
>> I think RelationExtensionLock should be a new type of IPC wait event,
>> rather than a whole new category.
>
> Hmm, I thought the wait event types of IPC seems related to events
> that communicates to other processes for the same purpose, for example
> parallel query, sync repli etc. On the other hand, the relation
> extension locks are one kind of the lock mechanism. That's way I added
> a new category. But maybe it can be fit to the IPC wait event.
>

Attached updated patch. I've done a performance measurement again on
the same configuration as before since the acquiring/releasing
procedures have been changed.

----- PATCHED -----
tps = 162.579320 (excluding connections establishing)
tps = 162.144352 (excluding connections establishing)
tps = 160.659403 (excluding connections establishing)
tps = 161.213995 (excluding connections establishing)
tps = 164.560460 (excluding connections establishing)
----- HEAD -----
tps = 157.738645 (excluding connections establishing)
tps = 146.178575 (excluding connections establishing)
tps = 143.788961 (excluding connections establishing)
tps = 144.886594 (excluding connections establishing)
tps = 145.496337 (excluding connections establishing)

* micro-benchmark
PATCHED = 1.61757e+07 (cycles/sec)
HEAD = 1.48685e+06 (cycles/sec)
The patched is 10 times faster than current HEAD.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachment

Moving_extension_lock_out_of_heavyweight_lock_v9.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

02 December 2017, 00:28:44

On Fri, Dec 1, 2017 at 10:14 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> The patched is 10 times faster than current HEAD.

Nifty.

The first hunk in monitoring.sgml looks unnecessary.

The second hunk breaks the formatting of the documentation; you need
to adjust the "morerows" value from 9 to 8 here:

         <entry morerows="9"><literal>Lock</literal></entry>

And similarly make this one 18:

         <entry morerows="17"><literal>IPC</literal></entry>

+* Relation extension locks.  The relation extension lock manager is
+specialized in relation extensions. In PostgreSQL 11 relation extension
+lock has been moved out of regular lock. It's similar to regular locks
+but doesn't have full dead lock detection, group locking and multiple
+lock modes. When conflicts occur, lock waits are implemented using
+condition variables.

Higher up, it says that "Postgres uses four types of interprocess
locks", but because you added this, it's now a list of five items.

I suggest moving the section on relation locks to the end and
rewriting the text here as follows: Only one process can extend a
relation at a time; we use a specialized lock manager for this
purpose, which is much simpler than the regular lock manager.  It is
similar to the lightweight lock mechanism, but is ever simpler because
there is only one lock mode and only one lock can be taken at a time.
A process holding a relation extension lock is interruptible, unlike a
process holding an LWLock.

+#define RelExtLockTargetTagToIndex(relextlock_tag) \
+    (tag_hash((const void *) relextlock_tag, sizeof(RelExtLockTag)) \
+        % N_RELEXTLOCK_ENTS)

How about using a static inline function for this?

+#define SET_RELEXTLOCK_TAG(locktag, d, r) \
+    ((locktag).dbid = (d), \
+     (locktag).relid = (r))

How about getting rid of this and just doing the assignments instead?

+#define RELEXT_VAL_LOCK     ((uint32) ((1 << 25)))
+#define RELEXT_LOCK_MASK    ((uint32) ((1 << 25)))

It seems confusing to have two macros for the same value and an
almost-interchangeable purpose.  Maybe just call it RELEXT_LOCK_BIT?

+RelationExtensionLockWaiterCount(Relation relation)

Hmm.  This is sort of problematic, because with then new design we
have no guarantee that the return value is actually accurate.  I don't
think that's a functional problem, but the optics aren't great.

+    if (held_relextlock.nLocks > 0)
+    {
+        RelExtLockRelease(held_relextlock.relid, true);
+    }

Excess braces.

+int
+RelExtLockHoldingLockCount(void)
+{
+    return held_relextlock.nLocks;
+}

Maybe IsAnyRelationExtensionLockHeld(), returning bool?

+    /* If the lock is held by me, no need to wait */

If we already hold the lock, no need to wait.

+     * Luckily if we're trying to acquire the same lock as what we
+     * had held just before, we don't need to get the entry from the
+     * array by hashing.

We're not trying to acquire a lock here.  "If the last relation
extension lock we touched is the same one for which we now need to
wait, we can use our cached pointer to the lock instead of recomputing
it."

+            registered_wait_list = true;

Isn't it really registered_wait_count?  The only list here is
encapsulated in the CV.

+    /* Before retuning, decrement the wait count if we had been waiting */

returning -> returning, but I'd rewrite this as "Release any wait
count we hold."

+ * Acquire the relation extension lock. If we're trying to acquire the same
+ * lock as what already held, we just increment nLock locally and return
+ * without touching the RelExtLock array.

"Acquire a relation extension lock."  I think you can forget the rest
of this; it duplicates comments in the function body.

+     * Since we don't support dead lock detection for relation extension
+     * lock and don't control the order of lock acquisition, it cannot not
+     * happen that trying to take a new lock while holding an another lock.

Since we don't do deadlock detection, caller must not try to take a
new relation extension lock while already holding them.

+        if (relid == held_relextlock.relid)
+        {
+            held_relextlock.nLocks++;
+            return true;
+        }
+        else
+            elog(ERROR,
+                 "cannot acquire relation extension locks for
multiple relations at the same");

I'd prefer if (relid != held_relextlock.relid) elog(ERROR, ...) to
save a level of indentation for the rest.

+     * If we're trying to acquire the same lock as what we just released
+     * we don't need to get the entry from the array by hashing. we expect
+     * to happen this case because it's a common case in acquisition of
+     * relation extension locks.

"If the last relation extension lock we touched is the same one for we
now need to acquire, we can use our cached pointer to the lock instead
of recomputing it.  This is likely to be a common case in practice."

+        /* Could not got the lock, return iff in conditional locking */

"locking conditionally"

+        ConditionVariableSleep(&(relextlock->cv),
WAIT_EVENT_RELATION_EXTENSION_LOCK);

Break line at comma

+    /* Decrement wait count if we had been waiting */

"Release any wait count we hold."

+    /* Always return true if not conditional lock */

"We got the lock!"

+    /* If force releasing, release all locks we're holding */
+    if (force)
+        held_relextlock.nLocks = 0;
+    else
+        held_relextlock.nLocks--;
+
+    Assert(held_relextlock.nLocks >= 0);
+
+    /* Return if we're still holding the lock even after computation */
+    if (held_relextlock.nLocks > 0)
+        return;

I thought you were going to have the caller adjust nLocks?

+    /* Get RelExtLock entry from the array */
+    SET_RELEXTLOCK_TAG(tag, MyDatabaseId, relid);
+    relextlock = &RelExtLockArray[RelExtLockTargetTagToIndex(&tag)];

This seems to make no sense in RelExtLockRelease -- isn't the cache
guaranteed valid?

+    /* Wake up waiters if there is someone looking at this lock */

"If there may be waiters, wake them up."

+     * We allow to take a relation extension lock after took a
+     * heavy-weight lock. However, since we don't have dead lock
+     * detection mechanism between heavy-weight lock and relation
+     * extension lock it's not allowed taking an another heavy-weight
+     * lock while holding a relation extension lock.

"Relation extension locks don't participate in deadlock detection, so
make sure we don't try to acquire a heavyweight lock while holding
one."

+    /* Release relation extension locks */

"If we hold a relation extension lock, release it."

+/* Number of partitions the shared relation extension lock tables are
divided into */
+#define LOG2_NUM_RELEXTLOCK_PARTITIONS 4
+#define NUM_RELEXTLOCK_PARTITIONS      (1 << LOG2_NUM_RELEXTLOCK_PARTITIONS)

Dead code.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

02 December 2017, 02:32:00

On Fri, Dec 1, 2017 at 1:28 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> [ lots of minor comments ]

When I took a break from sitting at the computer, I realized that I
think this has a more serious problem: won't it permanently leak
reference counts if someone hits ^C or an error occurs while the lock
is held?  I think it will -- it probably needs to do cleanup at the
places where we do LWLockReleaseAll() that includes decrementing the
shared refcount if necessary, rather than doing cleanup at the places
we release heavyweight locks.

I might be wrong about the details here -- this is off the top of my head.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

08 December 2017, 14:20:10

On Sat, Dec 2, 2017 at 3:28 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Dec 1, 2017 at 10:14 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> The patched is 10 times faster than current HEAD.
>
> Nifty.

Thank you for your dedicated reviewing the patch.

> The first hunk in monitoring.sgml looks unnecessary.

You meant the following hunk?

diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 8d461c8..7aa7981 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -669,8 +669,8 @@ postgres   27093  0.0  0.0  30096  2752 ?
Ss   11:34   0:00 postgres: ser
           Heavyweight locks, also known as lock manager locks or simply locks,
           primarily protect SQL-visible objects such as tables.  However,
           they are also used to ensure mutual exclusion for certain internal
-          operations such as relation extension.
<literal>wait_event</literal> will
-          identify the type of lock awaited.
+          operations such as waiting for a transaction to finish.
+          <literal>wait_event</literal> will identify the type of lock awaited.
          </para>
         </listitem>
         <listitem>

I think that since the extension locks are no longer a part of
heavyweight locks we should change the explanation.

> The second hunk breaks the formatting of the documentation; you need
> to adjust the "morerows" value from 9 to 8 here:
>
>          <entry morerows="9"><literal>Lock</literal></entry>
>
> And similarly make this one 18:
>
>          <entry morerows="17"><literal>IPC</literal></entry>

Fixed.

> +* Relation extension locks.  The relation extension lock manager is
> +specialized in relation extensions. In PostgreSQL 11 relation extension
> +lock has been moved out of regular lock. It's similar to regular locks
> +but doesn't have full dead lock detection, group locking and multiple
> +lock modes. When conflicts occur, lock waits are implemented using
> +condition variables.
>
> Higher up, it says that "Postgres uses four types of interprocess
> locks", but because you added this, it's now a list of five items.

Fixed.

> I suggest moving the section on relation locks to the end and
> rewriting the text here as follows: Only one process can extend a
> relation at a time; we use a specialized lock manager for this
> purpose, which is much simpler than the regular lock manager.  It is
> similar to the lightweight lock mechanism, but is ever simpler because
> there is only one lock mode and only one lock can be taken at a time.
> A process holding a relation extension lock is interruptible, unlike a
> process holding an LWLock.

Agreed and fixed.

> +#define RelExtLockTargetTagToIndex(relextlock_tag) \
> +    (tag_hash((const void *) relextlock_tag, sizeof(RelExtLockTag)) \
> +        % N_RELEXTLOCK_ENTS)
>
> How about using a static inline function for this?

Fixed.

> +#define SET_RELEXTLOCK_TAG(locktag, d, r) \
> +    ((locktag).dbid = (d), \
> +     (locktag).relid = (r))
>
> How about getting rid of this and just doing the assignments instead?

Fixed.

> +#define RELEXT_VAL_LOCK     ((uint32) ((1 << 25)))
> +#define RELEXT_LOCK_MASK    ((uint32) ((1 << 25)))
>
> It seems confusing to have two macros for the same value and an
> almost-interchangeable purpose.  Maybe just call it RELEXT_LOCK_BIT?

Fixed.

>
> +RelationExtensionLockWaiterCount(Relation relation)
>
> Hmm.  This is sort of problematic, because with then new design we
> have no guarantee that the return value is actually accurate.  I don't
> think that's a functional problem, but the optics aren't great.

Yeah, with this patch we could overestimate it and then add extra
blocks to the relation. Since the number of extra blocks is capped at
512 I think it would not become serious problem.

> +    if (held_relextlock.nLocks > 0)
> +    {
> +        RelExtLockRelease(held_relextlock.relid, true);
> +    }
>
> Excess braces.

Fixed.

>
> +int
> +RelExtLockHoldingLockCount(void)
> +{
> +    return held_relextlock.nLocks;
> +}
>
> Maybe IsAnyRelationExtensionLockHeld(), returning bool?

Fixed.

> +    /* If the lock is held by me, no need to wait */
>
> If we already hold the lock, no need to wait.

Fixed.

> +     * Luckily if we're trying to acquire the same lock as what we
> +     * had held just before, we don't need to get the entry from the
> +     * array by hashing.
>
> We're not trying to acquire a lock here.  "If the last relation
> extension lock we touched is the same one for which we now need to
> wait, we can use our cached pointer to the lock instead of recomputing
> it."

Fixed.

> +            registered_wait_list = true;
>
> Isn't it really registered_wait_count?  The only list here is
> encapsulated in the CV.

Changed to "waiting".

>
> +    /* Before retuning, decrement the wait count if we had been waiting */
>
> returning -> returning, but I'd rewrite this as "Release any wait
> count we hold."

Fixed.

> + * Acquire the relation extension lock. If we're trying to acquire the same
> + * lock as what already held, we just increment nLock locally and return
> + * without touching the RelExtLock array.
>
> "Acquire a relation extension lock."  I think you can forget the rest
> of this; it duplicates comments in the function body.

Fixed.

> +     * Since we don't support dead lock detection for relation extension
> +     * lock and don't control the order of lock acquisition, it cannot not
> +     * happen that trying to take a new lock while holding an another lock.
>
> Since we don't do deadlock detection, caller must not try to take a
> new relation extension lock while already holding them.

Fixed.

>
> +        if (relid == held_relextlock.relid)
> +        {
> +            held_relextlock.nLocks++;
> +            return true;
> +        }
> +        else
> +            elog(ERROR,
> +                 "cannot acquire relation extension locks for
> multiple relations at the same");
>
> I'd prefer if (relid != held_relextlock.relid) elog(ERROR, ...) to
> save a level of indentation for the rest.

Fixed.

>
> +     * If we're trying to acquire the same lock as what we just released
> +     * we don't need to get the entry from the array by hashing. we expect
> +     * to happen this case because it's a common case in acquisition of
> +     * relation extension locks.
>
> "If the last relation extension lock we touched is the same one for we
> now need to acquire, we can use our cached pointer to the lock instead
> of recomputing it.  This is likely to be a common case in practice."

Fixed.

>
> +        /* Could not got the lock, return iff in conditional locking */
>
> "locking conditionally"

Fixed.

> +        ConditionVariableSleep(&(relextlock->cv),
> WAIT_EVENT_RELATION_EXTENSION_LOCK);
> Break line at comma
>

Fixed.

> +    /* Decrement wait count if we had been waiting */
>
> "Release any wait count we hold."

Fixed.

> +    /* Always return true if not conditional lock */
>
> "We got the lock!"

Fixed.

> +    /* If force releasing, release all locks we're holding */
> +    if (force)
> +        held_relextlock.nLocks = 0;
> +    else
> +        held_relextlock.nLocks--;
> +
> +    Assert(held_relextlock.nLocks >= 0);
> +
> +    /* Return if we're still holding the lock even after computation */
> +    if (held_relextlock.nLocks > 0)
> +        return;
>
> I thought you were going to have the caller adjust nLocks?

Yeah, I was supposed to change so but since we always release either
one lock or all relext locks I thought it'd better to pass a bool
rather than an int.

> +    /* Get RelExtLock entry from the array */
> +    SET_RELEXTLOCK_TAG(tag, MyDatabaseId, relid);
> +    relextlock = &RelExtLockArray[RelExtLockTargetTagToIndex(&tag)];
>
> This seems to make no sense in RelExtLockRelease -- isn't the cache
> guaranteed valid?

Right, fixed.

>
> +    /* Wake up waiters if there is someone looking at this lock */
>
> "If there may be waiters, wake them up."

Fixed.

> +     * We allow to take a relation extension lock after took a
> +     * heavy-weight lock. However, since we don't have dead lock
> +     * detection mechanism between heavy-weight lock and relation
> +     * extension lock it's not allowed taking an another heavy-weight
> +     * lock while holding a relation extension lock.
>
> "Relation extension locks don't participate in deadlock detection, so
> make sure we don't try to acquire a heavyweight lock while holding
> one."

Fixed.

> +    /* Release relation extension locks */
>
> "If we hold a relation extension lock, release it."

Fixed.

> +/* Number of partitions the shared relation extension lock tables are
> divided into */
> +#define LOG2_NUM_RELEXTLOCK_PARTITIONS 4
> +#define NUM_RELEXTLOCK_PARTITIONS      (1 << LOG2_NUM_RELEXTLOCK_PARTITIONS)
>
> Dead code.

Fixed.

> When I took a break from sitting at the computer, I realized that I
> think this has a more serious problem: won't it permanently leak
> reference counts if someone hits ^C or an error occurs while the lock
> is held?  I think it will -- it probably needs to do cleanup at the
> places where we do LWLockReleaseAll() that includes decrementing the
> shared refcount if necessary, rather than doing cleanup at the places
> we release heavyweight locks.
> I might be wrong about the details here -- this is off the top of my head.

Good catch. It can leak reference counts if someone hits ^C or an
error occurs while waiting. Fixed in the latest patch. But since
RelExtLockReleaseAll() is called even when such situations I think we
don't need to change the place where releasing the all relext lock. We
just moved it from heavyweight locks. Am I missing something?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachment

Moving_extension_lock_out_of_heavyweight_lock_v10.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

08 December 2017, 23:00:24

On Fri, Dec 8, 2017 at 3:20 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> The first hunk in monitoring.sgml looks unnecessary.
>
> You meant the following hunk?
>
> diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
> index 8d461c8..7aa7981 100644
> --- a/doc/src/sgml/monitoring.sgml
> +++ b/doc/src/sgml/monitoring.sgml
> @@ -669,8 +669,8 @@ postgres   27093  0.0  0.0  30096  2752 ?
> Ss   11:34   0:00 postgres: ser
>            Heavyweight locks, also known as lock manager locks or simply locks,
>            primarily protect SQL-visible objects such as tables.  However,
>            they are also used to ensure mutual exclusion for certain internal
> -          operations such as relation extension.
> <literal>wait_event</literal> will
> -          identify the type of lock awaited.
> +          operations such as waiting for a transaction to finish.
> +          <literal>wait_event</literal> will identify the type of lock awaited.
>           </para>
>          </listitem>
>          <listitem>
>
> I think that since the extension locks are no longer a part of
> heavyweight locks we should change the explanation.

Yes, you are right.

>> +RelationExtensionLockWaiterCount(Relation relation)
>>
>> Hmm.  This is sort of problematic, because with then new design we
>> have no guarantee that the return value is actually accurate.  I don't
>> think that's a functional problem, but the optics aren't great.
>
> Yeah, with this patch we could overestimate it and then add extra
> blocks to the relation. Since the number of extra blocks is capped at
> 512 I think it would not become serious problem.

How about renaming it EstimateNumberOfExtensionLockWaiters?

>> +    /* If force releasing, release all locks we're holding */
>> +    if (force)
>> +        held_relextlock.nLocks = 0;
>> +    else
>> +        held_relextlock.nLocks--;
>> +
>> +    Assert(held_relextlock.nLocks >= 0);
>> +
>> +    /* Return if we're still holding the lock even after computation */
>> +    if (held_relextlock.nLocks > 0)
>> +        return;
>>
>> I thought you were going to have the caller adjust nLocks?
>
> Yeah, I was supposed to change so but since we always release either
> one lock or all relext locks I thought it'd better to pass a bool
> rather than an int.

I don't see why you need to pass either one.  The caller can set
held_relextlock.nLocks either with -- or = 0, and then call
RelExtLockRelease() only if the resulting value is 0.

>> When I took a break from sitting at the computer, I realized that I
>> think this has a more serious problem: won't it permanently leak
>> reference counts if someone hits ^C or an error occurs while the lock
>> is held?  I think it will -- it probably needs to do cleanup at the
>> places where we do LWLockReleaseAll() that includes decrementing the
>> shared refcount if necessary, rather than doing cleanup at the places
>> we release heavyweight locks.
>> I might be wrong about the details here -- this is off the top of my head.
>
> Good catch. It can leak reference counts if someone hits ^C or an
> error occurs while waiting. Fixed in the latest patch. But since
> RelExtLockReleaseAll() is called even when such situations I think we
> don't need to change the place where releasing the all relext lock. We
> just moved it from heavyweight locks. Am I missing something?

Hmm, that might be an OK way to handle it.  I don't see a problem off
the top of my head.  It might be clearer to rename it to
RelExtLockCleanup() though, since it is not just releasing the lock
but also any wait count we hold.

+/* Must be greater than MAX_BACKENDS - which is 2^23-1, so we're fine. */
+#define RELEXT_WAIT_COUNT_MASK    ((uint32) ((1 << 24) - 1))

Let's drop the comment here and instead add a StaticAssertStmt() that
checks this.

I am slightly puzzled, though.  If I read this correctly, bits 0-23
are used for the waiter count, bit 24 is always 0, bit 25 indicates
the presence or absence of an exclusive lock, and bits 26+ are always
0.  That seems slightly odd.  Shouldn't we either use the highest
available bit for the locker (bit 31) or the lowest one (bit 24)?  The
former seems better, in case MAX_BACKENDS changes later.  We could
make RELEXT_WAIT_COUNT_MASK bigger too, just in case.

+        /* Make a lock tag */
+        tag.dbid = MyDatabaseId;
+        tag.relid = relid;

What about shared relations?  I bet we need to use 0 in that case.
Otherwise, if backends in two different databases try to extend the
same shared relation at the same time, we'll (probably) fail to notice
that they conflict.

+ * To avoid unnecessary recomputations of the hash code, we try to do this
+ * just once per function, and then pass it around as needed.  we can
+ * extract the index number of RelExtLockArray.

This is just a copy-and-paste from lock.c, but actually we have a more
sophisticated scheme here.  I think you can just drop this comment
altogether, really.

+    return (tag_hash((const void *) locktag, sizeof(RelExtLockTag))
+            % N_RELEXTLOCK_ENTS);

I would drop the outermost set of parentheses.  Is the cast to (const
void *) really doing anything?

+                 "cannot acquire relation extension locks for
multiple relations at the same");

cannot simultaneously acquire more than one distinct relation lock?
As you have it, you'd have to add the word "time" at the end, but my
version is shorter.

+        /* Sleep until the lock is released */

Really, there's no guarantee that the lock will be released when we
wake up.  I think just /* Sleep until something happens, then recheck
*/

+        lock_free = (oldstate & RELEXT_LOCK_BIT) == 0;
+        if (lock_free)
+            desired_state += RELEXT_LOCK_BIT;
+
+        if (pg_atomic_compare_exchange_u32(&relextlock->state,
+                                           &oldstate, desired_state))
+        {
+            if (lock_free)
+                return false;
+            else
+                return true;
+        }

Hmm.  If the lock is not free, we attempt to compare-and-swap anyway,
but then return false?  Why not just lock_free = (oldstate &
RELEXT_LOCK_BIT) == 0; if (!lock_free) return true; if
(pg_atomic_compare_exchange(&relextlock->state, &oldstate, oldstate |
RELEXT_LOCK_BIT)) return false;

+    Assert(IsAnyRelationExtensionLockHeld() == 0);

Since this is return bool now, it should just be
Assert(!IsAnyRelationExtensionLockHeld()).

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

11 December 2017, 10:51:57

On Sat, Dec 9, 2017 at 2:00 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Dec 8, 2017 at 3:20 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> The first hunk in monitoring.sgml looks unnecessary.
>>
>> You meant the following hunk?
>>
>> diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
>> index 8d461c8..7aa7981 100644
>> --- a/doc/src/sgml/monitoring.sgml
>> +++ b/doc/src/sgml/monitoring.sgml
>> @@ -669,8 +669,8 @@ postgres   27093  0.0  0.0  30096  2752 ?
>> Ss   11:34   0:00 postgres: ser
>>            Heavyweight locks, also known as lock manager locks or simply locks,
>>            primarily protect SQL-visible objects such as tables.  However,
>>            they are also used to ensure mutual exclusion for certain internal
>> -          operations such as relation extension.
>> <literal>wait_event</literal> will
>> -          identify the type of lock awaited.
>> +          operations such as waiting for a transaction to finish.
>> +          <literal>wait_event</literal> will identify the type of lock awaited.
>>           </para>
>>          </listitem>
>>          <listitem>
>>
>> I think that since the extension locks are no longer a part of
>> heavyweight locks we should change the explanation.
>
> Yes, you are right.
>
>>> +RelationExtensionLockWaiterCount(Relation relation)
>>>
>>> Hmm.  This is sort of problematic, because with then new design we
>>> have no guarantee that the return value is actually accurate.  I don't
>>> think that's a functional problem, but the optics aren't great.
>>
>> Yeah, with this patch we could overestimate it and then add extra
>> blocks to the relation. Since the number of extra blocks is capped at
>> 512 I think it would not become serious problem.
>
> How about renaming it EstimateNumberOfExtensionLockWaiters?

Agreed, fixed.

>
>>> +    /* If force releasing, release all locks we're holding */
>>> +    if (force)
>>> +        held_relextlock.nLocks = 0;
>>> +    else
>>> +        held_relextlock.nLocks--;
>>> +
>>> +    Assert(held_relextlock.nLocks >= 0);
>>> +
>>> +    /* Return if we're still holding the lock even after computation */
>>> +    if (held_relextlock.nLocks > 0)
>>> +        return;
>>>
>>> I thought you were going to have the caller adjust nLocks?
>>
>> Yeah, I was supposed to change so but since we always release either
>> one lock or all relext locks I thought it'd better to pass a bool
>> rather than an int.
>
> I don't see why you need to pass either one.  The caller can set
> held_relextlock.nLocks either with -- or = 0, and then call
> RelExtLockRelease() only if the resulting value is 0.

Fixed.

>
>>> When I took a break from sitting at the computer, I realized that I
>>> think this has a more serious problem: won't it permanently leak
>>> reference counts if someone hits ^C or an error occurs while the lock
>>> is held?  I think it will -- it probably needs to do cleanup at the
>>> places where we do LWLockReleaseAll() that includes decrementing the
>>> shared refcount if necessary, rather than doing cleanup at the places
>>> we release heavyweight locks.
>>> I might be wrong about the details here -- this is off the top of my head.
>>
>> Good catch. It can leak reference counts if someone hits ^C or an
>> error occurs while waiting. Fixed in the latest patch. But since
>> RelExtLockReleaseAll() is called even when such situations I think we
>> don't need to change the place where releasing the all relext lock. We
>> just moved it from heavyweight locks. Am I missing something?
>
> Hmm, that might be an OK way to handle it.  I don't see a problem off
> the top of my head.  It might be clearer to rename it to
> RelExtLockCleanup() though, since it is not just releasing the lock
> but also any wait count we hold.

Yeah, it seems better. Fixed.

> +/* Must be greater than MAX_BACKENDS - which is 2^23-1, so we're fine. */
> +#define RELEXT_WAIT_COUNT_MASK    ((uint32) ((1 << 24) - 1))
>
> Let's drop the comment here and instead add a StaticAssertStmt() that
> checks this.

Fixed. I added StaticAssertStmt() to InitRelExtLocks().

>
> I am slightly puzzled, though.  If I read this correctly, bits 0-23
> are used for the waiter count, bit 24 is always 0, bit 25 indicates
> the presence or absence of an exclusive lock, and bits 26+ are always
> 0.  That seems slightly odd.  Shouldn't we either use the highest
> available bit for the locker (bit 31) or the lowest one (bit 24)?  The
> former seems better, in case MAX_BACKENDS changes later.  We could
> make RELEXT_WAIT_COUNT_MASK bigger too, just in case.

I agree with the former. Fixed.

> +        /* Make a lock tag */
> +        tag.dbid = MyDatabaseId;
> +        tag.relid = relid;
>
> What about shared relations?  I bet we need to use 0 in that case.
> Otherwise, if backends in two different databases try to extend the
> same shared relation at the same time, we'll (probably) fail to notice
> that they conflict.
>

You're right. I changed it so that we set invalidOId to tag.dbid if
the relation is shared relation.


> + * To avoid unnecessary recomputations of the hash code, we try to do this
> + * just once per function, and then pass it around as needed.  we can
> + * extract the index number of RelExtLockArray.
>
> This is just a copy-and-paste from lock.c, but actually we have a more
> sophisticated scheme here.  I think you can just drop this comment
> altogether, really.

Fixed.

>
> +    return (tag_hash((const void *) locktag, sizeof(RelExtLockTag))
> +            % N_RELEXTLOCK_ENTS);
>
> I would drop the outermost set of parentheses.  Is the cast to (const
> void *) really doing anything?
>

Fixed.

> +                 "cannot acquire relation extension locks for
> multiple relations at the same");
>
> cannot simultaneously acquire more than one distinct relation lock?
> As you have it, you'd have to add the word "time" at the end, but my
> version is shorter.

I wanted to mean, cannot acquire relation extension locks for multiple
relations at the "time". Fixed.

>
> +        /* Sleep until the lock is released */
>
> Really, there's no guarantee that the lock will be released when we
> wake up.  I think just /* Sleep until something happens, then recheck
> */

Fixed.

> +        lock_free = (oldstate & RELEXT_LOCK_BIT) == 0;
> +        if (lock_free)
> +            desired_state += RELEXT_LOCK_BIT;
> +
> +        if (pg_atomic_compare_exchange_u32(&relextlock->state,
> +                                           &oldstate, desired_state))
> +        {
> +            if (lock_free)
> +                return false;
> +            else
> +                return true;
> +        }
>
> Hmm.  If the lock is not free, we attempt to compare-and-swap anyway,
> but then return false?  Why not just lock_free = (oldstate &
> RELEXT_LOCK_BIT) == 0; if (!lock_free) return true; if
> (pg_atomic_compare_exchange(&relextlock->state, &oldstate, oldstate |
> RELEXT_LOCK_BIT)) return false;

Fixed.

>
> +    Assert(IsAnyRelationExtensionLockHeld() == 0);
>
> Since this is return bool now, it should just be
> Assert(!IsAnyRelationExtensionLockHeld()).

Fixed.

Attached updated version patch. Please review it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachment

Moving_extension_lock_out_of_heavyweight_lock_v11.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

12 December 2017, 02:15:50

On Sun, Dec 10, 2017 at 11:51 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> Attached updated version patch. Please review it.

I went over this today; please find attached an updated version which
I propose to commit.

Changes:

- Various formatting fixes, including running pgindent.

- Various comment updates.

- Make RELEXT_WAIT_COUNT_MASK equal RELEXT_LOCK_BIT - 1 rather than
some unnecessarily smaller number.

- In InitRelExtLocks, don't bother using mul_size; we already know it
won't overflow, because we did the same thing in RelExtLockShmemSize.

- When we run into an error trying to release a lock, log it as a
WARNING and don't mark it as translatable.  Follows lock.c.  An ERROR
here probably just recurses infinitely.

- Don't bother passing OID to RelExtLockRelease.

- Reorder functions a bit for (IMHO) better clarity.

- Make UnlockRelationForExtension just use a single message for both
failure modes.  They are closely-enough related that I think that's
fine.

- Make WaitForRelationExtensionLockToBeFree complain if we already
hold an extension lock.

- In RelExtLockCleanup, clear held_relextlock.waiting.  This would've
made for a nasty bug.

- Also in that function, assert that we don't hold both a lock and a wait count.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Attachment

extension-lock-v12.patch

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

12 December 2017, 02:25:12

Hi,

On 2017-12-11 15:15:50 -0500, Robert Haas wrote:
> +* Relation extension locks. Only one process can extend a relation at
> +a time; we use a specialized lock manager for this purpose, which is
> +much simpler than the regular lock manager.  It is similar to the
> +lightweight lock mechanism, but is ever simpler because there is only
> +one lock mode and only one lock can be taken at a time. A process holding
> +a relation extension lock is interruptible, unlike a process holding an
> +LWLock.


> +/*-------------------------------------------------------------------------
> + *
> + * extension_lock.c
> + *      Relation extension lock manager
> + *
> + * This specialized lock manager is used only for relation extension
> + * locks.  Unlike the heavyweight lock manager, it doesn't provide
> + * deadlock detection or group locking.  Unlike lwlock.c, extension lock
> + * waits are interruptible.  Unlike both systems, there is only one lock
> + * mode.
> + *
> + * False sharing is possible.  We have a fixed-size array of locks, and
> + * every database OID/relation OID combination is mapped to a slot in
> + * the array.  Therefore, if two processes try to extend relations that
> + * map to the same array slot, they will contend even though it would
> + * be OK to let both proceed at once.  Since these locks are typically
> + * taken only for very short periods of time, this doesn't seem likely
> + * to be a big problem in practice.  If it is, we could make the array
> + * bigger.

For me "very short periods of time" and journaled metadatachanging
filesystem operations don't quite mesh.  Language lawyering aside, this
seems quite likely to bite us down the road.

It's imo perfectly fine to say that there's only a limited number of
file extension locks, but that there's a far from neglegible chance of
conflict even without the array being full doesn't seem nice. Think this
needs use some open addressing like conflict handling or something
alike.

Greetings,

Andres Freund

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

12 December 2017, 02:55:42

On Mon, Dec 11, 2017 at 3:25 PM, Andres Freund <andres@anarazel.de> wrote:
> For me "very short periods of time" and journaled metadatachanging
> filesystem operations don't quite mesh.  Language lawyering aside, this
> seems quite likely to bite us down the road.
>
> It's imo perfectly fine to say that there's only a limited number of
> file extension locks, but that there's a far from neglegible chance of
> conflict even without the array being full doesn't seem nice. Think this
> needs use some open addressing like conflict handling or something
> alike.

I guess we could consider that, but I'm not really convinced that it's
solving a real problem.  Right now, you start having meaningful chance
of lock-manager lock contention when the number of concurrent
processes in the system requesting heavyweight locks is still in the
single digits, because there are only 16 lock-manager locks.  With
this, there are effectively 1024 partitions.

Now I realize you're going to point out, not wrongly, that we're
contending on the locks themselves rather than the locks protecting
the locks, and that this makes everything worse because the hold time
is much longer.  Fair enough.  On the other hand, what workload would
actually be harmed?  I think you basically have to imagine a lot of
relations being extended simultaneously, like a parallel bulk load,
and an underlying filesystem which performs individual operations
slowly but scales really well.  I'm slightly skeptical that's how
real-world filesystems behave.

It might be a good idea, though, to test how parallel bulk loading
behaves with this patch applied, maybe even after reducing
N_RELEXTLOCK_ENTS to simulate an unfortunate number of collisions.

This isn't a zero-sum game.  If we add collision resolution, we're
going to slow down the ordinary uncontended case; the bookkeeping will
get significantly more complicated.  That is only worth doing if the
current behavior produces pathological cases on workloads that are
actually somewhat realistic.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

12 December 2017, 03:10:24

On Tue, Dec 12, 2017 at 5:15 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sun, Dec 10, 2017 at 11:51 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> Attached updated version patch. Please review it.
>
> I went over this today; please find attached an updated version which
> I propose to commit.
>
> Changes:
>
> - Various formatting fixes, including running pgindent.
>
> - Various comment updates.
>
> - Make RELEXT_WAIT_COUNT_MASK equal RELEXT_LOCK_BIT - 1 rather than
> some unnecessarily smaller number.
>
> - In InitRelExtLocks, don't bother using mul_size; we already know it
> won't overflow, because we did the same thing in RelExtLockShmemSize.
>
> - When we run into an error trying to release a lock, log it as a
> WARNING and don't mark it as translatable.  Follows lock.c.  An ERROR
> here probably just recurses infinitely.
>
> - Don't bother passing OID to RelExtLockRelease.
>
> - Reorder functions a bit for (IMHO) better clarity.
>
> - Make UnlockRelationForExtension just use a single message for both
> failure modes.  They are closely-enough related that I think that's
> fine.
>
> - Make WaitForRelationExtensionLockToBeFree complain if we already
> hold an extension lock.
>
> - In RelExtLockCleanup, clear held_relextlock.waiting.  This would've
> made for a nasty bug.
>
> - Also in that function, assert that we don't hold both a lock and a wait count.
>

Thank you for updating the patch. Here is two minor comments.

+ * we acquire the same relation extension lock repeatedly.  nLocks is 0 is the
+ * number of times we've acquired that lock;

Should it be "nLocks is the number of times we've acquired that lock:"?

+    /* Remember lock held by this backend */
+    held_relextlock.relid = relid;
+    held_relextlock.lock = relextlock;
+    held_relextlock.nLocks = 1;

We set held_relextlock.relid and held_relextlock.lock again. Can we remove them?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

12 December 2017, 03:32:31

On 2017-12-11 15:55:42 -0500, Robert Haas wrote:
> On Mon, Dec 11, 2017 at 3:25 PM, Andres Freund <andres@anarazel.de> wrote:
> > For me "very short periods of time" and journaled metadatachanging
> > filesystem operations don't quite mesh.  Language lawyering aside, this
> > seems quite likely to bite us down the road.
> >
> > It's imo perfectly fine to say that there's only a limited number of
> > file extension locks, but that there's a far from neglegible chance of
> > conflict even without the array being full doesn't seem nice. Think this
> > needs use some open addressing like conflict handling or something
> > alike.
> 
> I guess we could consider that, but I'm not really convinced that it's
> solving a real problem.  Right now, you start having meaningful chance
> of lock-manager lock contention when the number of concurrent
> processes in the system requesting heavyweight locks is still in the
> single digits, because there are only 16 lock-manager locks.  With
> this, there are effectively 1024 partitions.
> 
> Now I realize you're going to point out, not wrongly, that we're
> contending on the locks themselves rather than the locks protecting
> the locks, and that this makes everything worse because the hold time
> is much longer.

Indeed.


> Fair enough.  On the other hand, what workload would actually be
> harmed?  I think you basically have to imagine a lot of relations
> being extended simultaneously, like a parallel bulk load, and an
> underlying filesystem which performs individual operations slowly but
> scales really well.  I'm slightly skeptical that's how real-world
> filesystems behave.

Or just two independent relations on two different filesystems.


> It might be a good idea, though, to test how parallel bulk loading
> behaves with this patch applied, maybe even after reducing
> N_RELEXTLOCK_ENTS to simulate an unfortunate number of collisions.

Yea, that sounds like a good plan. Measure two COPYs to relations on
different filesystems, reduce N_RELEXTLOCK_ENTS to 1, and measure
performance. Then increase the concurrency of the copies to each
relation.


> This isn't a zero-sum game.  If we add collision resolution, we're
> going to slow down the ordinary uncontended case; the bookkeeping will
> get significantly more complicated.  That is only worth doing if the
> current behavior produces pathological cases on workloads that are
> actually somewhat realistic.

Yea, measuring sounds like a good plan.

Greetings,

Andres Freund

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

12 December 2017, 21:42:35

On Mon, Dec 11, 2017 at 4:10 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> Thank you for updating the patch. Here is two minor comments.
>
> + * we acquire the same relation extension lock repeatedly.  nLocks is 0 is the
> + * number of times we've acquired that lock;
>
> Should it be "nLocks is the number of times we've acquired that lock:"?

Yes.

> +    /* Remember lock held by this backend */
> +    held_relextlock.relid = relid;
> +    held_relextlock.lock = relextlock;
> +    held_relextlock.nLocks = 1;
>
> We set held_relextlock.relid and held_relextlock.lock again. Can we remove them?

Yes.

Can you also try the experiment Andres mentions: "Measure two COPYs to
relations on different filesystems, reduce N_RELEXTLOCK_ENTS to 1, and
measure performance. Then increase the concurrency of the copies to
each relation."  We want to see whether and how much this regresses
performance in that case.  It simulates the case of a hash collision.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

13 December 2017, 13:02:45

On Wed, Dec 13, 2017 at 12:42 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Dec 11, 2017 at 4:10 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> Thank you for updating the patch. Here is two minor comments.
>>
>> + * we acquire the same relation extension lock repeatedly.  nLocks is 0 is the
>> + * number of times we've acquired that lock;
>>
>> Should it be "nLocks is the number of times we've acquired that lock:"?
>
> Yes.
>
>> +    /* Remember lock held by this backend */
>> +    held_relextlock.relid = relid;
>> +    held_relextlock.lock = relextlock;
>> +    held_relextlock.nLocks = 1;
>>
>> We set held_relextlock.relid and held_relextlock.lock again. Can we remove them?
>
> Yes.
>
> Can you also try the experiment Andres mentions: "Measure two COPYs to
> relations on different filesystems, reduce N_RELEXTLOCK_ENTS to 1, and
> measure performance.

Yes. I'll measure the performance on such environment.

> Then increase the concurrency of the copies to
> each relation."  We want to see whether and how much this regresses
> performance in that case.  It simulates the case of a hash collision.
>

When we add extra blocks on a relation do we access to the disk? I
guess we just call lseek and write and don't access to the disk. If so
the performance degradation regression might not be much.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

13 December 2017, 13:30:48

On 2017-12-13 16:02:45 +0900, Masahiko Sawada wrote:
> When we add extra blocks on a relation do we access to the disk? I
> guess we just call lseek and write and don't access to the disk. If so
> the performance degradation regression might not be much.

Usually changes in the file size require the filesystem to perform
metadata operations, which in turn requires journaling on most
FSs. Which'll often result in synchronous disk writes.

Greetings,

Andres Freund

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

13 December 2017, 14:57:42

On Wed, Dec 13, 2017 at 4:30 PM, Andres Freund <andres@anarazel.de> wrote:
> On 2017-12-13 16:02:45 +0900, Masahiko Sawada wrote:
>> When we add extra blocks on a relation do we access to the disk? I
>> guess we just call lseek and write and don't access to the disk. If so
>> the performance degradation regression might not be much.
>
> Usually changes in the file size require the filesystem to perform
> metadata operations, which in turn requires journaling on most
> FSs. Which'll often result in synchronous disk writes.
>

Thank you. I understood the reason why this measurement should use two
different filesystems.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

14 December 2017, 16:45:48

On Wed, Dec 13, 2017 at 5:57 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> On Wed, Dec 13, 2017 at 4:30 PM, Andres Freund <andres@anarazel.de> wrote:
>> On 2017-12-13 16:02:45 +0900, Masahiko Sawada wrote:
>>> When we add extra blocks on a relation do we access to the disk? I
>>> guess we just call lseek and write and don't access to the disk. If so
>>> the performance degradation regression might not be much.
>>
>> Usually changes in the file size require the filesystem to perform
>> metadata operations, which in turn requires journaling on most
>> FSs. Which'll often result in synchronous disk writes.
>>
>
> Thank you. I understood the reason why this measurement should use two
> different filesystems.
>

Here is the result.
I've measured the through-put with some cases on my virtual machine.
Each client loads 48k file to each different relations located on
either xfs filesystem or ext4 filesystem, for 30 sec.

Case 1: COPYs to relations on different filessystems(xfs and ext4) and
N_RELEXTLOCK_ENTS is 1024

clients = 2, avg = 296.2068
clients = 5, avg = 372.0707
clients = 10, avg = 389.8850
clients = 50, avg = 428.8050

Case 2: COPYs to relations on different filessystems(xfs and ext4) and
N_RELEXTLOCK_ENTS is 1

clients = 2, avg = 294.3633
clients = 5, avg = 358.9364
clients = 10, avg = 383.6945
clients = 50, avg = 424.3687

And the result of current HEAD is following.

clients = 2, avg = 284.9976
clients = 5, avg = 356.1726
clients = 10, avg = 375.9856
clients = 50, avg = 429.5745

In case2, the through-put got decreased compare to case 1 but it seems
to be almost same as current HEAD. Because the speed of acquiring and
releasing extension lock got x10 faster than current HEAD as I
mentioned before, the performance degradation may not have gotten
decreased than I expected even in case 2.
Since my machine doesn't have enough resources the result of clients =
50 might not be a valid result.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

17 December 2017, 09:27:24

On Thu, Dec 14, 2017 at 5:45 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> Here is the result.
> I've measured the through-put with some cases on my virtual machine.
> Each client loads 48k file to each different relations located on
> either xfs filesystem or ext4 filesystem, for 30 sec.
>
> Case 1: COPYs to relations on different filessystems(xfs and ext4) and
> N_RELEXTLOCK_ENTS is 1024
>
> clients = 2, avg = 296.2068
> clients = 5, avg = 372.0707
> clients = 10, avg = 389.8850
> clients = 50, avg = 428.8050
>
> Case 2: COPYs to relations on different filessystems(xfs and ext4) and
> N_RELEXTLOCK_ENTS is 1
>
> clients = 2, avg = 294.3633
> clients = 5, avg = 358.9364
> clients = 10, avg = 383.6945
> clients = 50, avg = 424.3687
>
> And the result of current HEAD is following.
>
> clients = 2, avg = 284.9976
> clients = 5, avg = 356.1726
> clients = 10, avg = 375.9856
> clients = 50, avg = 429.5745
>
> In case2, the through-put got decreased compare to case 1 but it seems
> to be almost same as current HEAD. Because the speed of acquiring and
> releasing extension lock got x10 faster than current HEAD as I
> mentioned before, the performance degradation may not have gotten
> decreased than I expected even in case 2.
> Since my machine doesn't have enough resources the result of clients =
> 50 might not be a valid result.

I have to admit that result is surprising to me.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

18 December 2017, 11:04:08

On Sun, Dec 17, 2017 at 12:27 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Dec 14, 2017 at 5:45 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> Here is the result.
>> I've measured the through-put with some cases on my virtual machine.
>> Each client loads 48k file to each different relations located on
>> either xfs filesystem or ext4 filesystem, for 30 sec.
>>
>> Case 1: COPYs to relations on different filessystems(xfs and ext4) and
>> N_RELEXTLOCK_ENTS is 1024
>>
>> clients = 2, avg = 296.2068
>> clients = 5, avg = 372.0707
>> clients = 10, avg = 389.8850
>> clients = 50, avg = 428.8050
>>
>> Case 2: COPYs to relations on different filessystems(xfs and ext4) and
>> N_RELEXTLOCK_ENTS is 1
>>
>> clients = 2, avg = 294.3633
>> clients = 5, avg = 358.9364
>> clients = 10, avg = 383.6945
>> clients = 50, avg = 424.3687
>>
>> And the result of current HEAD is following.
>>
>> clients = 2, avg = 284.9976
>> clients = 5, avg = 356.1726
>> clients = 10, avg = 375.9856
>> clients = 50, avg = 429.5745
>>
>> In case2, the through-put got decreased compare to case 1 but it seems
>> to be almost same as current HEAD. Because the speed of acquiring and
>> releasing extension lock got x10 faster than current HEAD as I
>> mentioned before, the performance degradation may not have gotten
>> decreased than I expected even in case 2.
>> Since my machine doesn't have enough resources the result of clients =
>> 50 might not be a valid result.
>
> I have to admit that result is surprising to me.
>

I think the environment I used for performance measurement did not
have enough resources. I will do the same benchmark on an another
environment to see if it was a valid result, and will share it.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

19 December 2017, 06:22:28

On Mon, Dec 18, 2017 at 2:04 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> On Sun, Dec 17, 2017 at 12:27 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Thu, Dec 14, 2017 at 5:45 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> Here is the result.
>>> I've measured the through-put with some cases on my virtual machine.
>>> Each client loads 48k file to each different relations located on
>>> either xfs filesystem or ext4 filesystem, for 30 sec.
>>>
>>> Case 1: COPYs to relations on different filessystems(xfs and ext4) and
>>> N_RELEXTLOCK_ENTS is 1024
>>>
>>> clients = 2, avg = 296.2068
>>> clients = 5, avg = 372.0707
>>> clients = 10, avg = 389.8850
>>> clients = 50, avg = 428.8050
>>>
>>> Case 2: COPYs to relations on different filessystems(xfs and ext4) and
>>> N_RELEXTLOCK_ENTS is 1
>>>
>>> clients = 2, avg = 294.3633
>>> clients = 5, avg = 358.9364
>>> clients = 10, avg = 383.6945
>>> clients = 50, avg = 424.3687
>>>
>>> And the result of current HEAD is following.
>>>
>>> clients = 2, avg = 284.9976
>>> clients = 5, avg = 356.1726
>>> clients = 10, avg = 375.9856
>>> clients = 50, avg = 429.5745
>>>
>>> In case2, the through-put got decreased compare to case 1 but it seems
>>> to be almost same as current HEAD. Because the speed of acquiring and
>>> releasing extension lock got x10 faster than current HEAD as I
>>> mentioned before, the performance degradation may not have gotten
>>> decreased than I expected even in case 2.
>>> Since my machine doesn't have enough resources the result of clients =
>>> 50 might not be a valid result.
>>
>> I have to admit that result is surprising to me.
>>
>
> I think the environment I used for performance measurement did not
> have enough resources. I will do the same benchmark on an another
> environment to see if it was a valid result, and will share it.
>

I did performance measurement on an different environment where has 4
cores and physically separated two disk volumes. Also I've change the
benchmarking so that COPYs load only 300 integer tuples which are not
fit within single page, and changed tables to unlogged tables to
observe the overhead of locking/unlocking relext locks.

Case 1: COPYs to relations on different filessystems(xfs and ext4) and
N_RELEXTLOCK_ENTS is 1024

clients = 1, avg = 3033.8933
clients = 2, avg = 5992.9077
clients = 4, avg = 8055.9515
clients = 8, avg = 8468.9306
clients = 16, avg = 7718.6879

Case 2: COPYs to relations on different filessystems(xfs and ext4) and
N_RELEXTLOCK_ENTS is 1

clients = 1, avg = 3012.4993
clients = 2, avg = 5854.9966
clients = 4, avg = 7380.6082
clients = 8, avg = 7091.8367
clients = 16, avg = 7573.2904

And the result of current HEAD is following.

clients = 1, avg = 2962.2416
clients = 2, avg = 5856.9774
clients = 4, avg = 7561.1376
clients = 8, avg = 7252.0192
clients = 16, avg = 7916.7651

As per the above results, compared with current HEAD the through-put
of case 1 got increased up to 17%. On the other hand, the through-put
of case 2 got decreased 2%~5%.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Mithun Cy

Date:

02 January 2018, 12:09:16

On Tue, Dec 19, 2017 at 5:52 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> On Mon, Dec 18, 2017 at 2:04 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> On Sun, Dec 17, 2017 at 12:27 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>>>
>>> I have to admit that result is surprising to me.
>>
>> I think the environment I used for performance measurement did not
>> have enough resources. I will do the same benchmark on an another
>> environment to see if it was a valid result, and will share it.
>>
> I did performance measurement on an different environment where has 4
> cores and physically separated two disk volumes. Also I've change the
> benchmarking so that COPYs load only 300 integer tuples which are not
> fit within single page, and changed tables to unlogged tables to
> observe the overhead of locking/unlocking relext locks.

I ran same test as asked by Robert it was just an extension of tests
[1] pointed by Amit Kapila,

Machine : cthulhu
------------------------
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                128
On-line CPU(s) list:   0-127
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             8
NUMA node(s):          8
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 47
Model name:            Intel(R) Xeon(R) CPU E7- 8830  @ 2.13GHz
Stepping:              2
CPU MHz:               1064.000
CPU max MHz:           2129.0000
CPU min MHz:           1064.0000
BogoMIPS:              4266.59
Virtualization:        VT-x
Hypervisor vendor:     vertical
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              24576K
NUMA node0 CPU(s):     0-7,64-71
NUMA node1 CPU(s):     8-15,72-79
NUMA node2 CPU(s):     16-23,80-87
NUMA node3 CPU(s):     24-31,88-95
NUMA node4 CPU(s):     32-39,96-103
NUMA node5 CPU(s):     40-47,104-111
NUMA node6 CPU(s):     48-55,112-119
NUMA node7 CPU(s):     56-63,120-127

It has 2 discs with different filesytem as below
/dev/mapper/vg_mag-data2        ext4      5.1T  3.6T  1.2T  76% /mnt/data-mag2
/dev/mapper/vg_mag-data1        xfs       5.1T  1.6T  3.6T  31% /mnt/data-mag

I have created 2 tables each one on above filesystem.

test_size_copy.sh --> automated script to run copy test.
copy_script1, copy_script2 -> copy pg_bench script's used by
test_size_copy.sh to load to 2 different tables.

To run above copy_scripts in parallel I have run it with equal weights as below.
./pgbench -c $threads -j $threads -f copy_script1@1 -f copy_script2@1
-T 120 postgres >> test_results.txt


Results :
-----------

Clients        HEAD-TPS
---------        ---------------
1                84.460734
2                121.359035
4                175.886335
8                268.764828
16              369.996667
32              439.032756
64              482.185392


Clients    N_RELEXTLOCK_ENTS = 1024    %diff with DEAD
----------------------------------------------------------------------------------
1        87.165777        3.20272258112273
2        131.094037        8.02165409439848
4        181.667104        3.2866504381935
8        267.412856        -0.503031594595423
16        376.118671        1.65461058058666
32        460.756357        4.94805927419228
64        492.723975        2.18558736428913

Not much of an improvement from HEAD

Clients    N_RELEXTLOCK_ENTS = 1    %diff with HEAD
-----------------------------------------------------------------------------
1        86.288574        2.16412990206786
2        131.398667        8.27266960387414
4        168.681079        -4.09654109854526
8        245.841999        -8.52895416806549
16        321.972147        -12.9797169226933
32        375.783299        -14.4065462395703
64        360.134531        -25.3120196142317


So in case of  N_RELEXTLOCK_ENTS = 1 we can see regression as high 25%. ?


[1]https://www.postgresql.org/message-id/CAFiTN-tkX6gs-jL8VrPxg6OG9VUAKnObUq7r7pWQqASzdF5OwA%40mail.gmail.com
-- 
Thanks and Regards
Mithun C Y
EnterpriseDB: http://www.enterprisedb.com

On Thu, Mar 01, 2018 at 04:01:28PM -0500, Robert Haas wrote:
> If you have a clever idea how to make this work with as few atomic
> operations as the current patch uses while at the same time reducing
> the possibility of contention, I'm all ears.  But I don't see how to
> do that.

This thread has no activity since the beginning of the commit fest, and
it seems that it would be hard to reach something committable for v11,
so I am marking it as returned with feedback.
--
Michael

Attachment

signature.asc

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

10 April 2018, 15:40:54

On Fri, Mar 30, 2018 at 4:43 PM, Michael Paquier <michael@paquier.xyz> wrote:
> On Thu, Mar 01, 2018 at 04:01:28PM -0500, Robert Haas wrote:
>> If you have a clever idea how to make this work with as few atomic
>> operations as the current patch uses while at the same time reducing
>> the possibility of contention, I'm all ears.  But I don't see how to
>> do that.
>
> This thread has no activity since the beginning of the commit fest, and
> it seems that it would be hard to reach something committable for v11,
> so I am marking it as returned with feedback.

Thank you.

The probability of performance degradation can be reduced by
increasing N_RELEXTLOCK_ENTS. But as Robert mentioned, while keeping
fast and simple implementation like acquiring lock by a few atomic
operation it's hard to improve or at least keep the current
performance on all cases. I was thinking that this patch is necessary
by parallel DML operations and vacuum but if the community cannot
accept this approach it might be better to mark it as "Rejected" and
then I should reconsider the design of parallel vacuum.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

10 April 2018, 22:40:12

On Tue, Apr 10, 2018 at 5:40 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> The probability of performance degradation can be reduced by
> increasing N_RELEXTLOCK_ENTS. But as Robert mentioned, while keeping
> fast and simple implementation like acquiring lock by a few atomic
> operation it's hard to improve or at least keep the current
> performance on all cases. I was thinking that this patch is necessary
> by parallel DML operations and vacuum but if the community cannot
> accept this approach it might be better to mark it as "Rejected" and
> then I should reconsider the design of parallel vacuum.

I'm sorry that I didn't get time to work further on this during the
CommitFest.  In terms of moving forward, I'd still like to hear what
Andres has to say about the comments I made on March 1st.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

11 April 2018, 07:08:04

On Wed, Apr 11, 2018 at 1:40 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, Apr 10, 2018 at 5:40 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> The probability of performance degradation can be reduced by
>> increasing N_RELEXTLOCK_ENTS. But as Robert mentioned, while keeping
>> fast and simple implementation like acquiring lock by a few atomic
>> operation it's hard to improve or at least keep the current
>> performance on all cases. I was thinking that this patch is necessary
>> by parallel DML operations and vacuum but if the community cannot
>> accept this approach it might be better to mark it as "Rejected" and
>> then I should reconsider the design of parallel vacuum.
>
> I'm sorry that I didn't get time to work further on this during the
> CommitFest.

Never mind. There was a lot of items especially at the last CommitFest.

> In terms of moving forward, I'd still like to hear what
> Andres has to say about the comments I made on March 1st.

Yeah, agreed.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

26 April 2018, 00:30:50

On Tue, Apr 10, 2018 at 9:08 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> Never mind. There was a lot of items especially at the last CommitFest.
>
>> In terms of moving forward, I'd still like to hear what
>> Andres has to say about the comments I made on March 1st.
>
> Yeah, agreed.

$ ping -n andres.freund
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
^C
--- andres.freund ping statistics ---
6 packets transmitted, 0 packets received, 100.0% packet loss

Meanwhile, https://www.postgresql.org/message-id/4c171ffe-e3ee-acc5-9066-a40d52bc5ae9@postgrespro.ru
shows that this patch has some benefits for other cases, which is a
point in favor IMHO.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

26 April 2018, 12:10:46

On Thu, Apr 26, 2018 at 3:30 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, Apr 10, 2018 at 9:08 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> Never mind. There was a lot of items especially at the last CommitFest.
>>
>>> In terms of moving forward, I'd still like to hear what
>>> Andres has to say about the comments I made on March 1st.
>>
>> Yeah, agreed.
>
> $ ping -n andres.freund
> Request timeout for icmp_seq 0
> Request timeout for icmp_seq 1
> Request timeout for icmp_seq 2
> Request timeout for icmp_seq 3
> Request timeout for icmp_seq 4
> ^C
> --- andres.freund ping statistics ---
> 6 packets transmitted, 0 packets received, 100.0% packet loss
>
> Meanwhile, https://www.postgresql.org/message-id/4c171ffe-e3ee-acc5-9066-a40d52bc5ae9@postgrespro.ru
> shows that this patch has some benefits for other cases, which is a
> point in favor IMHO.

Thank you for sharing. That's good to know.

Andres pointed out the performance degradation due to hash collision
when multiple loading. I think the point is that it happens at where
users don't know.  Therefore even if we make N_RELEXTLOCK_ENTS
configurable parameter, since users don't know the hash collision they
don't know when they should tune it.

So it's just an idea but how about adding an SQL-callable function
that returns the estimated number of lock waiters of the given
relation? Since user knows how many processes are loading to the
relation, if a returned value by the function is greater than the
expected value user  can know hash collision and will be able to start
to consider to increase N_RELEXTLOCK_ENTS.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

27 April 2018, 01:08:24

On Thu, Apr 26, 2018 at 2:10 AM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> Thank you for sharing. That's good to know.
>
> Andres pointed out the performance degradation due to hash collision
> when multiple loading. I think the point is that it happens at where
> users don't know.  Therefore even if we make N_RELEXTLOCK_ENTS
> configurable parameter, since users don't know the hash collision they
> don't know when they should tune it.
>
> So it's just an idea but how about adding an SQL-callable function
> that returns the estimated number of lock waiters of the given
> relation? Since user knows how many processes are loading to the
> relation, if a returned value by the function is greater than the
> expected value user  can know hash collision and will be able to start
> to consider to increase N_RELEXTLOCK_ENTS.

I don't think that's a very useful suggestion.  Changing
N_RELEXTLOCK_ENTS requires a recompile, which is going to be
impractical for most users.  Even if we made it a GUC, we don't want
users to have to tune stuff like this.  If we actually think this is
going to be a problem, we'd probably better rethink the desgin.

I think the real question is whether the scenario is common enough to
worry about.  In practice, you'd have to be extremely unlucky to be
doing many bulk loads at the same time that all happened to hash to
the same bucket.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

27 April 2018, 01:10:03

Hi,

On 2018-04-26 15:08:24 -0400, Robert Haas wrote:
> I don't think that's a very useful suggestion.  Changing
> N_RELEXTLOCK_ENTS requires a recompile, which is going to be
> impractical for most users.  Even if we made it a GUC, we don't want
> users to have to tune stuff like this.  If we actually think this is
> going to be a problem, we'd probably better rethink the desgin.

Agreed.


> I think the real question is whether the scenario is common enough to
> worry about.  In practice, you'd have to be extremely unlucky to be
> doing many bulk loads at the same time that all happened to hash to
> the same bucket.

With a bunch of parallel bulkloads into partitioned tables that really
doesn't seem that unlikely?

Greetings,

Andres Freund

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

27 April 2018, 01:25:22

On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund <andres@anarazel.de> wrote:
>> I think the real question is whether the scenario is common enough to
>> worry about.  In practice, you'd have to be extremely unlucky to be
>> doing many bulk loads at the same time that all happened to hash to
>> the same bucket.
>
> With a bunch of parallel bulkloads into partitioned tables that really
> doesn't seem that unlikely?

It increases the likelihood of collisions, but probably decreases the
number of cases where the contention gets really bad.

For example, suppose each table has 100 partitions and you are
bulk-loading 10 of them at a time.  It's virtually certain that you
will have some collisions, but the amount of contention within each
bucket will remain fairly low because each backend spends only 1% of
its time in the bucket corresponding to any given partition.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

RE: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

"Alex Ignatov"

Date:

21 May 2018, 20:59:51



-----Original Message-----
From: Robert Haas <robertmhaas@gmail.com>
Sent: Thursday, April 26, 2018 10:25 PM
To: Andres Freund <andres@anarazel.de>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Michael Paquier <michael@paquier.xyz>; Mithun Cy
<mithun.cy@enterprisedb.com>;Tom Lane <tgl@sss.pgh.pa.us>; Thomas Munro <thomas.munro@enterprisedb.com>; Amit Kapila
<amit.kapila16@gmail.com>;PostgreSQL-development <pgsql-hackers@postgresql.org> 
Subject: Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund <andres@anarazel.de> wrote:
>> I think the real question is whether the scenario is common enough to
>> worry about.  In practice, you'd have to be extremely unlucky to be
>> doing many bulk loads at the same time that all happened to hash to
>> the same bucket.
>
> With a bunch of parallel bulkloads into partitioned tables that really
> doesn't seem that unlikely?

It increases the likelihood of collisions, but probably decreases the number of cases where the contention gets really
bad.

For example, suppose each table has 100 partitions and you are bulk-loading 10 of them at a time.  It's virtually
certainthat you will have some collisions, but the amount of contention within each bucket will remain fairly low
becauseeach backend spends only 1% of its time in the bucket corresponding to any given partition. 

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Hello!
I want to try to test this patch on 302(704 ht) core machine.

Patching on master (commit 81256cd05f0745353c6572362155b57250a0d2a0) is ok but
got some error while compiling :

gistvacuum.c: In function ‘gistvacuumcleanup’:
gistvacuum.c:92:3: error: too many arguments to function ‘LockRelationForExtension’
   LockRelationForExtension(rel, ExclusiveLock);
   ^
In file included from gistvacuum.c:21:0:
../../../../src/include/storage/extension_lock.h:30:13: note: declared here
 extern void LockRelationForExtension(Relation relation);
             ^
gistvacuum.c:95:3: error: too many arguments to function ‘UnlockRelationForExtension’
   UnlockRelationForExtension(rel, ExclusiveLock);
   ^
In file included from gistvacuum.c:21:0:
../../../../src/include/storage/extension_lock.h:31:13: note: declared here
 extern void UnlockRelationForExtension(Relation relation);


--
Alex Ignatov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

RE: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

"Alex Ignatov"

Date:

21 May 2018, 21:05:44


--
Alex Ignatov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

-----Original Message-----
From: Alex Ignatov <a.ignatov@postgrespro.ru>
Sent: Monday, May 21, 2018 6:00 PM
To: 'Robert Haas' <robertmhaas@gmail.com>; 'Andres Freund' <andres@anarazel.de>
Cc: 'Masahiko Sawada' <sawada.mshk@gmail.com>; 'Michael Paquier' <michael@paquier.xyz>; 'Mithun Cy'
<mithun.cy@enterprisedb.com>;'Tom Lane' <tgl@sss.pgh.pa.us>; 'Thomas Munro' <thomas.munro@enterprisedb.com>; 'Amit
Kapila'<amit.kapila16@gmail.com>; 'PostgreSQL-development' <pgsql-hackers@postgresql.org> 
Subject: RE: [HACKERS] Moving relation extension locks out of heavyweight lock manager




-----Original Message-----
From: Robert Haas <robertmhaas@gmail.com>
Sent: Thursday, April 26, 2018 10:25 PM
To: Andres Freund <andres@anarazel.de>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Michael Paquier <michael@paquier.xyz>; Mithun Cy
<mithun.cy@enterprisedb.com>;Tom Lane <tgl@sss.pgh.pa.us>; Thomas Munro <thomas.munro@enterprisedb.com>; Amit Kapila
<amit.kapila16@gmail.com>;PostgreSQL-development <pgsql-hackers@postgresql.org> 
Subject: Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund <andres@anarazel.de> wrote:
>> I think the real question is whether the scenario is common enough to
>> worry about.  In practice, you'd have to be extremely unlucky to be
>> doing many bulk loads at the same time that all happened to hash to
>> the same bucket.
>
> With a bunch of parallel bulkloads into partitioned tables that really
> doesn't seem that unlikely?

It increases the likelihood of collisions, but probably decreases the number of cases where the contention gets really
bad.

For example, suppose each table has 100 partitions and you are bulk-loading 10 of them at a time.  It's virtually
certainthat you will have some collisions, but the amount of contention within each bucket will remain fairly low
becauseeach backend spends only 1% of its time in the bucket corresponding to any given partition. 

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Hello!
I want to try to test this patch on 302(704 ht) core machine.

Patching on master (commit 81256cd05f0745353c6572362155b57250a0d2a0) is ok but got some error while compiling :

gistvacuum.c: In function ‘gistvacuumcleanup’:
gistvacuum.c:92:3: error: too many arguments to function ‘LockRelationForExtension’
   LockRelationForExtension(rel, ExclusiveLock);
   ^
In file included from gistvacuum.c:21:0:
../../../../src/include/storage/extension_lock.h:30:13: note: declared here  extern void
LockRelationForExtension(Relationrelation); 
             ^
gistvacuum.c:95:3: error: too many arguments to function ‘UnlockRelationForExtension’
   UnlockRelationForExtension(rel, ExclusiveLock);
   ^
In file included from gistvacuum.c:21:0:
../../../../src/include/storage/extension_lock.h:31:13: note: declared here  extern void
UnlockRelationForExtension(Relationrelation); 



Sorry, forgot to mention that patch version is extension-lock-v12.patch

--
Alex Ignatov
Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

22 May 2018, 07:30:44

On Tue, May 22, 2018 at 12:05 AM, Alex Ignatov <a.ignatov@postgrespro.ru> wrote:
>
>
> --
> Alex Ignatov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>
> -----Original Message-----
> From: Alex Ignatov <a.ignatov@postgrespro.ru>
> Sent: Monday, May 21, 2018 6:00 PM
> To: 'Robert Haas' <robertmhaas@gmail.com>; 'Andres Freund' <andres@anarazel.de>
> Cc: 'Masahiko Sawada' <sawada.mshk@gmail.com>; 'Michael Paquier' <michael@paquier.xyz>; 'Mithun Cy'
<mithun.cy@enterprisedb.com>;'Tom Lane' <tgl@sss.pgh.pa.us>; 'Thomas Munro' <thomas.munro@enterprisedb.com>; 'Amit
Kapila'<amit.kapila16@gmail.com>; 'PostgreSQL-development' <pgsql-hackers@postgresql.org> 
> Subject: RE: [HACKERS] Moving relation extension locks out of heavyweight lock manager
>
>
>
>
> -----Original Message-----
> From: Robert Haas <robertmhaas@gmail.com>
> Sent: Thursday, April 26, 2018 10:25 PM
> To: Andres Freund <andres@anarazel.de>
> Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Michael Paquier <michael@paquier.xyz>; Mithun Cy
<mithun.cy@enterprisedb.com>;Tom Lane <tgl@sss.pgh.pa.us>; Thomas Munro <thomas.munro@enterprisedb.com>; Amit Kapila
<amit.kapila16@gmail.com>;PostgreSQL-development <pgsql-hackers@postgresql.org> 
> Subject: Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager
>
> On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund <andres@anarazel.de> wrote:
>>> I think the real question is whether the scenario is common enough to
>>> worry about.  In practice, you'd have to be extremely unlucky to be
>>> doing many bulk loads at the same time that all happened to hash to
>>> the same bucket.
>>
>> With a bunch of parallel bulkloads into partitioned tables that really
>> doesn't seem that unlikely?
>
> It increases the likelihood of collisions, but probably decreases the number of cases where the contention gets
reallybad. 
>
> For example, suppose each table has 100 partitions and you are bulk-loading 10 of them at a time.  It's virtually
certainthat you will have some collisions, but the amount of contention within each bucket will remain fairly low
becauseeach backend spends only 1% of its time in the bucket corresponding to any given partition. 
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
>
> Hello!
> I want to try to test this patch on 302(704 ht) core machine.
>
> Patching on master (commit 81256cd05f0745353c6572362155b57250a0d2a0) is ok but got some error while compiling :

Thank you for reporting.
Attached an rebased patch with current HEAD.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachment

extension-lock-v13.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lockmanager

From

Konstantin Knizhnik

Date:

04 June 2018, 19:47:29


On 26.04.2018 09:10, Masahiko Sawada wrote:
> On Thu, Apr 26, 2018 at 3:30 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Tue, Apr 10, 2018 at 9:08 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> Never mind. There was a lot of items especially at the last CommitFest.
>>>
>>>> In terms of moving forward, I'd still like to hear what
>>>> Andres has to say about the comments I made on March 1st.
>>> Yeah, agreed.
>> $ ping -n andres.freund
>> Request timeout for icmp_seq 0
>> Request timeout for icmp_seq 1
>> Request timeout for icmp_seq 2
>> Request timeout for icmp_seq 3
>> Request timeout for icmp_seq 4
>> ^C
>> --- andres.freund ping statistics ---
>> 6 packets transmitted, 0 packets received, 100.0% packet loss
>>
>> Meanwhile, https://www.postgresql.org/message-id/4c171ffe-e3ee-acc5-9066-a40d52bc5ae9@postgrespro.ru
>> shows that this patch has some benefits for other cases, which is a
>> point in favor IMHO.
> Thank you for sharing. That's good to know.
>
> Andres pointed out the performance degradation due to hash collision
> when multiple loading. I think the point is that it happens at where
> users don't know.  Therefore even if we make N_RELEXTLOCK_ENTS
> configurable parameter, since users don't know the hash collision they
> don't know when they should tune it.
>
> So it's just an idea but how about adding an SQL-callable function
> that returns the estimated number of lock waiters of the given
> relation? Since user knows how many processes are loading to the
> relation, if a returned value by the function is greater than the
> expected value user  can know hash collision and will be able to start
> to consider to increase N_RELEXTLOCK_ENTS.
>
> Regards,
>
> --
> Masahiko Sawada
> NIPPON TELEGRAPH AND TELEPHONE CORPORATION
> NTT Open Source Software Center
>
We in PostgresProc were faced with lock extension contention problem at 
two more customers and tried to use this patch (v13) to address this issue.
Unfortunately replacing heavy lock with lwlock couldn't completely 
eliminate contention, now most of backends are blocked on conditional 
variable:

0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
#0  0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x00000000007024ee in WaitEventSetWait ()
#2  0x0000000000718fa6 in ConditionVariableSleep ()
#3  0x000000000071954d in RelExtLockAcquire ()
#4  0x00000000004ba99d in RelationGetBufferForTuple ()
#5  0x00000000004b3f18 in heap_insert ()
#6  0x00000000006109c8 in ExecInsert ()
#7  0x0000000000611a49 in ExecModifyTable ()
#8  0x00000000005ef97a in standard_ExecutorRun ()
#9  0x000000000072440a in ProcessQuery ()
#10 0x0000000000724631 in PortalRunMulti ()
#11 0x00000000007250ec in PortalRun ()
#12 0x0000000000721287 in exec_simple_query ()
#13 0x0000000000722532 in PostgresMain ()
#14 0x000000000047a9eb in ServerLoop ()
#15 0x00000000006b9fe9 in PostmasterMain ()
#16 0x000000000047b431 in main ()

Obviously there is nothing surprising here: if a lot of processes try to 
acquire the same exclusive lock, then high contention is expected.
I just want to notice that this patch is not able to completely 
eliminate the problem with large number of concurrent inserts to the 
same table.

Second problem we observed was even more critical: if backed is granted 
relation extension lock and then got some error before releasing this lock,
then abort of the current transaction doesn't release this lock (unlike 
heavy weight lock) and the relation is kept locked.
So database is actually stalled and server has to be restarted.

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

05 June 2018, 00:42:23

Hi,

On 2018-06-04 16:47:29 +0300, Konstantin Knizhnik wrote:
> We in PostgresProc were faced with lock extension contention problem at two
> more customers and tried to use this patch (v13) to address this issue.
> Unfortunately replacing heavy lock with lwlock couldn't completely eliminate
> contention, now most of backends are blocked on conditional variable:
> 
> 0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
> #0  0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
> #1  0x00000000007024ee in WaitEventSetWait ()
> #2  0x0000000000718fa6 in ConditionVariableSleep ()
> #3  0x000000000071954d in RelExtLockAcquire ()

That doesn't necessarily mean that the postgres code is to fault
here. It's entirely possible that the filesystem or storage is the
bottleneck.  Could you briefly describe workload & hardware?


> Second problem we observed was even more critical: if backed is granted
> relation extension lock and then got some error before releasing this lock,
> then abort of the current transaction doesn't release this lock (unlike
> heavy weight lock) and the relation is kept locked.
> So database is actually stalled and server has to be restarted.

That obvioulsy needs to be fixed...

Greetings,

Andres Freund

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

05 June 2018, 10:22:58

On Mon, Jun 4, 2018 at 10:47 PM, Konstantin Knizhnik
<k.knizhnik@postgrespro.ru> wrote:
>
>
> On 26.04.2018 09:10, Masahiko Sawada wrote:
>>
>> On Thu, Apr 26, 2018 at 3:30 AM, Robert Haas <robertmhaas@gmail.com>
>> wrote:
>>>
>>> On Tue, Apr 10, 2018 at 9:08 PM, Masahiko Sawada <sawada.mshk@gmail.com>
>>> wrote:
>>>>
>>>> Never mind. There was a lot of items especially at the last CommitFest.
>>>>
>>>>> In terms of moving forward, I'd still like to hear what
>>>>> Andres has to say about the comments I made on March 1st.
>>>>
>>>> Yeah, agreed.
>>>
>>> $ ping -n andres.freund
>>> Request timeout for icmp_seq 0
>>> Request timeout for icmp_seq 1
>>> Request timeout for icmp_seq 2
>>> Request timeout for icmp_seq 3
>>> Request timeout for icmp_seq 4
>>> ^C
>>> --- andres.freund ping statistics ---
>>> 6 packets transmitted, 0 packets received, 100.0% packet loss
>>>
>>> Meanwhile,
>>> https://www.postgresql.org/message-id/4c171ffe-e3ee-acc5-9066-a40d52bc5ae9@postgrespro.ru
>>> shows that this patch has some benefits for other cases, which is a
>>> point in favor IMHO.
>>
>> Thank you for sharing. That's good to know.
>>
>> Andres pointed out the performance degradation due to hash collision
>> when multiple loading. I think the point is that it happens at where
>> users don't know.  Therefore even if we make N_RELEXTLOCK_ENTS
>> configurable parameter, since users don't know the hash collision they
>> don't know when they should tune it.
>>
>> So it's just an idea but how about adding an SQL-callable function
>> that returns the estimated number of lock waiters of the given
>> relation? Since user knows how many processes are loading to the
>> relation, if a returned value by the function is greater than the
>> expected value user  can know hash collision and will be able to start
>> to consider to increase N_RELEXTLOCK_ENTS.
>>
>> Regards,
>>
>> --
>> Masahiko Sawada
>> NIPPON TELEGRAPH AND TELEPHONE CORPORATION
>> NTT Open Source Software Center
>>
> We in PostgresProc were faced with lock extension contention problem at two
> more customers and tried to use this patch (v13) to address this issue.
> Unfortunately replacing heavy lock with lwlock couldn't completely eliminate
> contention, now most of backends are blocked on conditional variable:
>
> 0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
> #0  0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
> #1  0x00000000007024ee in WaitEventSetWait ()
> #2  0x0000000000718fa6 in ConditionVariableSleep ()
> #3  0x000000000071954d in RelExtLockAcquire ()
> #4  0x00000000004ba99d in RelationGetBufferForTuple ()
> #5  0x00000000004b3f18 in heap_insert ()
> #6  0x00000000006109c8 in ExecInsert ()
> #7  0x0000000000611a49 in ExecModifyTable ()
> #8  0x00000000005ef97a in standard_ExecutorRun ()
> #9  0x000000000072440a in ProcessQuery ()
> #10 0x0000000000724631 in PortalRunMulti ()
> #11 0x00000000007250ec in PortalRun ()
> #12 0x0000000000721287 in exec_simple_query ()
> #13 0x0000000000722532 in PostgresMain ()
> #14 0x000000000047a9eb in ServerLoop ()
> #15 0x00000000006b9fe9 in PostmasterMain ()
> #16 0x000000000047b431 in main ()
>
> Obviously there is nothing surprising here: if a lot of processes try to
> acquire the same exclusive lock, then high contention is expected.
> I just want to notice that this patch is not able to completely eliminate
> the problem with large number of concurrent inserts to the same table.
>
> Second problem we observed was even more critical: if backed is granted
> relation extension lock and then got some error before releasing this lock,
> then abort of the current transaction doesn't release this lock (unlike
> heavy weight lock) and the relation is kept locked.
> So database is actually stalled and server has to be restarted.
>

Thank you for reporting.

Regarding the second problem, I tried to reproduce that bug with
latest version patch (v13) but could not. When transaction aborts, we
call ReousrceOwnerRelease()->ResourceOwnerReleaseInternal()->ProcReleaseLocks()->RelExtLockCleanup()
and clear either relext lock bits we are holding or waiting. If we
raise an error after we added a relext lock bit but before we
increment its holding count then the relext lock is remained, but I
couldn't see the code raises an error between them. Could you please
share the concrete reproduction steps of the cause of database stalled
if possible?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lockmanager

From

Konstantin Knizhnik

Date:

05 June 2018, 15:47:40

On 04.06.2018 21:42, Andres Freund wrote:

Hi,

On 2018-06-04 16:47:29 +0300, Konstantin Knizhnik wrote:

We in PostgresProc were faced with lock extension contention problem at two
more customers and tried to use this patch (v13) to address this issue.
Unfortunately replacing heavy lock with lwlock couldn't completely eliminate
contention, now most of backends are blocked on conditional variable:

0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
#0  0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x00000000007024ee in WaitEventSetWait ()
#2  0x0000000000718fa6 in ConditionVariableSleep ()
#3  0x000000000071954d in RelExtLockAcquire ()

That doesn't necessarily mean that the postgres code is to fault
here. It's entirely possible that the filesystem or storage is the
bottleneck.  Could you briefly describe workload & hardware?

Workload is combination of inserts and selects.
Looks like shared locks obtained by select cause starvation of inserts, trying to get exclusive relation extension lock.
The problem is fixed by fair lwlock patch, implemented by Alexander Korotkov. This patch prevents granting of shared lock if wait queue is not empty.
May be we should use this patch or find some other way to prevent starvation of writers on relation extension locks for such workloads.

Second problem we observed was even more critical: if backed is granted
relation extension lock and then got some error before releasing this lock,
then abort of the current transaction doesn't release this lock (unlike
heavy weight lock) and the relation is kept locked.
So database is actually stalled and server has to be restarted.

That obvioulsy needs to be fixed...

Sorry, looks like the problem is more obscure than I expected.
What we have observed is that all backends are blocked in lwlock (sorry stack trace is not complete):

#0  0x00007ff5a9c566d6 in futex_abstimed_wait_cancelable (private=128, abstime=0x0, expected=0, futex_word=0x7ff3c57b9b38) at ../sysdeps/unix/sysv/lin
ux/futex-internal.h:205                                                                                                                              
#1  do_futex_wait (sem=sem@entry=0x7ff3c57b9b38, abstime=0x0) at sem_waitcommon.c:111              
#2  0x00007ff5a9c567c8 in __new_sem_wait_slow (sem=sem@entry=0x7ff3c57b9b38, abstime=0x0) at sem_waitcommon.c:181                                     #3  0x00007ff5a9c56839 in __new_sem_wait (sem=sem@entry=0x7ff3c57b9b38) at sem_wait.c:42                                                              #4  0x000056290c901582 in PGSemaphoreLock (sema=0x7ff3c57b9b38) at pg_sema.c:310
#5  0x000056290c97923c in LWLockAcquire (lock=0x7ff3c7038c64, mode=LW_SHARED) at ./build/../src/backend/storage/lmgr/lwlock.c:1233


I happen after error in disk write operation. Unfortunately we do not have core files and not able to reproduce the problem.
All LW locks should be cleared by LWLockReleaseAll but ... for some reasons it doesn't happen.
We will continue investigation and try to reproduce the problem.
I will let you know if we find the reason of the problem.

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lockmanager

From

Konstantin Knizhnik

Date:

05 June 2018, 15:47:47


On 05.06.2018 07:22, Masahiko Sawada wrote:
> On Mon, Jun 4, 2018 at 10:47 PM, Konstantin Knizhnik
> <k.knizhnik@postgrespro.ru> wrote:
>>
>> On 26.04.2018 09:10, Masahiko Sawada wrote:
>>> On Thu, Apr 26, 2018 at 3:30 AM, Robert Haas <robertmhaas@gmail.com>
>>> wrote:
>>>> On Tue, Apr 10, 2018 at 9:08 PM, Masahiko Sawada <sawada.mshk@gmail.com>
>>>> wrote:
>>>>> Never mind. There was a lot of items especially at the last CommitFest.
>>>>>
>>>>>> In terms of moving forward, I'd still like to hear what
>>>>>> Andres has to say about the comments I made on March 1st.
>>>>> Yeah, agreed.
>>>> $ ping -n andres.freund
>>>> Request timeout for icmp_seq 0
>>>> Request timeout for icmp_seq 1
>>>> Request timeout for icmp_seq 2
>>>> Request timeout for icmp_seq 3
>>>> Request timeout for icmp_seq 4
>>>> ^C
>>>> --- andres.freund ping statistics ---
>>>> 6 packets transmitted, 0 packets received, 100.0% packet loss
>>>>
>>>> Meanwhile,
>>>> https://www.postgresql.org/message-id/4c171ffe-e3ee-acc5-9066-a40d52bc5ae9@postgrespro.ru
>>>> shows that this patch has some benefits for other cases, which is a
>>>> point in favor IMHO.
>>> Thank you for sharing. That's good to know.
>>>
>>> Andres pointed out the performance degradation due to hash collision
>>> when multiple loading. I think the point is that it happens at where
>>> users don't know.  Therefore even if we make N_RELEXTLOCK_ENTS
>>> configurable parameter, since users don't know the hash collision they
>>> don't know when they should tune it.
>>>
>>> So it's just an idea but how about adding an SQL-callable function
>>> that returns the estimated number of lock waiters of the given
>>> relation? Since user knows how many processes are loading to the
>>> relation, if a returned value by the function is greater than the
>>> expected value user  can know hash collision and will be able to start
>>> to consider to increase N_RELEXTLOCK_ENTS.
>>>
>>> Regards,
>>>
>>> --
>>> Masahiko Sawada
>>> NIPPON TELEGRAPH AND TELEPHONE CORPORATION
>>> NTT Open Source Software Center
>>>
>> We in PostgresProc were faced with lock extension contention problem at two
>> more customers and tried to use this patch (v13) to address this issue.
>> Unfortunately replacing heavy lock with lwlock couldn't completely eliminate
>> contention, now most of backends are blocked on conditional variable:
>>
>> 0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
>> #0  0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
>> #1  0x00000000007024ee in WaitEventSetWait ()
>> #2  0x0000000000718fa6 in ConditionVariableSleep ()
>> #3  0x000000000071954d in RelExtLockAcquire ()
>> #4  0x00000000004ba99d in RelationGetBufferForTuple ()
>> #5  0x00000000004b3f18 in heap_insert ()
>> #6  0x00000000006109c8 in ExecInsert ()
>> #7  0x0000000000611a49 in ExecModifyTable ()
>> #8  0x00000000005ef97a in standard_ExecutorRun ()
>> #9  0x000000000072440a in ProcessQuery ()
>> #10 0x0000000000724631 in PortalRunMulti ()
>> #11 0x00000000007250ec in PortalRun ()
>> #12 0x0000000000721287 in exec_simple_query ()
>> #13 0x0000000000722532 in PostgresMain ()
>> #14 0x000000000047a9eb in ServerLoop ()
>> #15 0x00000000006b9fe9 in PostmasterMain ()
>> #16 0x000000000047b431 in main ()
>>
>> Obviously there is nothing surprising here: if a lot of processes try to
>> acquire the same exclusive lock, then high contention is expected.
>> I just want to notice that this patch is not able to completely eliminate
>> the problem with large number of concurrent inserts to the same table.
>>
>> Second problem we observed was even more critical: if backed is granted
>> relation extension lock and then got some error before releasing this lock,
>> then abort of the current transaction doesn't release this lock (unlike
>> heavy weight lock) and the relation is kept locked.
>> So database is actually stalled and server has to be restarted.
>>
> Thank you for reporting.
>
> Regarding the second problem, I tried to reproduce that bug with
> latest version patch (v13) but could not. When transaction aborts, we
> call ReousrceOwnerRelease()->ResourceOwnerReleaseInternal()->ProcReleaseLocks()->RelExtLockCleanup()
> and clear either relext lock bits we are holding or waiting. If we
> raise an error after we added a relext lock bit but before we
> increment its holding count then the relext lock is remained, but I
> couldn't see the code raises an error between them. Could you please
> share the concrete reproduction

Sorry, my original guess that LW-locks are not released in case of 
transaction abort is not correct.
There was really situation when all backends were blocked in relation 
extension lock and looks like it happens after disk write error,
but as far as it happens at customer's site, we have no time for 
investigation and not able to reproduce this problem locally.

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Alexander Korotkov

Date:

05 June 2018, 16:09:08

On Tue, Jun 5, 2018 at 12:48 PM Konstantin Knizhnik
<k.knizhnik@postgrespro.ru> wrote:
> Workload is combination of inserts and selects.
> Looks like shared locks obtained by select cause starvation of inserts, trying to get exclusive relation extension
lock.
> The problem is fixed by fair lwlock patch, implemented by Alexander Korotkov. This patch prevents granting of shared
lockif wait queue is not empty. 
> May be we should use this patch or find some other way to prevent starvation of writers on relation extension locks
forsuch workloads. 

Fair lwlock patch really fixed starvation of exclusive lwlock waiters.
But that starvation happens not on relation extension lock – selects
don't get shared relation extension lock.  The real issue there was
not relation extension lock itself, but the time spent inside this
lock.  It appears that buffer replacement happening inside relation
extension lock is affected by starvation on exclusive buffer mapping
lwlocks and buffer content lwlocks, caused by many concurrent shared
lockers.  So, fair lwlock patch have no direct influence to relation
extension lock, which is naturally not even lwlock...

I'll post fair lwlock path in a separate thread.  It requires detailed
consideration and benchmarking, because there is a risk of regression
on specific workloads.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

05 June 2018, 16:29:25

On Tue, Jun 5, 2018 at 6:47 PM, Konstantin Knizhnik
<k.knizhnik@postgrespro.ru> wrote:
>
>
> On 05.06.2018 07:22, Masahiko Sawada wrote:
>>
>> On Mon, Jun 4, 2018 at 10:47 PM, Konstantin Knizhnik
>> <k.knizhnik@postgrespro.ru> wrote:
>>>
>>>
>>> On 26.04.2018 09:10, Masahiko Sawada wrote:
>>>>
>>>> On Thu, Apr 26, 2018 at 3:30 AM, Robert Haas <robertmhaas@gmail.com>
>>>> wrote:
>>>>>
>>>>> On Tue, Apr 10, 2018 at 9:08 PM, Masahiko Sawada
>>>>> <sawada.mshk@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Never mind. There was a lot of items especially at the last
>>>>>> CommitFest.
>>>>>>
>>>>>>> In terms of moving forward, I'd still like to hear what
>>>>>>> Andres has to say about the comments I made on March 1st.
>>>>>>
>>>>>> Yeah, agreed.
>>>>>
>>>>> $ ping -n andres.freund
>>>>> Request timeout for icmp_seq 0
>>>>> Request timeout for icmp_seq 1
>>>>> Request timeout for icmp_seq 2
>>>>> Request timeout for icmp_seq 3
>>>>> Request timeout for icmp_seq 4
>>>>> ^C
>>>>> --- andres.freund ping statistics ---
>>>>> 6 packets transmitted, 0 packets received, 100.0% packet loss
>>>>>
>>>>> Meanwhile,
>>>>>
>>>>> https://www.postgresql.org/message-id/4c171ffe-e3ee-acc5-9066-a40d52bc5ae9@postgrespro.ru
>>>>> shows that this patch has some benefits for other cases, which is a
>>>>> point in favor IMHO.
>>>>
>>>> Thank you for sharing. That's good to know.
>>>>
>>>> Andres pointed out the performance degradation due to hash collision
>>>> when multiple loading. I think the point is that it happens at where
>>>> users don't know.  Therefore even if we make N_RELEXTLOCK_ENTS
>>>> configurable parameter, since users don't know the hash collision they
>>>> don't know when they should tune it.
>>>>
>>>> So it's just an idea but how about adding an SQL-callable function
>>>> that returns the estimated number of lock waiters of the given
>>>> relation? Since user knows how many processes are loading to the
>>>> relation, if a returned value by the function is greater than the
>>>> expected value user  can know hash collision and will be able to start
>>>> to consider to increase N_RELEXTLOCK_ENTS.
>>>>
>>>> Regards,
>>>>
>>>> --
>>>> Masahiko Sawada
>>>> NIPPON TELEGRAPH AND TELEPHONE CORPORATION
>>>> NTT Open Source Software Center
>>>>
>>> We in PostgresProc were faced with lock extension contention problem at
>>> two
>>> more customers and tried to use this patch (v13) to address this issue.
>>> Unfortunately replacing heavy lock with lwlock couldn't completely
>>> eliminate
>>> contention, now most of backends are blocked on conditional variable:
>>>
>>> 0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
>>> #0  0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
>>> #1  0x00000000007024ee in WaitEventSetWait ()
>>> #2  0x0000000000718fa6 in ConditionVariableSleep ()
>>> #3  0x000000000071954d in RelExtLockAcquire ()
>>> #4  0x00000000004ba99d in RelationGetBufferForTuple ()
>>> #5  0x00000000004b3f18 in heap_insert ()
>>> #6  0x00000000006109c8 in ExecInsert ()
>>> #7  0x0000000000611a49 in ExecModifyTable ()
>>> #8  0x00000000005ef97a in standard_ExecutorRun ()
>>> #9  0x000000000072440a in ProcessQuery ()
>>> #10 0x0000000000724631 in PortalRunMulti ()
>>> #11 0x00000000007250ec in PortalRun ()
>>> #12 0x0000000000721287 in exec_simple_query ()
>>> #13 0x0000000000722532 in PostgresMain ()
>>> #14 0x000000000047a9eb in ServerLoop ()
>>> #15 0x00000000006b9fe9 in PostmasterMain ()
>>> #16 0x000000000047b431 in main ()
>>>
>>> Obviously there is nothing surprising here: if a lot of processes try to
>>> acquire the same exclusive lock, then high contention is expected.
>>> I just want to notice that this patch is not able to completely eliminate
>>> the problem with large number of concurrent inserts to the same table.
>>>
>>> Second problem we observed was even more critical: if backed is granted
>>> relation extension lock and then got some error before releasing this
>>> lock,
>>> then abort of the current transaction doesn't release this lock (unlike
>>> heavy weight lock) and the relation is kept locked.
>>> So database is actually stalled and server has to be restarted.
>>>
>> Thank you for reporting.
>>
>> Regarding the second problem, I tried to reproduce that bug with
>> latest version patch (v13) but could not. When transaction aborts, we
>> call
>> ReousrceOwnerRelease()->ResourceOwnerReleaseInternal()->ProcReleaseLocks()->RelExtLockCleanup()
>> and clear either relext lock bits we are holding or waiting. If we
>> raise an error after we added a relext lock bit but before we
>> increment its holding count then the relext lock is remained, but I
>> couldn't see the code raises an error between them. Could you please
>> share the concrete reproduction
>
>
> Sorry, my original guess that LW-locks are not released in case of
> transaction abort is not correct.
> There was really situation when all backends were blocked in relation
> extension lock and looks like it happens after disk write error,

You're saying that it is not correct that LWlock are not released but
it's correct that all backends were blocked in relext lock, but in
other your mail you're saying something opposite. Which is correct?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lockmanager

From

Konstantin Knizhnik

Date:

05 June 2018, 17:50:49


On 05.06.2018 13:29, Masahiko Sawada wrote:
> On Tue, Jun 5, 2018 at 6:47 PM, Konstantin Knizhnik
> <k.knizhnik@postgrespro.ru> wrote:
>>
>> On 05.06.2018 07:22, Masahiko Sawada wrote:
>>> On Mon, Jun 4, 2018 at 10:47 PM, Konstantin Knizhnik
>>> <k.knizhnik@postgrespro.ru> wrote:
>>>>
>>>> On 26.04.2018 09:10, Masahiko Sawada wrote:
>>>>> On Thu, Apr 26, 2018 at 3:30 AM, Robert Haas <robertmhaas@gmail.com>
>>>>> wrote:
>>>>>> On Tue, Apr 10, 2018 at 9:08 PM, Masahiko Sawada
>>>>>> <sawada.mshk@gmail.com>
>>>>>> wrote:
>>>>>>> Never mind. There was a lot of items especially at the last
>>>>>>> CommitFest.
>>>>>>>
>>>>>>>> In terms of moving forward, I'd still like to hear what
>>>>>>>> Andres has to say about the comments I made on March 1st.
>>>>>>> Yeah, agreed.
>>>>>> $ ping -n andres.freund
>>>>>> Request timeout for icmp_seq 0
>>>>>> Request timeout for icmp_seq 1
>>>>>> Request timeout for icmp_seq 2
>>>>>> Request timeout for icmp_seq 3
>>>>>> Request timeout for icmp_seq 4
>>>>>> ^C
>>>>>> --- andres.freund ping statistics ---
>>>>>> 6 packets transmitted, 0 packets received, 100.0% packet loss
>>>>>>
>>>>>> Meanwhile,
>>>>>>
>>>>>> https://www.postgresql.org/message-id/4c171ffe-e3ee-acc5-9066-a40d52bc5ae9@postgrespro.ru
>>>>>> shows that this patch has some benefits for other cases, which is a
>>>>>> point in favor IMHO.
>>>>> Thank you for sharing. That's good to know.
>>>>>
>>>>> Andres pointed out the performance degradation due to hash collision
>>>>> when multiple loading. I think the point is that it happens at where
>>>>> users don't know.  Therefore even if we make N_RELEXTLOCK_ENTS
>>>>> configurable parameter, since users don't know the hash collision they
>>>>> don't know when they should tune it.
>>>>>
>>>>> So it's just an idea but how about adding an SQL-callable function
>>>>> that returns the estimated number of lock waiters of the given
>>>>> relation? Since user knows how many processes are loading to the
>>>>> relation, if a returned value by the function is greater than the
>>>>> expected value user  can know hash collision and will be able to start
>>>>> to consider to increase N_RELEXTLOCK_ENTS.
>>>>>
>>>>> Regards,
>>>>>
>>>>> --
>>>>> Masahiko Sawada
>>>>> NIPPON TELEGRAPH AND TELEPHONE CORPORATION
>>>>> NTT Open Source Software Center
>>>>>
>>>> We in PostgresProc were faced with lock extension contention problem at
>>>> two
>>>> more customers and tried to use this patch (v13) to address this issue.
>>>> Unfortunately replacing heavy lock with lwlock couldn't completely
>>>> eliminate
>>>> contention, now most of backends are blocked on conditional variable:
>>>>
>>>> 0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
>>>> #0  0x00007fb03a318903 in __epoll_wait_nocancel () from /lib64/libc.so.6
>>>> #1  0x00000000007024ee in WaitEventSetWait ()
>>>> #2  0x0000000000718fa6 in ConditionVariableSleep ()
>>>> #3  0x000000000071954d in RelExtLockAcquire ()
>>>> #4  0x00000000004ba99d in RelationGetBufferForTuple ()
>>>> #5  0x00000000004b3f18 in heap_insert ()
>>>> #6  0x00000000006109c8 in ExecInsert ()
>>>> #7  0x0000000000611a49 in ExecModifyTable ()
>>>> #8  0x00000000005ef97a in standard_ExecutorRun ()
>>>> #9  0x000000000072440a in ProcessQuery ()
>>>> #10 0x0000000000724631 in PortalRunMulti ()
>>>> #11 0x00000000007250ec in PortalRun ()
>>>> #12 0x0000000000721287 in exec_simple_query ()
>>>> #13 0x0000000000722532 in PostgresMain ()
>>>> #14 0x000000000047a9eb in ServerLoop ()
>>>> #15 0x00000000006b9fe9 in PostmasterMain ()
>>>> #16 0x000000000047b431 in main ()
>>>>
>>>> Obviously there is nothing surprising here: if a lot of processes try to
>>>> acquire the same exclusive lock, then high contention is expected.
>>>> I just want to notice that this patch is not able to completely eliminate
>>>> the problem with large number of concurrent inserts to the same table.
>>>>
>>>> Second problem we observed was even more critical: if backed is granted
>>>> relation extension lock and then got some error before releasing this
>>>> lock,
>>>> then abort of the current transaction doesn't release this lock (unlike
>>>> heavy weight lock) and the relation is kept locked.
>>>> So database is actually stalled and server has to be restarted.
>>>>
>>> Thank you for reporting.
>>>
>>> Regarding the second problem, I tried to reproduce that bug with
>>> latest version patch (v13) but could not. When transaction aborts, we
>>> call
>>> ReousrceOwnerRelease()->ResourceOwnerReleaseInternal()->ProcReleaseLocks()->RelExtLockCleanup()
>>> and clear either relext lock bits we are holding or waiting. If we
>>> raise an error after we added a relext lock bit but before we
>>> increment its holding count then the relext lock is remained, but I
>>> couldn't see the code raises an error between them. Could you please
>>> share the concrete reproduction
>>
>> Sorry, my original guess that LW-locks are not released in case of
>> transaction abort is not correct.
>> There was really situation when all backends were blocked in relation
>> extension lock and looks like it happens after disk write error,
> You're saying that it is not correct that LWlock are not released but
> it's correct that all backends were blocked in relext lock, but in
> other your mail you're saying something opposite. Which is correct?
I am sorry for confusion. I have not investigated core files myself and 
just share information received from our engineer.
Looks like this problem may be related with relation extension locks at all.
Sorry for false alarm.

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

05 June 2018, 19:02:35

On 2018-06-05 13:09:08 +0300, Alexander Korotkov wrote:
> On Tue, Jun 5, 2018 at 12:48 PM Konstantin Knizhnik
> <k.knizhnik@postgrespro.ru> wrote:
> > Workload is combination of inserts and selects.
> > Looks like shared locks obtained by select cause starvation of inserts, trying to get exclusive relation extension
lock.
> > The problem is fixed by fair lwlock patch, implemented by Alexander Korotkov. This patch prevents granting of
sharedlock if wait queue is not empty.
 
> > May be we should use this patch or find some other way to prevent starvation of writers on relation extension locks
forsuch workloads.
 
> 
> Fair lwlock patch really fixed starvation of exclusive lwlock waiters.
> But that starvation happens not on relation extension lock – selects
> don't get shared relation extension lock.  The real issue there was
> not relation extension lock itself, but the time spent inside this
> lock.

Yea, that makes a lot more sense to me.


> It appears that buffer replacement happening inside relation
> extension lock is affected by starvation on exclusive buffer mapping
> lwlocks and buffer content lwlocks, caused by many concurrent shared
> lockers.  So, fair lwlock patch have no direct influence to relation
> extension lock, which is naturally not even lwlock...

Yea, that makes sense. I wonder how much the fix here is to "pre-clear"
a victim buffer, and how much is a saner buffer replacement
implementation (either by going away from O(NBuffers), or by having a
queue of clean victim buffers like my bgwriter replacement).


> I'll post fair lwlock path in a separate thread.  It requires detailed
> consideration and benchmarking, because there is a risk of regression
> on specific workloads.

I bet that doing it naively will regress massively in a number of cases.

Greetings,

Andres Freund

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Alexander Korotkov

Date:

05 June 2018, 20:05:32

On Tue, Jun 5, 2018 at 4:02 PM Andres Freund <andres@anarazel.de> wrote:
> On 2018-06-05 13:09:08 +0300, Alexander Korotkov wrote:
> > It appears that buffer replacement happening inside relation
> > extension lock is affected by starvation on exclusive buffer mapping
> > lwlocks and buffer content lwlocks, caused by many concurrent shared
> > lockers.  So, fair lwlock patch have no direct influence to relation
> > extension lock, which is naturally not even lwlock...
>
> Yea, that makes sense. I wonder how much the fix here is to "pre-clear"
> a victim buffer, and how much is a saner buffer replacement
> implementation (either by going away from O(NBuffers), or by having a
> queue of clean victim buffers like my bgwriter replacement).

The particular thing I observed on our environment is BufferAlloc()
waiting hours on buffer partition lock.  Increasing NUM_BUFFER_PARTITIONS
didn't give any significant help.  It appears that very hot page (root page of
some frequently used index) reside on that partition, so this partition was
continuously under shared lock.  So, in order to resolve without changing
LWLock, we probably should move our buffers hash table to something
lockless.

> > I'll post fair lwlock path in a separate thread.  It requires detailed
> > consideration and benchmarking, because there is a risk of regression
> > on specific workloads.
>
> I bet that doing it naively will regress massively in a number of cases.

Yes, I suspect the same.  However, I tend to think that something is wrong
with LWLock itself.  It seems that it is the only of our locks, which provides
some lockers almost infinite starvations under certain workloads.  In contrast,
even our SpinLock gives all the waiting processes nearly same chances to
acquire it.  So, I think idea of improving LWLock in this aspect deserves
at least further investigation.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

06 June 2018, 19:33:47

On Tue, Jun 5, 2018 at 7:35 PM, Alexander Korotkov <a.korotkov@postgrespro.ru> wrote:

On Tue, Jun 5, 2018 at 4:02 PM Andres Freund <andres@anarazel.de> wrote:
> On 2018-06-05 13:09:08 +0300, Alexander Korotkov wrote:
> > It appears that buffer replacement happening inside relation
> > extension lock is affected by starvation on exclusive buffer mapping
> > lwlocks and buffer content lwlocks, caused by many concurrent shared
> > lockers. So, fair lwlock patch have no direct influence to relation
> > extension lock, which is naturally not even lwlock...
>
> Yea, that makes sense. I wonder how much the fix here is to "pre-clear"
> a victim buffer, and how much is a saner buffer replacement
> implementation (either by going away from O(NBuffers), or by having a
> queue of clean victim buffers like my bgwriter replacement).

The particular thing I observed on our environment is BufferAlloc()
waiting hours on buffer partition lock. Increasing NUM_BUFFER_PARTITIONS
didn't give any significant help. It appears that very hot page (root page of
some frequently used index) reside on that partition, so this partition was
continuously under shared lock. So, in order to resolve without changing
LWLock, we probably should move our buffers hash table to something
lockless.

I think Robert's chash stuff [1] might be helpful to reduce the contention you are seeing.

[1] - https://www.postgresql.org/message-id/CA%2BTgmoYE4t-Pt%2Bv08kMO5u_XN-HNKBWtfMgcUXEGBrQiVgdV9Q%40mail.gmail.com

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

26 June 2018, 13:16:48

On Fri, Apr 27, 2018 at 4:25 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund <andres@anarazel.de> wrote:
>>> I think the real question is whether the scenario is common enough to
>>> worry about.  In practice, you'd have to be extremely unlucky to be
>>> doing many bulk loads at the same time that all happened to hash to
>>> the same bucket.
>>
>> With a bunch of parallel bulkloads into partitioned tables that really
>> doesn't seem that unlikely?
>
> It increases the likelihood of collisions, but probably decreases the
> number of cases where the contention gets really bad.
>
> For example, suppose each table has 100 partitions and you are
> bulk-loading 10 of them at a time.  It's virtually certain that you
> will have some collisions, but the amount of contention within each
> bucket will remain fairly low because each backend spends only 1% of
> its time in the bucket corresponding to any given partition.
>

I share another result of performance evaluation between current HEAD
and current HEAD with v13 patch(N_RELEXTLOCK_ENTS = 1024).

Type of table: normal table, unlogged table
Number of child tables : 16, 64 (all tables are located on the same tablespace)
Number of clients : 32
Number of trials : 100
Duration: 180 seconds for each trials

The hardware spec of server is Intel Xeon 2.4GHz (HT 160cores), 256GB
RAM, NVMe SSD 1.5TB.
Each clients load 10kB random data across all partitioned tables.

Here is the result.

 childs |   type   | target  |  avg_tps   | diff with HEAD
--------+----------+---------+------------+------------------
     16 | normal   | HEAD    |   1643.833 |
     16 | normal   | Patched |  1619.5404 |      0.985222
     16 | unlogged | HEAD    |  9069.3543 |
     16 | unlogged | Patched |  9368.0263 |      1.032932
     64 | normal   | HEAD    |   1598.698 |
     64 | normal   | Patched |  1587.5906 |      0.993052
     64 | unlogged | HEAD    |  9629.7315 |
     64 | unlogged | Patched | 10208.2196 |      1.060073
(8 rows)

For normal tables, loading tps decreased 1% ~ 2% with this patch
whereas it increased 3% ~ 6% for unlogged tables. There were
collisions at 0 ~ 5 relation extension lock slots between 2 relations
in the 64 child tables case but it didn't seem to affect the tps.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Michael Paquier

Date:

01 October 2018, 12:54:06

On Wed, Jun 06, 2018 at 07:03:47PM +0530, Amit Kapila wrote:
> I think Robert's chash stuff [1] might be helpful to reduce the contention
> you are seeing.

Latest patch available does not apply, so I moved it to next CF.  The
thread has died a bit as well...
--
Michael

Attachment

signature.asc

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dmitry Dolgov

Date:

29 November 2018, 19:18:34

> On Mon, Oct 1, 2018 at 8:54 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Wed, Jun 06, 2018 at 07:03:47PM +0530, Amit Kapila wrote:
> > I think Robert's chash stuff [1] might be helpful to reduce the contention
> > you are seeing.
>
> Latest patch available does not apply, so I moved it to next CF.  The
> thread has died a bit as well...

Unfortunately, patch is still needs to be rebased. Could you do this, are there
any plans about the patch?

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

30 November 2018, 04:02:51

On Fri, Nov 30, 2018 at 1:17 AM Dmitry Dolgov <9erthalion6@gmail.com> wrote:
>
> > On Mon, Oct 1, 2018 at 8:54 AM Michael Paquier <michael@paquier.xyz> wrote:
> >
> > On Wed, Jun 06, 2018 at 07:03:47PM +0530, Amit Kapila wrote:
> > > I think Robert's chash stuff [1] might be helpful to reduce the contention
> > > you are seeing.
> >
> > Latest patch available does not apply, so I moved it to next CF.  The
> > thread has died a bit as well...
>
> Unfortunately, patch is still needs to be rebased. Could you do this, are there
> any plans about the patch?

I have a plan but it's a future plan. This patch is for parallel
vacuum patch. As I mentioned at that thread[1], I'm  focusing on only
parallel index vacuum, which would not require the relation extension
lock improvements for now. Therefore, I want to withdraw this patch
and to reactivate when we need this enhancement.

So I think we can mark it as 'Returned with feedback'.

[1] https://www.postgresql.org/message-id/CAD21AoDhAutvKbQ37Btf4taMVbQaOaSvOpxpLgu814T1-OqYGg%40mail.gmail.com

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

03 February 2020, 14:03:22

On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Fri, Apr 27, 2018 at 4:25 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> > On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund <andres@anarazel.de> wrote:
> >>> I think the real question is whether the scenario is common enough to
> >>> worry about.  In practice, you'd have to be extremely unlucky to be
> >>> doing many bulk loads at the same time that all happened to hash to
> >>> the same bucket.
> >>
> >> With a bunch of parallel bulkloads into partitioned tables that really
> >> doesn't seem that unlikely?
> >
> > It increases the likelihood of collisions, but probably decreases the
> > number of cases where the contention gets really bad.
> >
> > For example, suppose each table has 100 partitions and you are
> > bulk-loading 10 of them at a time.  It's virtually certain that you
> > will have some collisions, but the amount of contention within each
> > bucket will remain fairly low because each backend spends only 1% of
> > its time in the bucket corresponding to any given partition.
> >
>
> I share another result of performance evaluation between current HEAD
> and current HEAD with v13 patch(N_RELEXTLOCK_ENTS = 1024).
>
> Type of table: normal table, unlogged table
> Number of child tables : 16, 64 (all tables are located on the same tablespace)
> Number of clients : 32
> Number of trials : 100
> Duration: 180 seconds for each trials
>
> The hardware spec of server is Intel Xeon 2.4GHz (HT 160cores), 256GB
> RAM, NVMe SSD 1.5TB.
> Each clients load 10kB random data across all partitioned tables.
>
> Here is the result.
>
>  childs |   type   | target  |  avg_tps   | diff with HEAD
> --------+----------+---------+------------+------------------
>      16 | normal   | HEAD    |   1643.833 |
>      16 | normal   | Patched |  1619.5404 |      0.985222
>      16 | unlogged | HEAD    |  9069.3543 |
>      16 | unlogged | Patched |  9368.0263 |      1.032932
>      64 | normal   | HEAD    |   1598.698 |
>      64 | normal   | Patched |  1587.5906 |      0.993052
>      64 | unlogged | HEAD    |  9629.7315 |
>      64 | unlogged | Patched | 10208.2196 |      1.060073
> (8 rows)
>
> For normal tables, loading tps decreased 1% ~ 2% with this patch
> whereas it increased 3% ~ 6% for unlogged tables. There were
> collisions at 0 ~ 5 relation extension lock slots between 2 relations
> in the 64 child tables case but it didn't seem to affect the tps.
>

AFAIU, this resembles the workload that Andres was worried about.   I
think we should once run this test in a different environment, but
considering this to be correct and repeatable, where do we go with
this patch especially when we know it improves many workloads [1] as
well.  We know that on a pathological case constructed by Mithun [2],
this causes regression as well.  I am not sure if the test done by
Mithun really mimics any real-world workload as he has tested by
making N_RELEXTLOCK_ENTS = 1 to hit the worst case.

Sawada-San, if you have a script or data for the test done by you,
then please share it so that others can also try to reproduce it.

[1] - https://www.postgresql.org/message-id/4c171ffe-e3ee-acc5-9066-a40d52bc5ae9%40postgrespro.ru
[2] - https://www.postgresql.org/message-id/CAD__Oug52j%3DDQMoP2b%3DVY7wZb0S9wMNu4irXOH3-ZjFkzWZPGg%40mail.gmail.com

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

05 February 2020, 09:36:45

On Mon, Feb 3, 2020 at 8:03 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Fri, Apr 27, 2018 at 4:25 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> > > On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund <andres@anarazel.de> wrote:
> > >>> I think the real question is whether the scenario is common enough to
> > >>> worry about.  In practice, you'd have to be extremely unlucky to be
> > >>> doing many bulk loads at the same time that all happened to hash to
> > >>> the same bucket.
> > >>
> > >> With a bunch of parallel bulkloads into partitioned tables that really
> > >> doesn't seem that unlikely?
> > >
> > > It increases the likelihood of collisions, but probably decreases the
> > > number of cases where the contention gets really bad.
> > >
> > > For example, suppose each table has 100 partitions and you are
> > > bulk-loading 10 of them at a time.  It's virtually certain that you
> > > will have some collisions, but the amount of contention within each
> > > bucket will remain fairly low because each backend spends only 1% of
> > > its time in the bucket corresponding to any given partition.
> > >
> >
> > I share another result of performance evaluation between current HEAD
> > and current HEAD with v13 patch(N_RELEXTLOCK_ENTS = 1024).
> >
> > Type of table: normal table, unlogged table
> > Number of child tables : 16, 64 (all tables are located on the same tablespace)
> > Number of clients : 32
> > Number of trials : 100
> > Duration: 180 seconds for each trials
> >
> > The hardware spec of server is Intel Xeon 2.4GHz (HT 160cores), 256GB
> > RAM, NVMe SSD 1.5TB.
> > Each clients load 10kB random data across all partitioned tables.
> >
> > Here is the result.
> >
> >  childs |   type   | target  |  avg_tps   | diff with HEAD
> > --------+----------+---------+------------+------------------
> >      16 | normal   | HEAD    |   1643.833 |
> >      16 | normal   | Patched |  1619.5404 |      0.985222
> >      16 | unlogged | HEAD    |  9069.3543 |
> >      16 | unlogged | Patched |  9368.0263 |      1.032932
> >      64 | normal   | HEAD    |   1598.698 |
> >      64 | normal   | Patched |  1587.5906 |      0.993052
> >      64 | unlogged | HEAD    |  9629.7315 |
> >      64 | unlogged | Patched | 10208.2196 |      1.060073
> > (8 rows)
> >
> > For normal tables, loading tps decreased 1% ~ 2% with this patch
> > whereas it increased 3% ~ 6% for unlogged tables. There were
> > collisions at 0 ~ 5 relation extension lock slots between 2 relations
> > in the 64 child tables case but it didn't seem to affect the tps.
> >
>
> AFAIU, this resembles the workload that Andres was worried about.   I
> think we should once run this test in a different environment, but
> considering this to be correct and repeatable, where do we go with
> this patch especially when we know it improves many workloads [1] as
> well.  We know that on a pathological case constructed by Mithun [2],
> this causes regression as well.  I am not sure if the test done by
> Mithun really mimics any real-world workload as he has tested by
> making N_RELEXTLOCK_ENTS = 1 to hit the worst case.
>
> Sawada-San, if you have a script or data for the test done by you,
> then please share it so that others can also try to reproduce it.

Unfortunately the environment I used for performance verification is
no longer available.

I agree to run this test in a different environment. I've attached the
rebased version patch. I'm measuring the performance with/without
patch, so will share the results.

Regards,

--
Masahiko Sawada  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachment

v14-0001-Move-relation-extension-locks-out-of-heavyweigth.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Mahendra Singh Thalor

Date:

05 February 2020, 23:26:58

On Wed, 5 Feb 2020 at 12:07, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Feb 3, 2020 at 8:03 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Fri, Apr 27, 2018 at 4:25 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> > > > On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund <andres@anarazel.de> wrote:
> > > >>> I think the real question is whether the scenario is common enough to
> > > >>> worry about. In practice, you'd have to be extremely unlucky to be
> > > >>> doing many bulk loads at the same time that all happened to hash to
> > > >>> the same bucket.
> > > >>
> > > >> With a bunch of parallel bulkloads into partitioned tables that really
> > > >> doesn't seem that unlikely?
> > > >
> > > > It increases the likelihood of collisions, but probably decreases the
> > > > number of cases where the contention gets really bad.
> > > >
> > > > For example, suppose each table has 100 partitions and you are
> > > > bulk-loading 10 of them at a time. It's virtually certain that you
> > > > will have some collisions, but the amount of contention within each
> > > > bucket will remain fairly low because each backend spends only 1% of
> > > > its time in the bucket corresponding to any given partition.
> > > >
> > >
> > > I share another result of performance evaluation between current HEAD
> > > and current HEAD with v13 patch(N_RELEXTLOCK_ENTS = 1024).
> > >
> > > Type of table: normal table, unlogged table
> > > Number of child tables : 16, 64 (all tables are located on the same tablespace)
> > > Number of clients : 32
> > > Number of trials : 100
> > > Duration: 180 seconds for each trials
> > >
> > > The hardware spec of server is Intel Xeon 2.4GHz (HT 160cores), 256GB
> > > RAM, NVMe SSD 1.5TB.
> > > Each clients load 10kB random data across all partitioned tables.
> > >
> > > Here is the result.
> > >
> > > childs | type | target | avg_tps | diff with HEAD
> > > --------+----------+---------+------------+------------------
> > > 16 | normal | HEAD | 1643.833 |
> > > 16 | normal | Patched | 1619.5404 | 0.985222
> > > 16 | unlogged | HEAD | 9069.3543 |
> > > 16 | unlogged | Patched | 9368.0263 | 1.032932
> > > 64 | normal | HEAD | 1598.698 |
> > > 64 | normal | Patched | 1587.5906 | 0.993052
> > > 64 | unlogged | HEAD | 9629.7315 |
> > > 64 | unlogged | Patched | 10208.2196 | 1.060073
> > > (8 rows)
> > >
> > > For normal tables, loading tps decreased 1% ~ 2% with this patch
> > > whereas it increased 3% ~ 6% for unlogged tables. There were
> > > collisions at 0 ~ 5 relation extension lock slots between 2 relations
> > > in the 64 child tables case but it didn't seem to affect the tps.
> > >
> >
> > AFAIU, this resembles the workload that Andres was worried about. I
> > think we should once run this test in a different environment, but
> > considering this to be correct and repeatable, where do we go with
> > this patch especially when we know it improves many workloads [1] as
> > well. We know that on a pathological case constructed by Mithun [2],
> > this causes regression as well. I am not sure if the test done by
> > Mithun really mimics any real-world workload as he has tested by
> > making N_RELEXTLOCK_ENTS = 1 to hit the worst case.
> >
> > Sawada-San, if you have a script or data for the test done by you,
> > then please share it so that others can also try to reproduce it.
>
> Unfortunately the environment I used for performance verification is
> no longer available.
>
> I agree to run this test in a different environment. I've attached the
> rebased version patch. I'm measuring the performance with/without
> patch, so will share the results.
>

Thanks Sawada-san for patch.

From last few days, I was reading this thread and was reviewing v13 patch. To debug and test, I did re-base of v13 patch. I compared my re-based patch and v14 patch. I think, ordering of header files is not alphabetically in v14 patch. (I haven't reviewed v14 patch fully because before review, I wanted to test false sharing). While debugging, I didn't noticed any hang or lock related issue.

I did some testing to test false sharing(bulk insert, COPY data, bulk insert into partitions tables). Below is the testing summary.

Test setup(Bulk insert into partition tables):

autovacuum=off

shared_buffers=512MB -c max_wal_size=20GB -c checkpoint_timeout=12min

Basically, I created a table with 13 partitions. Using pgbench, I inserted bulk data. I used below pgbench command:

./pgbench -c $threads -j $threads -T 180 -f insert1.sql@1 -f insert2.sql@1 -f insert3.sql@1 -f insert4.sql@1 postgres

I took scripts from previews mails and modified. For reference, I am attaching test scripts. I tested with default 1024 slots(N_RELEXTLOCK_ENTS = 1024).

Clients HEAD (tps) With v14 patch (tps) %change (time: 180s)

1 92.979796 100.877446 +8.49 %

32 392.881863 388.470622 -1.12 %

56 551.753235 528.018852 -4.30 %

60 648.273767 653.251507 +0.76 %

64 645.975124 671.322140 +3.92 %

66 662.728010 673.399762 +1.61 %

70 647.103183 660.694914 +2.10 %

74 648.824027 676.487622 +4.26 %

From above results, we can see that in most cases, TPS is slightly increased with v14 patch. I am still testing and will post my results.

I want to test extension lock by blocking use of fsm(use_fsm=false in code). I think, if we block use of fsm, then load will increase into extension lock. Is this correct way to test?

Please let me know if you have any specific testing scenario.

--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

Attachment

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

06 February 2020, 07:13:59

On Thu, Feb 6, 2020 at 1:57 AM Mahendra Singh Thalor <mahi6run@gmail.com> wrote:
>
> On Wed, 5 Feb 2020 at 12:07, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Feb 3, 2020 at 8:03 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Fri, Apr 27, 2018 at 4:25 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> > > > > On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund <andres@anarazel.de> wrote:
> > > > >>> I think the real question is whether the scenario is common enough to
> > > > >>> worry about.  In practice, you'd have to be extremely unlucky to be
> > > > >>> doing many bulk loads at the same time that all happened to hash to
> > > > >>> the same bucket.
> > > > >>
> > > > >> With a bunch of parallel bulkloads into partitioned tables that really
> > > > >> doesn't seem that unlikely?
> > > > >
> > > > > It increases the likelihood of collisions, but probably decreases the
> > > > > number of cases where the contention gets really bad.
> > > > >
> > > > > For example, suppose each table has 100 partitions and you are
> > > > > bulk-loading 10 of them at a time.  It's virtually certain that you
> > > > > will have some collisions, but the amount of contention within each
> > > > > bucket will remain fairly low because each backend spends only 1% of
> > > > > its time in the bucket corresponding to any given partition.
> > > > >
> > > >
> > > > I share another result of performance evaluation between current HEAD
> > > > and current HEAD with v13 patch(N_RELEXTLOCK_ENTS = 1024).
> > > >
> > > > Type of table: normal table, unlogged table
> > > > Number of child tables : 16, 64 (all tables are located on the same tablespace)
> > > > Number of clients : 32
> > > > Number of trials : 100
> > > > Duration: 180 seconds for each trials
> > > >
> > > > The hardware spec of server is Intel Xeon 2.4GHz (HT 160cores), 256GB
> > > > RAM, NVMe SSD 1.5TB.
> > > > Each clients load 10kB random data across all partitioned tables.
> > > >
> > > > Here is the result.
> > > >
> > > >  childs |   type   | target  |  avg_tps   | diff with HEAD
> > > > --------+----------+---------+------------+------------------
> > > >      16 | normal   | HEAD    |   1643.833 |
> > > >      16 | normal   | Patched |  1619.5404 |      0.985222
> > > >      16 | unlogged | HEAD    |  9069.3543 |
> > > >      16 | unlogged | Patched |  9368.0263 |      1.032932
> > > >      64 | normal   | HEAD    |   1598.698 |
> > > >      64 | normal   | Patched |  1587.5906 |      0.993052
> > > >      64 | unlogged | HEAD    |  9629.7315 |
> > > >      64 | unlogged | Patched | 10208.2196 |      1.060073
> > > > (8 rows)
> > > >
> > > > For normal tables, loading tps decreased 1% ~ 2% with this patch
> > > > whereas it increased 3% ~ 6% for unlogged tables. There were
> > > > collisions at 0 ~ 5 relation extension lock slots between 2 relations
> > > > in the 64 child tables case but it didn't seem to affect the tps.
> > > >
> > >
> > > AFAIU, this resembles the workload that Andres was worried about.   I
> > > think we should once run this test in a different environment, but
> > > considering this to be correct and repeatable, where do we go with
> > > this patch especially when we know it improves many workloads [1] as
> > > well.  We know that on a pathological case constructed by Mithun [2],
> > > this causes regression as well.  I am not sure if the test done by
> > > Mithun really mimics any real-world workload as he has tested by
> > > making N_RELEXTLOCK_ENTS = 1 to hit the worst case.
> > >
> > > Sawada-San, if you have a script or data for the test done by you,
> > > then please share it so that others can also try to reproduce it.
> >
> > Unfortunately the environment I used for performance verification is
> > no longer available.
> >
> > I agree to run this test in a different environment. I've attached the
> > rebased version patch. I'm measuring the performance with/without
> > patch, so will share the results.
> >
>
> Thanks Sawada-san for patch.
>
> From last few days, I was reading this thread and was reviewing v13 patch.  To debug and test, I did re-base of v13
patch.I compared my re-based patch and v14 patch. I think,  ordering of header files is not alphabetically in v14
patch.(I haven't reviewed v14 patch fully because before review, I wanted to test false sharing).  While debugging, I
didn'tnoticed any hang or lock related issue. 
>
> I did some testing to test false sharing(bulk insert, COPY data, bulk insert into partitions tables).  Below is the
testingsummary. 
>
> Test setup(Bulk insert into partition tables):
> autovacuum=off
> shared_buffers=512MB -c max_wal_size=20GB -c checkpoint_timeout=12min
>
> Basically, I created a table with 13 partitions. Using pgbench, I inserted bulk data. I used below pgbench command:
> ./pgbench -c $threads -j $threads -T 180 -f insert1.sql@1 -f insert2.sql@1 -f insert3.sql@1 -f insert4.sql@1 postgres
>
> I took scripts from previews mails and modified. For reference, I am attaching test scripts.  I tested with default
1024slots(N_RELEXTLOCK_ENTS = 1024). 
>
> Clients          HEAD (tps)                     With v14 patch (tps)      %change      (time: 180s)
> 1                    92.979796                        100.877446                     +8.49 %
> 32                   392.881863                      388.470622                    -1.12 %
> 56                   551.753235                       528.018852                   -4.30 %
> 60                   648.273767                       653.251507                   +0.76 %
> 64                   645.975124                       671.322140                   +3.92 %
> 66                   662.728010                       673.399762                   +1.61 %
> 70                   647.103183                       660.694914                   +2.10 %
> 74                   648.824027                       676.487622                  +4.26 %
>
> From above results, we can see that in most cases, TPS is slightly increased with v14 patch. I am still testing and
willpost my results. 
>

The number at 56 and 74 client count seem slightly suspicious.   Can
you please repeat those tests?  Basically, I am not able to come up
with a theory why at 56 clients the performance with the patch is a
bit lower and then at 74 it is higher.

> I want to test extension lock by blocking use of fsm(use_fsm=false in code).  I think, if we block use of fsm, then
loadwill increase into extension lock.  Is this correct way to test? 
>

Hmm, I think instead of directly hacking the code, you might want to
use the operation (probably cluster or vacuum full) where we set
HEAP_INSERT_SKIP_FSM.  I think along with this you can try with
unlogged tables because that might stress the extension lock.

In the above test, you might want to test with a higher number of
partitions (say up to 100) as well.  Also, see if you want to use the
Copy command.

> Please let me know if you have any specific testing scenario.
>

Can you test the scenario mentioned by Konstantin Knizhnik [1] where
this patch has shown significant gain?  You might want to use a higher
core count machine to test it.

One thing we can do is to somehow measure the collisions on each bucket.

[1] - https://www.postgresql.org/message-id/ef81da49-d491-db86-3ef6-5138d091fe91%40postgrespro.ru

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

06 February 2020, 07:16:18

On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Type of table: normal table, unlogged table
> Number of child tables : 16, 64 (all tables are located on the same tablespace)
> Number of clients : 32
> Number of trials : 100
> Duration: 180 seconds for each trials
>
> The hardware spec of server is Intel Xeon 2.4GHz (HT 160cores), 256GB
> RAM, NVMe SSD 1.5TB.
> Each clients load 10kB random data across all partitioned tables.
>
> Here is the result.
>
>  childs |   type   | target  |  avg_tps   | diff with HEAD
> --------+----------+---------+------------+------------------
>      16 | normal   | HEAD    |   1643.833 |
>      16 | normal   | Patched |  1619.5404 |      0.985222
>      16 | unlogged | HEAD    |  9069.3543 |
>      16 | unlogged | Patched |  9368.0263 |      1.032932
>      64 | normal   | HEAD    |   1598.698 |
>      64 | normal   | Patched |  1587.5906 |      0.993052
>      64 | unlogged | HEAD    |  9629.7315 |
>      64 | unlogged | Patched | 10208.2196 |      1.060073
> (8 rows)
>
> For normal tables, loading tps decreased 1% ~ 2% with this patch
> whereas it increased 3% ~ 6% for unlogged tables. There were
> collisions at 0 ~ 5 relation extension lock slots between 2 relations
> in the 64 child tables case but it didn't seem to affect the tps.
>

How did you measure the collisions in this test?  I think it is better
if Mahendra can also use the same technique in measuring that count.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

06 February 2020, 08:32:40

On Thu, 6 Feb 2020 at 13:16, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Type of table: normal table, unlogged table
> > Number of child tables : 16, 64 (all tables are located on the same tablespace)
> > Number of clients : 32
> > Number of trials : 100
> > Duration: 180 seconds for each trials
> >
> > The hardware spec of server is Intel Xeon 2.4GHz (HT 160cores), 256GB
> > RAM, NVMe SSD 1.5TB.
> > Each clients load 10kB random data across all partitioned tables.
> >
> > Here is the result.
> >
> >  childs |   type   | target  |  avg_tps   | diff with HEAD
> > --------+----------+---------+------------+------------------
> >      16 | normal   | HEAD    |   1643.833 |
> >      16 | normal   | Patched |  1619.5404 |      0.985222
> >      16 | unlogged | HEAD    |  9069.3543 |
> >      16 | unlogged | Patched |  9368.0263 |      1.032932
> >      64 | normal   | HEAD    |   1598.698 |
> >      64 | normal   | Patched |  1587.5906 |      0.993052
> >      64 | unlogged | HEAD    |  9629.7315 |
> >      64 | unlogged | Patched | 10208.2196 |      1.060073
> > (8 rows)
> >
> > For normal tables, loading tps decreased 1% ~ 2% with this patch
> > whereas it increased 3% ~ 6% for unlogged tables. There were
> > collisions at 0 ~ 5 relation extension lock slots between 2 relations
> > in the 64 child tables case but it didn't seem to affect the tps.
> >
>
> How did you measure the collisions in this test?  I think it is better
> if Mahendra can also use the same technique in measuring that count.
>

I created a created a SQL function that returns the hash value of the
lock tag, which is tag_hash(locktag, sizeof(RelExtLockTag)) %
N_RELEXTLOCK_ENTS. And examined the hash values of all tables.

Regards,

--
Masahiko Sawada            http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Mahendra Singh Thalor

Date:

07 February 2020, 21:57:40

On Thu, 6 Feb 2020 at 09:44, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Feb 6, 2020 at 1:57 AM Mahendra Singh Thalor <mahi6run@gmail.com> wrote:
> >
> > On Wed, 5 Feb 2020 at 12:07, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Mon, Feb 3, 2020 at 8:03 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > On Fri, Apr 27, 2018 at 4:25 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> > > > > > On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund <andres@anarazel.de> wrote:
> > > > > >>> I think the real question is whether the scenario is common enough to
> > > > > >>> worry about. In practice, you'd have to be extremely unlucky to be
> > > > > >>> doing many bulk loads at the same time that all happened to hash to
> > > > > >>> the same bucket.
> > > > > >>
> > > > > >> With a bunch of parallel bulkloads into partitioned tables that really
> > > > > >> doesn't seem that unlikely?
> > > > > >
> > > > > > It increases the likelihood of collisions, but probably decreases the
> > > > > > number of cases where the contention gets really bad.
> > > > > >
> > > > > > For example, suppose each table has 100 partitions and you are
> > > > > > bulk-loading 10 of them at a time. It's virtually certain that you
> > > > > > will have some collisions, but the amount of contention within each
> > > > > > bucket will remain fairly low because each backend spends only 1% of
> > > > > > its time in the bucket corresponding to any given partition.
> > > > > >
> > > > >
> > > > > I share another result of performance evaluation between current HEAD
> > > > > and current HEAD with v13 patch(N_RELEXTLOCK_ENTS = 1024).
> > > > >
> > > > > Type of table: normal table, unlogged table
> > > > > Number of child tables : 16, 64 (all tables are located on the same tablespace)
> > > > > Number of clients : 32
> > > > > Number of trials : 100
> > > > > Duration: 180 seconds for each trials
> > > > >
> > > > > The hardware spec of server is Intel Xeon 2.4GHz (HT 160cores), 256GB
> > > > > RAM, NVMe SSD 1.5TB.
> > > > > Each clients load 10kB random data across all partitioned tables.
> > > > >
> > > > > Here is the result.
> > > > >
> > > > > childs | type | target | avg_tps | diff with HEAD
> > > > > --------+----------+---------+------------+------------------
> > > > > 16 | normal | HEAD | 1643.833 |
> > > > > 16 | normal | Patched | 1619.5404 | 0.985222
> > > > > 16 | unlogged | HEAD | 9069.3543 |
> > > > > 16 | unlogged | Patched | 9368.0263 | 1.032932
> > > > > 64 | normal | HEAD | 1598.698 |
> > > > > 64 | normal | Patched | 1587.5906 | 0.993052
> > > > > 64 | unlogged | HEAD | 9629.7315 |
> > > > > 64 | unlogged | Patched | 10208.2196 | 1.060073
> > > > > (8 rows)
> > > > >
> > > > > For normal tables, loading tps decreased 1% ~ 2% with this patch
> > > > > whereas it increased 3% ~ 6% for unlogged tables. There were
> > > > > collisions at 0 ~ 5 relation extension lock slots between 2 relations
> > > > > in the 64 child tables case but it didn't seem to affect the tps.
> > > > >
> > > >
> > > > AFAIU, this resembles the workload that Andres was worried about. I
> > > > think we should once run this test in a different environment, but
> > > > considering this to be correct and repeatable, where do we go with
> > > > this patch especially when we know it improves many workloads [1] as
> > > > well. We know that on a pathological case constructed by Mithun [2],
> > > > this causes regression as well. I am not sure if the test done by
> > > > Mithun really mimics any real-world workload as he has tested by
> > > > making N_RELEXTLOCK_ENTS = 1 to hit the worst case.
> > > >
> > > > Sawada-San, if you have a script or data for the test done by you,
> > > > then please share it so that others can also try to reproduce it.
> > >
> > > Unfortunately the environment I used for performance verification is
> > > no longer available.
> > >
> > > I agree to run this test in a different environment. I've attached the
> > > rebased version patch. I'm measuring the performance with/without
> > > patch, so will share the results.
> > >
> >
> > Thanks Sawada-san for patch.
> >
> > From last few days, I was reading this thread and was reviewing v13 patch. To debug and test, I did re-base of v13 patch. I compared my re-based patch and v14 patch. I think, ordering of header files is not alphabetically in v14 patch. (I haven't reviewed v14 patch fully because before review, I wanted to test false sharing). While debugging, I didn't noticed any hang or lock related issue.
> >
> > I did some testing to test false sharing(bulk insert, COPY data, bulk insert into partitions tables). Below is the testing summary.
> >
> > Test setup(Bulk insert into partition tables):
> > autovacuum=off
> > shared_buffers=512MB -c max_wal_size=20GB -c checkpoint_timeout=12min
> >
> > Basically, I created a table with 13 partitions. Using pgbench, I inserted bulk data. I used below pgbench command:
> > ./pgbench -c $threads -j $threads -T 180 -f insert1.sql@1 -f insert2.sql@1 -f insert3.sql@1 -f insert4.sql@1 postgres
> >
> > I took scripts from previews mails and modified. For reference, I am attaching test scripts. I tested with default 1024 slots(N_RELEXTLOCK_ENTS = 1024).
> >
> > Clients HEAD (tps) With v14 patch (tps) %change (time: 180s)
> > 1 92.979796 100.877446 +8.49 %
> > 32 392.881863 388.470622 -1.12 %
> > 56 551.753235 528.018852 -4.30 %
> > 60 648.273767 653.251507 +0.76 %
> > 64 645.975124 671.322140 +3.92 %
> > 66 662.728010 673.399762 +1.61 %
> > 70 647.103183 660.694914 +2.10 %
> > 74 648.824027 676.487622 +4.26 %
> >
> > From above results, we can see that in most cases, TPS is slightly increased with v14 patch. I am still testing and will post my results.
> >
>
> The number at 56 and 74 client count seem slightly suspicious. Can
> you please repeat those tests? Basically, I am not able to come up
> with a theory why at 56 clients the performance with the patch is a
> bit lower and then at 74 it is higher.

Okay. I will repeat test.

>
> > I want to test extension lock by blocking use of fsm(use_fsm=false in code). I think, if we block use of fsm, then load will increase into extension lock. Is this correct way to test?
> >
>
> Hmm, I think instead of directly hacking the code, you might want to
> use the operation (probably cluster or vacuum full) where we set
> HEAP_INSERT_SKIP_FSM. I think along with this you can try with
> unlogged tables because that might stress the extension lock.

Okay. I will test.

>
> In the above test, you might want to test with a higher number of
> partitions (say up to 100) as well. Also, see if you want to use the
> Copy command.

Okay. I will test.

>
> > Please let me know if you have any specific testing scenario.
> >
>
> Can you test the scenario mentioned by Konstantin Knizhnik [1] where
> this patch has shown significant gain? You might want to use a higher
> core count machine to test it.

I followed Konstantin Knizhnik steps and tested insert with high core . Below is the test summary:

Test setup:

autovacuum = off

max_connections = 1000

My testing machine:
$ lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 192
On-line CPU(s) list: 0-191
Thread(s) per core: 8
Core(s) per socket: 1
Socket(s): 24
NUMA node(s): 4
Model: IBM,8286-42A
L1d cache: 64K
L1i cache: 32K
L2 cache: 512K
L3 cache: 8192K
NUMA node0 CPU(s): 0-47
NUMA node1 CPU(s): 48-95
NUMA node2 CPU(s): 96-143
NUMA node3 CPU(s): 144-191

create table test (i int, md5 text);

insert.sql:
begin;
insert into test select i, md5(i::text) from generate_series(1,1000) AS i;
end;

pgbench command:
./pgbench postgres -c 1000 -j 36 -T 180 -P 10 -f insert.sql >> results.txt

I tested with 1000 clients. Below it the tps:

TPS on HEAD:
Run 1) : 608.908721
Run 2) : 599.962863
Run 3) : 606.378819
Run 4) : 607.174076
Run 5) : 598.531958

TPS with v14 patch: ( N_RELEXTLOCK_ENTS = 1024)
Run 1) : 649.488472
Run 2) : 657.902261
Run 3) : 654.478580
Run 4) : 648.085126
Run 5) : 647.171482

%change = +7.10 %

Apart from above test, I did some more tests (N_RELEXTLOCK_ENTS = 1024):

1) bulk insert into 1 table for T = 180s, 3600s, clients-100,1000, table- logged,unlogged

2) copy command

3) bulk load into table having 13 partitions

In all the cases, I can see 4-9% improvement in TPS as compared to HEAD.

@Konstantin Knizhnik, if you remember, then please let me know that how much tps gain was observed in your insert test? Is it nearby to my results?

--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Mahendra Singh Thalor

Date:

10 February 2020, 19:58:22

On Sat, 8 Feb 2020 at 00:27, Mahendra Singh Thalor <mahi6run@gmail.com> wrote:
>
> On Thu, 6 Feb 2020 at 09:44, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Feb 6, 2020 at 1:57 AM Mahendra Singh Thalor <mahi6run@gmail.com> wrote:
> > >
> > > On Wed, 5 Feb 2020 at 12:07, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Mon, Feb 3, 2020 at 8:03 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > > On Tue, Jun 26, 2018 at 12:47 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > > >
> > > > > > On Fri, Apr 27, 2018 at 4:25 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> > > > > > > On Thu, Apr 26, 2018 at 3:10 PM, Andres Freund <andres@anarazel.de> wrote:
> > > > > > >>> I think the real question is whether the scenario is common enough to
> > > > > > >>> worry about. In practice, you'd have to be extremely unlucky to be
> > > > > > >>> doing many bulk loads at the same time that all happened to hash to
> > > > > > >>> the same bucket.
> > > > > > >>
> > > > > > >> With a bunch of parallel bulkloads into partitioned tables that really
> > > > > > >> doesn't seem that unlikely?
> > > > > > >
> > > > > > > It increases the likelihood of collisions, but probably decreases the
> > > > > > > number of cases where the contention gets really bad.
> > > > > > >
> > > > > > > For example, suppose each table has 100 partitions and you are
> > > > > > > bulk-loading 10 of them at a time. It's virtually certain that you
> > > > > > > will have some collisions, but the amount of contention within each
> > > > > > > bucket will remain fairly low because each backend spends only 1% of
> > > > > > > its time in the bucket corresponding to any given partition.
> > > > > > >
> > > > > >
> > > > > > I share another result of performance evaluation between current HEAD
> > > > > > and current HEAD with v13 patch(N_RELEXTLOCK_ENTS = 1024).
> > > > > >
> > > > > > Type of table: normal table, unlogged table
> > > > > > Number of child tables : 16, 64 (all tables are located on the same tablespace)
> > > > > > Number of clients : 32
> > > > > > Number of trials : 100
> > > > > > Duration: 180 seconds for each trials
> > > > > >
> > > > > > The hardware spec of server is Intel Xeon 2.4GHz (HT 160cores), 256GB
> > > > > > RAM, NVMe SSD 1.5TB.
> > > > > > Each clients load 10kB random data across all partitioned tables.
> > > > > >
> > > > > > Here is the result.
> > > > > >
> > > > > > childs | type | target | avg_tps | diff with HEAD
> > > > > > --------+----------+---------+------------+------------------
> > > > > > 16 | normal | HEAD | 1643.833 |
> > > > > > 16 | normal | Patched | 1619.5404 | 0.985222
> > > > > > 16 | unlogged | HEAD | 9069.3543 |
> > > > > > 16 | unlogged | Patched | 9368.0263 | 1.032932
> > > > > > 64 | normal | HEAD | 1598.698 |
> > > > > > 64 | normal | Patched | 1587.5906 | 0.993052
> > > > > > 64 | unlogged | HEAD | 9629.7315 |
> > > > > > 64 | unlogged | Patched | 10208.2196 | 1.060073
> > > > > > (8 rows)
> > > > > >
> > > > > > For normal tables, loading tps decreased 1% ~ 2% with this patch
> > > > > > whereas it increased 3% ~ 6% for unlogged tables. There were
> > > > > > collisions at 0 ~ 5 relation extension lock slots between 2 relations
> > > > > > in the 64 child tables case but it didn't seem to affect the tps.
> > > > > >
> > > > >
> > > > > AFAIU, this resembles the workload that Andres was worried about. I
> > > > > think we should once run this test in a different environment, but
> > > > > considering this to be correct and repeatable, where do we go with
> > > > > this patch especially when we know it improves many workloads [1] as
> > > > > well. We know that on a pathological case constructed by Mithun [2],
> > > > > this causes regression as well. I am not sure if the test done by
> > > > > Mithun really mimics any real-world workload as he has tested by
> > > > > making N_RELEXTLOCK_ENTS = 1 to hit the worst case.
> > > > >
> > > > > Sawada-San, if you have a script or data for the test done by you,
> > > > > then please share it so that others can also try to reproduce it.
> > > >
> > > > Unfortunately the environment I used for performance verification is
> > > > no longer available.
> > > >
> > > > I agree to run this test in a different environment. I've attached the
> > > > rebased version patch. I'm measuring the performance with/without
> > > > patch, so will share the results.
> > > >
> > >
> > > Thanks Sawada-san for patch.
> > >
> > > From last few days, I was reading this thread and was reviewing v13 patch. To debug and test, I did re-base of v13 patch. I compared my re-based patch and v14 patch. I think, ordering of header files is not alphabetically in v14 patch. (I haven't reviewed v14 patch fully because before review, I wanted to test false sharing). While debugging, I didn't noticed any hang or lock related issue.
> > >
> > > I did some testing to test false sharing(bulk insert, COPY data, bulk insert into partitions tables). Below is the testing summary.
> > >
> > > Test setup(Bulk insert into partition tables):
> > > autovacuum=off
> > > shared_buffers=512MB -c max_wal_size=20GB -c checkpoint_timeout=12min
> > >
> > > Basically, I created a table with 13 partitions. Using pgbench, I inserted bulk data. I used below pgbench command:
> > > ./pgbench -c $threads -j $threads -T 180 -f insert1.sql@1 -f insert2.sql@1 -f insert3.sql@1 -f insert4.sql@1 postgres
> > >
> > > I took scripts from previews mails and modified. For reference, I am attaching test scripts. I tested with default 1024 slots(N_RELEXTLOCK_ENTS = 1024).
> > >
> > > Clients HEAD (tps) With v14 patch (tps) %change (time: 180s)
> > > 1 92.979796 100.877446 +8.49 %
> > > 32 392.881863 388.470622 -1.12 %
> > > 56 551.753235 528.018852 -4.30 %
> > > 60 648.273767 653.251507 +0.76 %
> > > 64 645.975124 671.322140 +3.92 %
> > > 66 662.728010 673.399762 +1.61 %
> > > 70 647.103183 660.694914 +2.10 %
> > > 74 648.824027 676.487622 +4.26 %
> > >
> > > From above results, we can see that in most cases, TPS is slightly increased with v14 patch. I am still testing and will post my results.
> > >
> >
> > The number at 56 and 74 client count seem slightly suspicious. Can
> > you please repeat those tests? Basically, I am not able to come up
> > with a theory why at 56 clients the performance with the patch is a
> > bit lower and then at 74 it is higher.
>
> Okay. I will repeat test.

I re-tested in different machine because in previous machine, results are in-consistent

My testing machine:
$ lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 192
On-line CPU(s) list: 0-191
Thread(s) per core: 8
Core(s) per socket: 1
Socket(s): 24
NUMA node(s): 4
Model: IBM,8286-42A
L1d cache: 64K
L1i cache: 32K
L2 cache: 512K
L3 cache: 8192K
NUMA node0 CPU(s): 0-47
NUMA node1 CPU(s): 48-95
NUMA node2 CPU(s): 96-143

NUMA node3 CPU(s): 144-191

./pgbench -c $threads -j $threads -T 180 -f insert1.sql@1 -f insert2.sql@1 -f insert3.sql@1 -f insert4.sql@1 postgres

Clients HEAD(tps) With v14 patch(tps) %change (time: 180s)
1 41.491486 41.375532    -0.27%
32 335.138568 330.028739                 -1.52%
56 353.783930 360.883710                  +2.00%
60 341.741925 359.028041                 +5.05%
64 338.521730 356.511423                  +5.13%
66 339.838921 352.761766                  +3.80%
70 339.305454 353.658425                +4.23%
74 332.016217 348.809042        +5.05%

From above results, it seems that there is very little regression with the patch(+-5%) that can be run to run variation.

>
> >
> > > I want to test extension lock by blocking use of fsm(use_fsm=false in code). I think, if we block use of fsm, then load will increase into extension lock. Is this correct way to test?
> > >
> >
> > Hmm, I think instead of directly hacking the code, you might want to
> > use the operation (probably cluster or vacuum full) where we set
> > HEAP_INSERT_SKIP_FSM. I think along with this you can try with
> > unlogged tables because that might stress the extension lock.
>

> Okay. I will test.

I tested with unlogged tables also. There also I was getting 3-6% gain in tps.

>
> >
> > In the above test, you might want to test with a higher number of
> > partitions (say up to 100) as well. Also, see if you want to use the
> > Copy command.
>

> Okay. I will test.

I tested with 500, 1000, 2000 paratitions. I observed max +5% regress in the tps and there was no performace degradation.

For example:

I created a table with 2000 paratitions and then I checked false sharing.

Slot Number	Slot Freq.	Slot Number	Slot Freq.	Slot Number	Slot Freq.
156	13	973	11	446	10
627	13	52	10	488	10
782	12	103	10	501	10
812	12	113	10	701	10
192	11	175	10	737	10
221	11	235	10	754	10
367	11	254	10	781	10
546	11	314	10	790	10
814	11	419	10	833	10
917	11	424	10	888	10

From above table, we can see that total 13 child tables are falling in same backet (slot 156) so I did bulk-loading only in those 13 child tables to check tps in false sharing but I noticed that there was no performance degradation.

--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

11 February 2020, 05:28:33

On Mon, Feb 10, 2020 at 10:28 PM Mahendra Singh Thalor
<mahi6run@gmail.com> wrote:
>
> On Sat, 8 Feb 2020 at 00:27, Mahendra Singh Thalor <mahi6run@gmail.com> wrote:
> >
> > On Thu, 6 Feb 2020 at 09:44, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > The number at 56 and 74 client count seem slightly suspicious.   Can
> > > you please repeat those tests?  Basically, I am not able to come up
> > > with a theory why at 56 clients the performance with the patch is a
> > > bit lower and then at 74 it is higher.
> >
> > Okay. I will repeat test.
>
> I re-tested in different machine because in previous machine, results are  in-consistent
>

Thanks for doing detailed tests.

> My testing machine:
> $ lscpu
> Architecture:          ppc64le
> Byte Order:            Little Endian
> CPU(s):                192
> On-line CPU(s) list:   0-191
> Thread(s) per core:    8
> Core(s) per socket:    1
> Socket(s):             24
> NUMA node(s):          4
> Model:                 IBM,8286-42A
> L1d cache:             64K
> L1i cache:             32K
> L2 cache:              512K
> L3 cache:              8192K
> NUMA node0 CPU(s):     0-47
> NUMA node1 CPU(s):     48-95
> NUMA node2 CPU(s):     96-143
> NUMA node3 CPU(s):     144-191
>
> ./pgbench -c $threads -j $threads -T 180 -f insert1.sql@1 -f insert2.sql@1 -f insert3.sql@1 -f insert4.sql@1
postgres
>
> Clients        HEAD(tps)            With v14 patch(tps)        %change       (time: 180s)
> 1                 41.491486              41.375532                  -0.27%
> 32              335.138568            330.028739                 -1.52%
> 56             353.783930             360.883710                  +2.00%
> 60             341.741925             359.028041                 +5.05%
> 64             338.521730             356.511423                  +5.13%
> 66             339.838921             352.761766                  +3.80%
> 70            339.305454              353.658425                +4.23%
> 74            332.016217              348.809042                 +5.05%
>
> From above results, it seems that there is very little regression with the patch(+-5%) that can be run to run
variation.
>

Hmm, I don't see 5% regression, rather it is a performance gain of ~5%
with the patch?  When we use regression, that indicates with the patch
performance (TPS) is reduced, but I don't see that in the above
numbers.  Kindly clarify.

> >
> > >
> > > > I want to test extension lock by blocking use of fsm(use_fsm=false in code).  I think, if we block use of fsm,
thenload will increase into extension lock.  Is this correct way to test?
 
> > > >
> > >
> > > Hmm, I think instead of directly hacking the code, you might want to
> > > use the operation (probably cluster or vacuum full) where we set
> > > HEAP_INSERT_SKIP_FSM.  I think along with this you can try with
> > > unlogged tables because that might stress the extension lock.
> >
> > Okay. I will test.
>
> I tested with unlogged tables also.  There also I was getting 3-6% gain in tps.
>
> >
> > >
> > > In the above test, you might want to test with a higher number of
> > > partitions (say up to 100) as well.  Also, see if you want to use the
> > > Copy command.
> >
> > Okay. I will test.
>
> I tested with 500, 1000, 2000 paratitions. I observed max +5% regress in the tps and there was no performace
degradation.
>

Again, I am not sure if you see performance dip here.  I think your
usage of the word 'regression' is not correct or at least confusing.

> For example:
> I created a table with 2000 paratitions and then I checked false sharing.
> Slot NumberSlot Freq.Slot NumberSlot Freq.Slot NumberSlot Freq.
> 156139731144610
> 62713521048810
> 782121031050110
> 812121131070110
> 192111751073710
> 221112351075410
> 367112541078110
> 546113141079010
> 814114191083310
> 917114241088810
>
> From above table, we can see that total 13 child tables are falling in same backet (slot 156) so I did bulk-loading
onlyin those 13 child tables to check tps in false sharing but I noticed that there was no performance degradation.
 
>

Okay.  Is it possible to share these numbers and scripts?

Thanks for doing the detailed tests for this patch.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

11 February 2020, 05:31:34

On Wed, Feb 5, 2020 at 12:07 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
>
> Unfortunately the environment I used for performance verification is
> no longer available.
>
> I agree to run this test in a different environment. I've attached the
> rebased version patch. I'm measuring the performance with/without
> patch, so will share the results.
>

Did you get a chance to run these tests?  Lately, Mahendra has done a
lot of performance testing of this patch and shared his results.  I
don't see much downside with the patch, rather there is a performance
increase of 3-9% in various scenarios.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Tom Lane

Date:

11 February 2020, 18:43:13

I took a brief look through this patch.  I agree with the fundamental
idea that we shouldn't need to use the heavyweight lock manager for
relation extension, since deadlock is not a concern and no backend
should ever need to hold more than one such lock at once.  But it feels
to me like this particular solution is rather seriously overengineered.
I would like to suggest that we do something similar to Robert Haas'
excellent hack (daa7527af) for the !HAVE_SPINLOCK case in lmgr/spin.c,
that is,

* Create some predetermined number N of LWLocks for relation extension.
* When we want to extend some relation R, choose one of those locks
  (say, R's relfilenode number mod N) and lock it.

1. As long as all backends agree on the relation-to-lock mapping, this
provides full security against concurrent extensions of the same
relation.

2. Occasionally a backend will be blocked when it doesn't need to be,
because of false sharing of a lock between two relations that need to
be extended at the same time.  But as long as N is large enough (and
I doubt that it needs to be very large), that will be a negligible
penalty.

3. Aside from being a lot simpler than the proposed extension_lock.c,
this approach involves absolutely negligible overhead beyond the raw
LWLockAcquire and LWLockRelease calls.  I suspect therefore that in
typical noncontended cases it will be faster.  It also does not require
any new resource management overhead, thus eliminating this patch's
small but real penalty on transaction exit/cleanup.

We'd need to do a bit of performance testing to choose a good value
for N.  I think that with N comparable to MaxBackends, the odds of
false sharing being a problem would be quite negligible ... but it
could be that we could get away with a lot less than that.

            regards, tom lane

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

12 February 2020, 05:05:48

On Wed, 12 Feb 2020 at 00:43, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> I took a brief look through this patch.  I agree with the fundamental
> idea that we shouldn't need to use the heavyweight lock manager for
> relation extension, since deadlock is not a concern and no backend
> should ever need to hold more than one such lock at once.  But it feels
> to me like this particular solution is rather seriously overengineered.
> I would like to suggest that we do something similar to Robert Haas'
> excellent hack (daa7527af) for the !HAVE_SPINLOCK case in lmgr/spin.c,
> that is,
>
> * Create some predetermined number N of LWLocks for relation extension.

My original proposal used LWLocks and hash tables for relation
extension but there was a discussion that using LWLocks is not good
because it's not interruptible[1]. Because of this reason and that we
don't need to have two lock level (shared, exclusive) for relation
extension lock we ended up with implementing dedicated lock manager
for extension lock. I think we will have that problem if we use LWLocks.

Regards,

[1] https://www.postgresql.org/message-id/CA%2BTgmoZnWYQvmeqeGyY%2B0j-Tfmx8cTzRadfxJQwK9A-nCQ7GkA%40mail.gmail.com

--
Masahiko Sawada            http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

12 February 2020, 07:54:37

Hi,

On 2020-02-11 08:01:34 +0530, Amit Kapila wrote:
> I don't see much downside with the patch, rather there is a
> performance increase of 3-9% in various scenarios.

As I wrote in [1] I started to look at this patch. My problem with itis
that it just seems like the wrong direction architecturally to
me. There's two main aspects to this:

1) It basically builds a another, more lightweight but less capable, of
   a lock manager that can lock more objects than we can have distinct
   locks for.  It is faster because it uses *one* hashtable without
   conflict handling, because it has fewer lock modes, and because it
   doesn't support detecting deadlocks. And probably some other things.

2) A lot of the contention around file extension comes from us doing
   multiple expensive things under one lock (determining current
   relation size, searching victim buffer, extending file), and in tiny
   increments (growing a 1TB table by 8kb). This patch doesn't address
   that at all.

I've focused on 1) in the email referenced above ([1]). Here I'll focus
on 2).

To quantify my concerns I instrumented postgres to measure the time for
various operations that are part of extending a file (all per
process). The hardware is a pretty fast nvme, with unlogged tables, on a
20/40 core/threads machine. The workload is copying a scale 10
pgbench_accounts into an unindexed, unlogged table using pgbench.

Here are the instrumentations for various client counts, when just
measuring 20s:

1 client:
LOG:  extension time: lock wait: 0.00 lock held: 3.19 filesystem: 1.29 buffersearch: 1.58

2 clients:
LOG:  extension time: lock wait: 0.47 lock held: 2.99 filesystem: 1.24 buffersearch: 1.43
LOG:  extension time: lock wait: 0.60 lock held: 3.05 filesystem: 1.23 buffersearch: 1.50

4 clients:
LOG:  extension time: lock wait: 3.92 lock held: 2.69 filesystem: 1.10 buffersearch: 1.29
LOG:  extension time: lock wait: 4.40 lock held: 2.02 filesystem: 0.81 buffersearch: 0.93
LOG:  extension time: lock wait: 3.86 lock held: 2.59 filesystem: 1.06 buffersearch: 1.22
LOG:  extension time: lock wait: 4.00 lock held: 2.65 filesystem: 1.08 buffersearch: 1.26

8 clients:
LOG:  extension time: lock wait: 6.94 lock held: 1.74 filesystem: 0.70 buffersearch: 0.80
LOG:  extension time: lock wait: 7.16 lock held: 1.81 filesystem: 0.73 buffersearch: 0.82
LOG:  extension time: lock wait: 6.93 lock held: 1.95 filesystem: 0.80 buffersearch: 0.89
LOG:  extension time: lock wait: 7.08 lock held: 1.87 filesystem: 0.76 buffersearch: 0.86
LOG:  extension time: lock wait: 6.95 lock held: 1.95 filesystem: 0.80 buffersearch: 0.89
LOG:  extension time: lock wait: 6.88 lock held: 2.01 filesystem: 0.83 buffersearch: 0.93
LOG:  extension time: lock wait: 6.94 lock held: 2.02 filesystem: 0.82 buffersearch: 0.93
LOG:  extension time: lock wait: 7.02 lock held: 1.95 filesystem: 0.80 buffersearch: 0.89

16 clients:
LOG:  extension time: lock wait: 10.37 lock held: 0.88 filesystem: 0.36 buffersearch: 0.39
LOG:  extension time: lock wait: 10.53 lock held: 0.90 filesystem: 0.37 buffersearch: 0.40
LOG:  extension time: lock wait: 10.72 lock held: 1.01 filesystem: 0.42 buffersearch: 0.45
LOG:  extension time: lock wait: 10.45 lock held: 1.25 filesystem: 0.52 buffersearch: 0.55
LOG:  extension time: lock wait: 10.66 lock held: 0.94 filesystem: 0.38 buffersearch: 0.41
LOG:  extension time: lock wait: 10.50 lock held: 1.27 filesystem: 0.53 buffersearch: 0.56
LOG:  extension time: lock wait: 10.53 lock held: 1.19 filesystem: 0.49 buffersearch: 0.53
LOG:  extension time: lock wait: 10.57 lock held: 1.22 filesystem: 0.50 buffersearch: 0.53
LOG:  extension time: lock wait: 10.72 lock held: 1.17 filesystem: 0.48 buffersearch: 0.52
LOG:  extension time: lock wait: 10.67 lock held: 1.32 filesystem: 0.55 buffersearch: 0.58
LOG:  extension time: lock wait: 10.95 lock held: 0.92 filesystem: 0.38 buffersearch: 0.40
LOG:  extension time: lock wait: 10.81 lock held: 1.24 filesystem: 0.51 buffersearch: 0.56
LOG:  extension time: lock wait: 10.62 lock held: 1.27 filesystem: 0.53 buffersearch: 0.56
LOG:  extension time: lock wait: 11.14 lock held: 0.94 filesystem: 0.38 buffersearch: 0.41
LOG:  extension time: lock wait: 11.20 lock held: 0.96 filesystem: 0.39 buffersearch: 0.42
LOG:  extension time: lock wait: 10.75 lock held: 1.41 filesystem: 0.58 buffersearch: 0.63
0.88 + 0.90 + 1.01 + 1.25 + 0.94 + 1.27 + 1.19 + 1.22 + 1.17 + 1.32 + 0.92 + 1.24 + 1.27 + 0.94 + 0.96 + 1.41
in *none* of these cases the drive gets even close to being
saturated. Like not even 1/3.

If you consider the total time with the lock held, and the total time of
the test, it becomes very quickly obvious that pretty quickly we spend
the majority of the total time with the lock held.
client count 1: 3.18/20 = 0.16
client count 2: 6.04/20 = 0.30
client count 4: 9.95/20 = 0.50
client count 8: 15.30/20 = 0.76
client count 16: 17.89/20 = 0.89

In other words, the reason that relation extension scales terribly
isn't, to a significant degree, because the locking is slow. It's
because we hold locks for the majority of the benchmark's time starting
even at just 4 clients.  Focusing on making the locking faster is just
optimizing for the wrong thing.  Amdahl's law will just restrict the
benefits to a pretty small amount.

Looking at a CPU time profile (i.e. it'll not include the time spent
waiting for a lock, once sleeping in the kernel) for time spent within
RelationGetBufferForTuple():

-   19.16%     0.29%  postgres  postgres            [.] RelationGetBufferForTuple
   - 18.88% RelationGetBufferForTuple
      - 13.18% ReadBufferExtended
         - ReadBuffer_common
            + 5.02% mdextend
            + 4.77% FlushBuffer.part.0
            + 0.61% BufTableLookup
              0.52% __memset_avx2_erms
      + 1.65% PageInit
      - 1.18% LockRelationForExtension
         - 1.16% LockAcquireExtended
            - 1.07% WaitOnLock
               - 1.01% ProcSleep
                  - 0.88% WaitLatchOrSocket
                       0.52% WaitEventSetWait
        0.65% RecordAndGetPageWithFreeSpace

the same workload using an assert enabled build, to get a simpler to
interpret profile:
-   13.28%     0.19%  postgres  postgres            [.] RelationGetBufferForTuple
   - 13.09% RelationGetBufferForTuple
      - 8.35% RelationAddExtraBlocks
         - 7.67% ReadBufferBI
            - 7.54% ReadBufferExtended
               - 7.52% ReadBuffer_common
                  - 3.64% BufferAlloc
                     + 2.39% FlushBuffer
                     + 0.27% BufTableLookup
                     + 0.24% BufTableDelete
                     + 0.15% LWLockAcquire
                       0.14% StrategyGetBuffer
                     + 0.13% BufTableHashCode
                  - 2.96% smgrextend
                     + mdextend
                  + 0.52% __memset_avx2_erms
                  + 0.14% smgrnblocks
                    0.11% __GI___clock_gettime (inlined)
         + 0.57% RecordPageWithFreeSpace
      - 1.23% RecordAndGetPageWithFreeSpace
         - 1.03% fsm_set_and_search
            + 0.50% fsm_readbuf
            + 0.20% LockBuffer
            + 0.18% UnlockReleaseBuffer
              0.11% fsm_set_avail
           0.19% fsm_search
      - 0.86% ReadBufferBI
         - 0.72% ReadBufferExtended
            - ReadBuffer_common
               - 0.58% BufferAlloc
                  + 0.20% BufTableLookup
                  + 0.10% LWLockAcquire
      + 0.81% PageInit
      - 0.67% LockRelationForExtension
         - 0.67% LockAcquire
            - LockAcquireExtended
               + 0.60% WaitOnLock

Which, I think, pretty clearly shows a few things:

1) It's crucial to move acquiring a victim buffer to the outside of the
   extension lock, as for copy acquiring the victim buffer will commonly
   cause a buffer having to be written out, due to the ringbuffer. This
   is even more crucial when using a logged table, as the writeout then
   also will often also trigger a WAL flush.

   While doing so will sometimes add a round of acquiring the buffer
   mapping locks, having to do the FlushBuffer while holding the
   extension lock is a huge problem.

   This'd also move a good bit of the cost of finding (i.e. clock sweep
   / ringbuffer replacement) and invalidating the old buffer mapping out
   of the lock.

2) We need to make the smgrwrite more efficient, it is costing a lot of
   time. A small additional experiment shows the cost of doing 8kb
   writes:

   I wrote a small program that just iteratively writes a 32GB file:

   pwrite using 8kb blocks:
   0.24user 17.88system 0:18.16 elapsed 99%CPU

   pwrite using 128kb blocks:
   0.00user 16.71system 0:17.01 elapsed 98%CPU

   pwrite using 256kb blocks:
   0.00user 15.95system 0:16.03 elapsed 99%CPU

   pwritev() using 16 8kb blocks to write 128kb at once:
   0.02user 15.94system 0:16.09 elapsed 99%CPU

   pwritev() using 32 8kb blocks to write 256kb at once:
   0.01user 14.90system 0:14.93 elapsed 99%CPU

   pwritev() using 128 8kb blocks to write 1MB at once:
   0.00user 13.96system 0:13.96 elapsed 99%CPU

   if I instead just use posix_fallocate() with 8kb blocks:
   0.28user 23.49system 0:23.78elapsed 99%CPU (0avgtext+0avgdata 1212maxresident)k
   0inputs+0outputs (0major+66minor)pagefaults 0swaps

   if I instead just use posix_fallocate() with 32 8kb blocks:
   0.01user 1.18system 0:01.19elapsed 99%CPU (0avgtext+0avgdata 1200maxresident)k
   0inputs+0outputs (0major+67minor)pagefaults 0swaps

   obviously fallocate doesn't quite have the same behaviour, and may incur
   a bit higher overhead for a later write.

   using a version that instead uses O_DIRECT + async IO, I get (but
   only when also doing posix_fallocate in larger chunks):
   0.05user 5.53system 0:12.53 elapsed 44%CPU

   So we get considerably higher write throughput, at a considerably
   lower CPU usage (because DMA replaces the CPU doing a memcpy()).

   So it looks like extending the file with posix_fallocate() might be a
   winner, but only if we actually can do so in larger chunks than 8kb
   at once.

   Alternatively it could be worthwhile to rejigger things so we don't
   extend the files with zeroes once, just to then immediately overwrite
   it with actual content. For some users it's probably possible to
   pre-generate a page with contents when extending the file (would need
   fiddling with block numbers etc).

3) We should move the PageInit() that's currently done with the
   extension lock held, to the outside. Since we get the buffer with
   RBM_ZERO_AND_LOCK these days, that should be safe.  Also, we don't
   need to zero the entire buffer both in RelationGetBufferForTuple()'s
   PageInit(), and in ReadBuffer_common() before calling smgrextend().

Greetings,

Andres Freund

[1] https://www.postgresql.org/message-id/20200211042229.msv23badgqljrdg2%40alap3.anarazel.de

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

12 February 2020, 08:10:17

On Tue, 11 Feb 2020 at 11:31, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Feb 5, 2020 at 12:07 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> >
> > Unfortunately the environment I used for performance verification is
> > no longer available.
> >
> > I agree to run this test in a different environment. I've attached the
> > rebased version patch. I'm measuring the performance with/without
> > patch, so will share the results.
> >
>
> Did you get a chance to run these tests?  Lately, Mahendra has done a
> lot of performance testing of this patch and shared his results.  I
> don't see much downside with the patch, rather there is a performance
> increase of 3-9% in various scenarios.

I've done performance tests on my laptop while changing the number of
partitions. 4 clients concurrently insert 32 tuples to randomly
selected partitions in a transaction. Therefore by changing the number
of partition the contention of relation extension lock would also be
changed. All tables are unlogged tables and N_RELEXTLOCK_ENTS is 1024.

Here is my test results:

* HEAD
nchilds = 64 tps = 33135
nchilds = 128 tps = 31249
nchilds = 256 tps = 29356

* Patched
nchilds = 64 tps = 32057
nchilds = 128 tps = 32426
nchilds = 256 tps = 29483

The performance has been slightly improved by the patch in two cases.
I've also attached the shell script I used to test.

When I set N_RELEXTLOCK_ENTS to 1 so that all relation locks conflicts
the result is:

nchilds = 64 tps = 30887
nchilds = 128 tps = 30015
nchilds = 256 tps = 27837

Regards,

-- 
Masahiko Sawada            http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachment

test.sh

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

12 February 2020, 12:16:48

On Wed, Feb 12, 2020 at 7:36 AM Masahiko Sawada
<masahiko.sawada@2ndquadrant.com> wrote:
>
> On Wed, 12 Feb 2020 at 00:43, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >
> > I took a brief look through this patch.  I agree with the fundamental
> > idea that we shouldn't need to use the heavyweight lock manager for
> > relation extension, since deadlock is not a concern and no backend
> > should ever need to hold more than one such lock at once.  But it feels
> > to me like this particular solution is rather seriously overengineered.
> > I would like to suggest that we do something similar to Robert Haas'
> > excellent hack (daa7527af) for the !HAVE_SPINLOCK case in lmgr/spin.c,
> > that is,
> >
> > * Create some predetermined number N of LWLocks for relation extension.
>
> My original proposal used LWLocks and hash tables for relation
> extension but there was a discussion that using LWLocks is not good
> because it's not interruptible[1]. Because of this reason and that we
> don't need to have two lock level (shared, exclusive) for relation
> extension lock we ended up with implementing dedicated lock manager
> for extension lock. I think we will have that problem if we use LWLocks.
>

Hmm, but we use LWLocks for (a) WALWrite/Flush (see the usage of
WALWriteLock), (b) writing the shared buffer contents (see
io_in_progress lock and its usage in FlushBuffer) and might be for few
other similar stuff.  Many times those take more time than extending a
block in relation especially when we combine the WAL write for
multiple commits.  So, if this is a problem for relation extension
lock, then the same thing holds true there also.  Now, there are cases
like when we extend the relation with multiple blocks, finding victim
buffer under this lock, etc. where this can be also equally or more
costly, but I think we can improve some of those cases (some of this
is even pointed by Andres in his email) if we agree on a fundamental
idea of using LWLocks as proposed by Tom.   I am not telling that we
implement Tom's idea without weighing its pros and cons, but it has an
appeal due to its simplicity.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

12 February 2020, 13:22:26

On Wed, Feb 12, 2020 at 10:24 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2020-02-11 08:01:34 +0530, Amit Kapila wrote:
> > I don't see much downside with the patch, rather there is a
> > performance increase of 3-9% in various scenarios.
>
> As I wrote in [1] I started to look at this patch. My problem with itis
> that it just seems like the wrong direction architecturally to
> me. There's two main aspects to this:
>
> 1) It basically builds a another, more lightweight but less capable, of
>    a lock manager that can lock more objects than we can have distinct
>    locks for.  It is faster because it uses *one* hashtable without
>    conflict handling, because it has fewer lock modes, and because it
>    doesn't support detecting deadlocks. And probably some other things.
>
> 2) A lot of the contention around file extension comes from us doing
>    multiple expensive things under one lock (determining current
>    relation size, searching victim buffer, extending file), and in tiny
>    increments (growing a 1TB table by 8kb). This patch doesn't address
>    that at all.
>

It seems to me both the two points try to address the performance
angle of the patch, but here our actual intention was to make this
lock block among parallel workers so that we can implement/improve
some of the parallel writes operations (like parallelly vacuuming the
heap or index, parallel bulk load, etc.).  Both independently are
worth accomplishing, but not w.r.t parallel writes.  Here, we were
doing some benchmarking to see if we haven't regressed performance in
any cases.

> I've focused on 1) in the email referenced above ([1]). Here I'll focus
> on 2).
>
>
>
> Which, I think, pretty clearly shows a few things:
>

I agree with all your below observations.

> 1) It's crucial to move acquiring a victim buffer to the outside of the
>    extension lock, as for copy acquiring the victim buffer will commonly
>    cause a buffer having to be written out, due to the ringbuffer. This
>    is even more crucial when using a logged table, as the writeout then
>    also will often also trigger a WAL flush.
>
>    While doing so will sometimes add a round of acquiring the buffer
>    mapping locks, having to do the FlushBuffer while holding the
>    extension lock is a huge problem.
>
>    This'd also move a good bit of the cost of finding (i.e. clock sweep
>    / ringbuffer replacement) and invalidating the old buffer mapping out
>    of the lock.
>

I think this mostly because of the way currently code is arranged to
extend a block via ReadBuffer* API.  IIUC, currently the main
operations under relation extension lock are as follows:
a. get the block number for extension via smgrnblocks.
b. find victim buffer
c. associate buffer with the block no. found in step-a.
d. initialize the block with zeros
e. write the block
f.  PageInit

I think if we can rearrange such that steps b and c can be done after
e or f, then we don't need to hold the extension lock to find the
victim buffer.

> 2) We need to make the smgrwrite more efficient, it is costing a lot of
>    time. A small additional experiment shows the cost of doing 8kb
>    writes:
>
>    I wrote a small program that just iteratively writes a 32GB file:
>
..
>
>
>    So it looks like extending the file with posix_fallocate() might be a
>    winner, but only if we actually can do so in larger chunks than 8kb
>    at once.
>

A good experiment and sounds like worth doing.

>
>
> 3) We should move the PageInit() that's currently done with the
>    extension lock held, to the outside. Since we get the buffer with
>    RBM_ZERO_AND_LOCK these days, that should be safe.  Also, we don't
>    need to zero the entire buffer both in RelationGetBufferForTuple()'s
>    PageInit(), and in ReadBuffer_common() before calling smgrextend().
>

Agreed.

I feel all three are independent improvements and can be done separately.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Tom Lane

Date:

12 February 2020, 19:53:49

Amit Kapila <amit.kapila16@gmail.com> writes:
> On Wed, Feb 12, 2020 at 7:36 AM Masahiko Sawada
> <masahiko.sawada@2ndquadrant.com> wrote:
>> On Wed, 12 Feb 2020 at 00:43, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> I would like to suggest that we do something similar to Robert Haas'
>>> excellent hack (daa7527af) for the !HAVE_SPINLOCK case in lmgr/spin.c,

>> My original proposal used LWLocks and hash tables for relation
>> extension but there was a discussion that using LWLocks is not good
>> because it's not interruptible[1].

> Hmm, but we use LWLocks for (a) WALWrite/Flush (see the usage of
> WALWriteLock), (b) writing the shared buffer contents (see
> io_in_progress lock and its usage in FlushBuffer) and might be for few
> other similar stuff.  Many times those take more time than extending a
> block in relation especially when we combine the WAL write for
> multiple commits.  So, if this is a problem for relation extension
> lock, then the same thing holds true there also.

Yeah.  I would say a couple more things:

* I see no reason to think that a relation extension lock would ever
be held long enough for noninterruptibility to be a real issue.  Our
expectations for query cancel response time are in the tens to
hundreds of msec anyway.

* There are other places where an LWLock can be held for a *long* time,
notably the CheckpointLock.  If we do think this is an issue, we could
devise a way to not insist on noninterruptibility.  The easiest fix
is just to do a matching RESUME_INTERRUPTS after getting the lock and
HOLD_INTERRUPTS again before releasing it; though maybe it'd be worth
offering some slightly cleaner way.  Point here is that LWLockAcquire
only does that because it's useful to the majority of callers, not
because it's graven in stone that it must be like that.

In general, if we think there are issues with LWLock, it seems to me
we'd be better off to try to fix them, not to invent a whole new
single-purpose lock manager that we'll have to debug and maintain.
I do not see anything about this problem that suggests that that would
provide a major win.  As Andres has noted, there are lots of other
aspects of it that are likely to be more useful to spend effort on.

            regards, tom lane

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

13 February 2020, 06:03:42

On Wed, Feb 12, 2020 at 10:23 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Amit Kapila <amit.kapila16@gmail.com> writes:
> > On Wed, Feb 12, 2020 at 7:36 AM Masahiko Sawada
> > <masahiko.sawada@2ndquadrant.com> wrote:
> >> On Wed, 12 Feb 2020 at 00:43, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >>> I would like to suggest that we do something similar to Robert Haas'
> >>> excellent hack (daa7527af) for the !HAVE_SPINLOCK case in lmgr/spin.c,
>
> >> My original proposal used LWLocks and hash tables for relation
> >> extension but there was a discussion that using LWLocks is not good
> >> because it's not interruptible[1].
>
> > Hmm, but we use LWLocks for (a) WALWrite/Flush (see the usage of
> > WALWriteLock), (b) writing the shared buffer contents (see
> > io_in_progress lock and its usage in FlushBuffer) and might be for few
> > other similar stuff.  Many times those take more time than extending a
> > block in relation especially when we combine the WAL write for
> > multiple commits.  So, if this is a problem for relation extension
> > lock, then the same thing holds true there also.
>
> Yeah.  I would say a couple more things:
>
> * I see no reason to think that a relation extension lock would ever
> be held long enough for noninterruptibility to be a real issue.  Our
> expectations for query cancel response time are in the tens to
> hundreds of msec anyway.
>
> * There are other places where an LWLock can be held for a *long* time,
> notably the CheckpointLock.  If we do think this is an issue, we could
> devise a way to not insist on noninterruptibility.  The easiest fix
> is just to do a matching RESUME_INTERRUPTS after getting the lock and
> HOLD_INTERRUPTS again before releasing it; though maybe it'd be worth
> offering some slightly cleaner way.
>

Yeah, this sounds like the better answer for noninterruptibility
aspect of this design.  One idea that occurred to me was to pass a
parameter to LWLOCK acquire/release APIs to indicate whether to
hold/resume interrupts, but I don't know if that is any better than
doing it at the required place.  I am not sure if all places are
careful whether they really want to hold interrupts, so if we provide
a new parameter at least new users of API will think about it.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

13 February 2020, 07:16:25

On Tue, Feb 11, 2020 at 9:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> I took a brief look through this patch.  I agree with the fundamental
> idea that we shouldn't need to use the heavyweight lock manager for
> relation extension, since deadlock is not a concern and no backend
> should ever need to hold more than one such lock at once.  But it feels
> to me like this particular solution is rather seriously overengineered.
> I would like to suggest that we do something similar to Robert Haas'
> excellent hack (daa7527af) for the !HAVE_SPINLOCK case in lmgr/spin.c,
> that is,
>
> * Create some predetermined number N of LWLocks for relation extension.
> * When we want to extend some relation R, choose one of those locks
>   (say, R's relfilenode number mod N) and lock it.
>

I am imagining something on the lines of BufferIOLWLockArray (here it
will be RelExtLWLockArray).  The size (N) could MaxBackends or some
percentage of it (depending on testing) and indexing into an array
could be as suggested (R's relfilenode number mod N).  We need to
initialize this during shared memory initialization.  Then, to extend
the relation with multiple blocks at-a-time (as we do in
RelationAddExtraBlocks), we can either use the already proven
technique of group clear xid mechanism (see ProcArrayGroupClearXid) or
have an additional state in the RelExtLWLockArray which will keep the
count of waiters (as done in latest patch of Sawada-san [1]).  We
might want to experiment with both approaches and see which yields
better results.

[1] - https://www.postgresql.org/message-id/CAD21AoADkWhkLEB_%3DkjLZeZ_ML9_hSQqNBWz%2Bd821QHf%3DO9LJQ%40mail.gmail.com
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dilip Kumar

Date:

13 February 2020, 07:52:33

On Thu, Feb 13, 2020 at 9:46 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Feb 11, 2020 at 9:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >
> > I took a brief look through this patch.  I agree with the fundamental
> > idea that we shouldn't need to use the heavyweight lock manager for
> > relation extension, since deadlock is not a concern and no backend
> > should ever need to hold more than one such lock at once.  But it feels
> > to me like this particular solution is rather seriously overengineered.
> > I would like to suggest that we do something similar to Robert Haas'
> > excellent hack (daa7527af) for the !HAVE_SPINLOCK case in lmgr/spin.c,
> > that is,
> >
> > * Create some predetermined number N of LWLocks for relation extension.
> > * When we want to extend some relation R, choose one of those locks
> >   (say, R's relfilenode number mod N) and lock it.
> >
>
> I am imagining something on the lines of BufferIOLWLockArray (here it
> will be RelExtLWLockArray).  The size (N) could MaxBackends or some
> percentage of it (depending on testing) and indexing into an array
> could be as suggested (R's relfilenode number mod N).  We need to
> initialize this during shared memory initialization.  Then, to extend
> the relation with multiple blocks at-a-time (as we do in
> RelationAddExtraBlocks), we can either use the already proven
> technique of group clear xid mechanism (see ProcArrayGroupClearXid) or
> have an additional state in the RelExtLWLockArray which will keep the
> count of waiters (as done in latest patch of Sawada-san [1]).  We
> might want to experiment with both approaches and see which yields
> better results.

IMHO, in this case, there is no point in using the "group clear" type
of mechanism mainly for two reasons 1) It will unnecessarily make
PGPROC structure heavy.
2) For our case, we don't need any specific pieces of information from
other waiters, we just need the count.


Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Mahendra Singh Thalor

Date:

13 February 2020, 09:16:54

On Thu, 13 Feb 2020 at 09:46, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Feb 11, 2020 at 9:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >
> > I took a brief look through this patch.  I agree with the fundamental
> > idea that we shouldn't need to use the heavyweight lock manager for
> > relation extension, since deadlock is not a concern and no backend
> > should ever need to hold more than one such lock at once.  But it feels
> > to me like this particular solution is rather seriously overengineered.
> > I would like to suggest that we do something similar to Robert Haas'
> > excellent hack (daa7527af) for the !HAVE_SPINLOCK case in lmgr/spin.c,
> > that is,
> >
> > * Create some predetermined number N of LWLocks for relation extension.
> > * When we want to extend some relation R, choose one of those locks
> >   (say, R's relfilenode number mod N) and lock it.
> >
>
> I am imagining something on the lines of BufferIOLWLockArray (here it
> will be RelExtLWLockArray).  The size (N) could MaxBackends or some
> percentage of it (depending on testing) and indexing into an array
> could be as suggested (R's relfilenode number mod N).  We need to
> initialize this during shared memory initialization.  Then, to extend
> the relation with multiple blocks at-a-time (as we do in
> RelationAddExtraBlocks), we can either use the already proven
> technique of group clear xid mechanism (see ProcArrayGroupClearXid) or
> have an additional state in the RelExtLWLockArray which will keep the
> count of waiters (as done in latest patch of Sawada-san [1]).  We
> might want to experiment with both approaches and see which yields
> better results.

Thanks all for the suggestions. I have started working on the
implementation based on the suggestion.  I will post a patch for this
in few days.

-- 
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

14 February 2020, 09:12:18

On Thu, 13 Feb 2020 at 13:16, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Feb 11, 2020 at 9:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >
> > I took a brief look through this patch.  I agree with the fundamental
> > idea that we shouldn't need to use the heavyweight lock manager for
> > relation extension, since deadlock is not a concern and no backend
> > should ever need to hold more than one such lock at once.  But it feels
> > to me like this particular solution is rather seriously overengineered.
> > I would like to suggest that we do something similar to Robert Haas'
> > excellent hack (daa7527af) for the !HAVE_SPINLOCK case in lmgr/spin.c,
> > that is,
> >
> > * Create some predetermined number N of LWLocks for relation extension.
> > * When we want to extend some relation R, choose one of those locks
> >   (say, R's relfilenode number mod N) and lock it.
> >
>
> I am imagining something on the lines of BufferIOLWLockArray (here it
> will be RelExtLWLockArray).  The size (N) could MaxBackends or some
> percentage of it (depending on testing) and indexing into an array
> could be as suggested (R's relfilenode number mod N).

I'm not sure it's good that the contention of LWLock slot depends on
MaxBackends. Because it means that the more MaxBackends is larger, the
less the LWLock slot conflicts, even if the same number of backends
actually connecting. Normally we don't want to increase unnecessarily
MaxBackends for security reasons. In the current patch we defined a
fixed length of array for extension lock but I agree that we need to
determine what approach is the best depending on testing.

Regards,

-- 
Masahiko Sawada            http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

14 February 2020, 11:20:02

On Fri, Feb 14, 2020 at 11:42 AM Masahiko Sawada
<masahiko.sawada@2ndquadrant.com> wrote:
>
> On Thu, 13 Feb 2020 at 13:16, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Feb 11, 2020 at 9:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > >
> > > I took a brief look through this patch.  I agree with the fundamental
> > > idea that we shouldn't need to use the heavyweight lock manager for
> > > relation extension, since deadlock is not a concern and no backend
> > > should ever need to hold more than one such lock at once.  But it feels
> > > to me like this particular solution is rather seriously overengineered.
> > > I would like to suggest that we do something similar to Robert Haas'
> > > excellent hack (daa7527af) for the !HAVE_SPINLOCK case in lmgr/spin.c,
> > > that is,
> > >
> > > * Create some predetermined number N of LWLocks for relation extension.
> > > * When we want to extend some relation R, choose one of those locks
> > >   (say, R's relfilenode number mod N) and lock it.
> > >
> >
> > I am imagining something on the lines of BufferIOLWLockArray (here it
> > will be RelExtLWLockArray).  The size (N) could MaxBackends or some
> > percentage of it (depending on testing) and indexing into an array
> > could be as suggested (R's relfilenode number mod N).
>
> I'm not sure it's good that the contention of LWLock slot depends on
> MaxBackends. Because it means that the more MaxBackends is larger, the
> less the LWLock slot conflicts, even if the same number of backends
> actually connecting. Normally we don't want to increase unnecessarily
> MaxBackends for security reasons. In the current patch we defined a
> fixed length of array for extension lock but I agree that we need to
> determine what approach is the best depending on testing.
>

I think MaxBackends will generally limit the number of different
relations that can simultaneously extend, but maybe tables with many
partitions might change the situation.  You are right that some tests
might suggest a good number, let Mahendra write a patch and then we
can test it.  Do you have any better idea?


-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Tom Lane

Date:

14 February 2020, 17:42:40

Amit Kapila <amit.kapila16@gmail.com> writes:
> I think MaxBackends will generally limit the number of different
> relations that can simultaneously extend, but maybe tables with many
> partitions might change the situation.  You are right that some tests
> might suggest a good number, let Mahendra write a patch and then we
> can test it.  Do you have any better idea?

In the first place, there certainly isn't more than one extension
happening at a time per backend, else the entire premise of this
thread is wrong.  Handwaving about partitions won't change that.

In the second place, it's ludicrous to expect that the underlying
platform/filesystem can support an infinite number of concurrent
file-extension operations.  At some level (e.g. where disk blocks
are handed out, or where a record of the operation is written to
a filesystem journal) it's quite likely that things are bottlenecked
down to *one* such operation at a time per filesystem.  So I'm not
that concerned about occasional false-sharing limiting our ability
to issue concurrent requests.  There are probably worse restrictions
at lower levels.

            regards, tom lane

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

14 February 2020, 18:28:29

On Wed, Feb 12, 2020 at 11:53 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Yeah.  I would say a couple more things:
>
> * I see no reason to think that a relation extension lock would ever
> be held long enough for noninterruptibility to be a real issue.  Our
> expectations for query cancel response time are in the tens to
> hundreds of msec anyway.

I don't agree, because (1) the time to perform a relation extension on
a busy system can be far longer than that and (2) if the disk is
failing, then it can be *really* long, or indefinite.

> * There are other places where an LWLock can be held for a *long* time,
> notably the CheckpointLock.  If we do think this is an issue, we could
> devise a way to not insist on noninterruptibility.  The easiest fix
> is just to do a matching RESUME_INTERRUPTS after getting the lock and
> HOLD_INTERRUPTS again before releasing it; though maybe it'd be worth
> offering some slightly cleaner way.  Point here is that LWLockAcquire
> only does that because it's useful to the majority of callers, not
> because it's graven in stone that it must be like that.

That's an interesting idea, but it doesn't make the lock acquisition
itself interruptible, which seems pretty important to me in this case.
I wonder if we could have an LWLockAcquireInterruptibly() or some such
that allows the lock acquisition itself to be interruptible. I think
that would require some rejiggering but it might be doable.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Tom Lane

Date:

14 February 2020, 18:43:35

Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Feb 12, 2020 at 11:53 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> * I see no reason to think that a relation extension lock would ever
>> be held long enough for noninterruptibility to be a real issue.  Our
>> expectations for query cancel response time are in the tens to
>> hundreds of msec anyway.

> I don't agree, because (1) the time to perform a relation extension on
> a busy system can be far longer than that and (2) if the disk is
> failing, then it can be *really* long, or indefinite.

I remain unconvinced ... wouldn't both of those claims apply to any disk
I/O request?  Are we going to try to ensure that no I/O ever happens
while holding an LWLock, and if so how?  (Again, CheckpointLock is a
counterexample, which has been that way for decades without reported
problems.  But actually I think buffer I/O locks are an even more
direct counterexample.)

>> * There are other places where an LWLock can be held for a *long* time,
>> notably the CheckpointLock.  If we do think this is an issue, we could
>> devise a way to not insist on noninterruptibility.  The easiest fix
>> is just to do a matching RESUME_INTERRUPTS after getting the lock and
>> HOLD_INTERRUPTS again before releasing it; though maybe it'd be worth
>> offering some slightly cleaner way.  Point here is that LWLockAcquire
>> only does that because it's useful to the majority of callers, not
>> because it's graven in stone that it must be like that.

> That's an interesting idea, but it doesn't make the lock acquisition
> itself interruptible, which seems pretty important to me in this case.

Good point: if you think the contained operation might run too long to
suit you, then you don't want other backends to be stuck behind it for
the same amount of time.

> I wonder if we could have an LWLockAcquireInterruptibly() or some such
> that allows the lock acquisition itself to be interruptible. I think
> that would require some rejiggering but it might be doable.

Yeah, I had the impression from a brief look at LWLockAcquire that
it was itself depending on not throwing errors partway through.
But with careful and perhaps-a-shade-slower coding, we could probably
make a version that didn't require that.

            regards, tom lane

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

14 February 2020, 19:20:15

On Fri, Feb 14, 2020 at 10:43 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I remain unconvinced ... wouldn't both of those claims apply to any disk
> I/O request?  Are we going to try to ensure that no I/O ever happens
> while holding an LWLock, and if so how?  (Again, CheckpointLock is a
> counterexample, which has been that way for decades without reported
> problems.  But actually I think buffer I/O locks are an even more
> direct counterexample.)

Yes, that's a problem. I proposed a patch a few years ago that
replaced the buffer I/O locks with condition variables, and I think
that's a good idea for lots of reasons, including this one. I never
quite got around to pushing that through to commit, but I think we
should do that. Aside from fixing this problem, it also prevents
certain scenarios where we can currently busy-loop.

I do realize that we're unlikely to ever solve this problem
completely, but I don't think that should discourage us from making
incremental progress. Just as debuggability is a sticking point for
you, what I'm going to call operate-ability is a sticking point for
me. My work here at EnterpriseDB exposes me on a fairly regular basis
to real broken systems, and I'm therefore really sensitive to the
concerns that people have when trying to recover after a system has
become, for one reason or another, really broken.

Interruptibility may not be the #1 concern in that area, but it's very
high on the list. EnterpriseDB customers, as a rule, *really* hate
being told to restart the database because one session is stuck. It
causes a lot of disruption for them and the person who does the
restart gets yelled at by their boss, and maybe their bosses boss and
the boss above that. It means that their whole application, which may
be mission-critical, is down until the database finishes restarting,
and that is not always a quick process, especially after an immediate
shutdown. I don't think we can ever make everything that can get stuck
interruptible, but the more we can do the better.

The work you and others have done over the years to add
CHECK_FOR_INTERRUPTS() to more places pays real dividends. Making
sessions that are blocked on disk I/O interruptible in at least some
of the more common cases would be a huge win. Other people may well
have different experiences, but my experience is that the disk
deciding to conk out for a while or just respond very very slowly is a
very common problem even (and sometimes especially) on very expensive
hardware. Obviously that's not great and you're in lots of trouble,
but being able to hit ^C and get control back significantly improves
your chances of being able to understand what has happened and recover
from it.

> > That's an interesting idea, but it doesn't make the lock acquisition
> > itself interruptible, which seems pretty important to me in this case.
>
> Good point: if you think the contained operation might run too long to
> suit you, then you don't want other backends to be stuck behind it for
> the same amount of time.

Right.

> > I wonder if we could have an LWLockAcquireInterruptibly() or some such
> > that allows the lock acquisition itself to be interruptible. I think
> > that would require some rejiggering but it might be doable.
>
> Yeah, I had the impression from a brief look at LWLockAcquire that
> it was itself depending on not throwing errors partway through.
> But with careful and perhaps-a-shade-slower coding, we could probably
> make a version that didn't require that.

Yeah, that was my thought, too, but I didn't study it that carefully,
so somebody would need to do that.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

14 February 2020, 19:40:20

Hi,

On 2020-02-12 11:53:49 -0500, Tom Lane wrote:
> In general, if we think there are issues with LWLock, it seems to me
> we'd be better off to try to fix them, not to invent a whole new
> single-purpose lock manager that we'll have to debug and maintain.

My impression is that what's being discussed here is doing exactly that,
except with s/lwlock/heavyweight locks/. We're basically replacing the
lock.c lock mapping table with an ad-hoc implementation, and now we're
also reinventing interruptability etc.

I still find the performance arguments pretty ludicruous, to be honest -
I think the numbers I posted about how much time we spend with the locks
held, back that up.  I have a bit more understanding for the parallel
worker arguments, but only a bit:

I think if we develop a custom solution for the extension lock, we're
just going to end up having to develop another custom solution for a
bunch of other types of locks.  It seems quite likely that we'll end up
also wanting TUPLE and also SPECULATIVE and PAGE type locks that we
don't want to share between leader & workers.

IMO the right thing here is to extend lock.c so we can better represent
whether certain types of lockmethods (& levels ?) are [not] to be
shared.

Greetings,

Andres Freund

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

14 February 2020, 19:47:18

Hi,

On 2020-02-14 09:42:40 -0500, Tom Lane wrote:
> In the second place, it's ludicrous to expect that the underlying
> platform/filesystem can support an infinite number of concurrent
> file-extension operations.  At some level (e.g. where disk blocks
> are handed out, or where a record of the operation is written to
> a filesystem journal) it's quite likely that things are bottlenecked
> down to *one* such operation at a time per filesystem.

That's probably true to some degree from a theoretical POV, but I think
it's so far from where we are at, that it's effectively wrong. I can
concurrently extend a few files at close to 10GB/s on a set of fast
devices below a *single* filesystem. Whereas postgres bottlenecks far
far before this.  Given that a lot of today's storage has latencies in
the 10-100s of microseconds, a journal flush doesn't necessarily cause
that much serialization - and OS journals do group commit like things
too.

Greetings,

Andres Freund

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

14 February 2020, 20:08:45

On Fri, Feb 14, 2020 at 11:40 AM Andres Freund <andres@anarazel.de> wrote:
> IMO the right thing here is to extend lock.c so we can better represent
> whether certain types of lockmethods (& levels ?) are [not] to be
> shared.

The part that I find awkward about that is the whole thing with the
deadlock detector. The deadlock detection code is old, crufty,
complex, and very difficult to test (or at least I have found it so).
A bug that I introduced when inventing group locking took like 5 years
for somebody to find.

One way of looking at the requirement that we have here is that
certain kinds of locks need to be exempted from group locking.
Basically, these are because they are a lower-level concept: a lock on
a relation is more of a "logical" concept, and you hold the lock until
eoxact, whereas a lock on an extend the relation is more of a
"physical" concept, and you give it up as soon as you are done. Page
locks are like relation extension locks in this regard. Unlike locks
on SQL-level objects, these should not be shared between members of a
lock group.

Now, if it weren't for the deadlock detector, that would be easy
enough. But figuring out what to do with the deadlock detector seems
really painful to me. I wonder if there's some way we can make an end
run around that problem. For instance, if we could make (and enforce)
a coding rule that you cannot acquire a heavyweight lock while holding
a relation extension or page lock, then maybe we could somehow teach
the deadlock detector to just ignore those kinds of locks, and teach
the lock acquisition machinery that they conflict between lock group
members.

On the other hand, I think you might also be understating the
differences between these kinds of locks and other heavyweight locks.
I suspect that the reason why we use lwlocks for buffers and
heavyweight locks here is because there are a conceptually infinite
number of relations, and lwlocks can't handle that. The only mechanism
we currently have that does handle that is the heavyweight lock
mechanism, and from my point of view, somebody just beat it with a
stick to make it fit this application. But the fact that it has been
made to fit does not mean that it is really fit for purpose. We use 2
of 9 lock levels, we don't need deadlock detection, we need different
behavior when group locking is in use, we release locks right away
rather than at eoxact. I don't think it's crazy to think that those
differences are significant enough to justify having a separate
mechanism, even if the one that is currently on the table is not
exactly what we want.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

14 February 2020, 21:07:34

Hi,

On 2020-02-14 12:08:45 -0500, Robert Haas wrote:
> On Fri, Feb 14, 2020 at 11:40 AM Andres Freund <andres@anarazel.de> wrote:
> > IMO the right thing here is to extend lock.c so we can better represent
> > whether certain types of lockmethods (& levels ?) are [not] to be
> > shared.
> 
> The part that I find awkward about that is the whole thing with the
> deadlock detector. The deadlock detection code is old, crufty,
> complex, and very difficult to test (or at least I have found it so).
> A bug that I introduced when inventing group locking took like 5 years
> for somebody to find.

Oh, I agree, lock.c and surrounding code is pretty crufty. Doubtful that
just building up a largely parallel piece of infrastructure next to it
is a good answer though.

> One way of looking at the requirement that we have here is that
> certain kinds of locks need to be exempted from group locking.
> Basically, these are because they are a lower-level concept: a lock on
> a relation is more of a "logical" concept, and you hold the lock until
> eoxact, whereas a lock on an extend the relation is more of a
> "physical" concept, and you give it up as soon as you are done. Page
> locks are like relation extension locks in this regard. Unlike locks
> on SQL-level objects, these should not be shared between members of a
> lock group.
> 
> Now, if it weren't for the deadlock detector, that would be easy
> enough. But figuring out what to do with the deadlock detector seems
> really painful to me. I wonder if there's some way we can make an end
> run around that problem. For instance, if we could make (and enforce)
> a coding rule that you cannot acquire a heavyweight lock while holding
> a relation extension or page lock, then maybe we could somehow teach
> the deadlock detector to just ignore those kinds of locks, and teach
> the lock acquisition machinery that they conflict between lock group
> members.

Yea, that seems possible.  I'm not really sure it's needed however? As
long as you're not teaching the locking mechanism new tricks that
influence the wait graph, why would the deadlock detector care? That's
quite different from the group locking case, where you explicitly needed
to teach it something fairly fundamental.

It might still be a good idea independently to add the rule & enforce
that acquire heavyweight locks while holding certain classes of locks is
not allowed.

> On the other hand, I think you might also be understating the
> differences between these kinds of locks and other heavyweight locks.
> I suspect that the reason why we use lwlocks for buffers and
> heavyweight locks here is because there are a conceptually infinite
> number of relations, and lwlocks can't handle that.

Right. For me that's *the* fundamental service that lock.c delivers. And
it's the fundamental bit this thread so far largely has been focusing
on.

> The only mechanism we currently have that does handle that is the
> heavyweight lock mechanism, and from my point of view, somebody just
> beat it with a stick to make it fit this application. But the fact
> that it has been made to fit does not mean that it is really fit for
> purpose. We use 2 of 9 lock levels, we don't need deadlock detection,
> we need different behavior when group locking is in use, we release
> locks right away rather than at eoxact. I don't think it's crazy to
> think that those differences are significant enough to justify having
> a separate mechanism, even if the one that is currently on the table
> is not exactly what we want.

Isn't that mostly true to varying degrees for the majority of lock types
in lock.c? Sure, perhaps historically that's a misuse of lock.c, but
it's been pretty ingrained by now.  I just don't see where leaving out
any of these features is going to give us fundamental advantages
justifying a different locking infrastructure.

E.g. not needing to support "conceptually infinite" number of relations
IMO does provide a fundamental advantage - no need for a mapping.  I'm
not yet seeing anything equivalent for the extension vs. lock.c style
lock case.

Greetings,

Andres Freund

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Robert Haas

Date:

14 February 2020, 21:34:03

On Fri, Feb 14, 2020 at 1:07 PM Andres Freund <andres@anarazel.de> wrote:
> Yea, that seems possible.  I'm not really sure it's needed however? As
> long as you're not teaching the locking mechanism new tricks that
> influence the wait graph, why would the deadlock detector care? That's
> quite different from the group locking case, where you explicitly needed
> to teach it something fairly fundamental.

Well, you have to teach it that locks of certain types conflict even
if they are in the same group, and that bleeds over pretty quickly
into the whole area of deadlock detection, because lock waits are the
edges in the graph that the deadlock detector processes.

> It might still be a good idea independently to add the rule & enforce
> that acquire heavyweight locks while holding certain classes of locks is
> not allowed.

I think that's absolutely essential, if we're going to continue using
the main lock manager for this. I remain somewhat unconvinced that
doing so is the best way forward, but it is *a* way forward.

> Right. For me that's *the* fundamental service that lock.c delivers. And
> it's the fundamental bit this thread so far largely has been focusing
> on.

For me, the deadlock detection is the far more complicated and problematic bit.

> Isn't that mostly true to varying degrees for the majority of lock types
> in lock.c? Sure, perhaps historically that's a misuse of lock.c, but
> it's been pretty ingrained by now.  I just don't see where leaving out
> any of these features is going to give us fundamental advantages
> justifying a different locking infrastructure.

I think the group locking + deadlock detection things are more
fundamental than you might be crediting, but I agree that having
parallel mechanisms has its own set of pitfalls.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

15 February 2020, 05:32:55

On Fri, Feb 14, 2020 at 8:12 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Amit Kapila <amit.kapila16@gmail.com> writes:
> > I think MaxBackends will generally limit the number of different
> > relations that can simultaneously extend, but maybe tables with many
> > partitions might change the situation.  You are right that some tests
> > might suggest a good number, let Mahendra write a patch and then we
> > can test it.  Do you have any better idea?
>
> In the first place, there certainly isn't more than one extension
> happening at a time per backend, else the entire premise of this
> thread is wrong.  Handwaving about partitions won't change that.
>

Having more number of partitions theoretically increases the chances
of false-sharing with the same number of concurrent sessions.  For ex.
two sessions operating on two relations vs. two sessions working on
two relations with 100 partitions each would increase the chances of
false-sharing.  Sawada-San and Mahendra have done many tests on
different systems and some monitoring with the previous patch that
with a decent number of fixed slots (1024), the false-sharing was very
less and even if it was there the effect was close to nothing.  So, in
short, this is not the point to worry about, but to ensure that we
don't create any significant regressions in this area.

> In the second place, it's ludicrous to expect that the underlying
> platform/filesystem can support an infinite number of concurrent
> file-extension operations.  At some level (e.g. where disk blocks
> are handed out, or where a record of the operation is written to
> a filesystem journal) it's quite likely that things are bottlenecked
> down to *one* such operation at a time per filesystem.  So I'm not
> that concerned about occasional false-sharing limiting our ability
> to issue concurrent requests.  There are probably worse restrictions
> at lower levels.
>

Agreed and what we have observed during the tests is what you have
said in this paragraph.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

15 February 2020, 06:08:55

On Fri, Feb 14, 2020 at 9:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Wed, Feb 12, 2020 at 11:53 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> > That's an interesting idea, but it doesn't make the lock acquisition
> > itself interruptible, which seems pretty important to me in this case.
>
> Good point: if you think the contained operation might run too long to
> suit you, then you don't want other backends to be stuck behind it for
> the same amount of time.
>

It is not clear to me why we should add that as a requirement for this
patch when other places like WALWriteLock, etc. have similar coding
patterns and we haven't heard a ton of complaints about making it
interruptable or if there are then I am not aware.

> > I wonder if we could have an LWLockAcquireInterruptibly() or some such
> > that allows the lock acquisition itself to be interruptible. I think
> > that would require some rejiggering but it might be doable.
>
> Yeah, I had the impression from a brief look at LWLockAcquire that
> it was itself depending on not throwing errors partway through.
> But with careful and perhaps-a-shade-slower coding, we could probably
> make a version that didn't require that.
>

If this becomes a requirement to move this patch, then surely we can
do that.  BTW, what exactly we need to ensure for it?  Is it something
on the lines of ensuring that in error path the state of the lock is
cleared?  Are we worried that interrupt handler might do something
which will change the state of lock we are acquiring?

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

17 February 2020, 00:05:31

Hi,

On 2020-02-14 13:34:03 -0500, Robert Haas wrote:
> On Fri, Feb 14, 2020 at 1:07 PM Andres Freund <andres@anarazel.de> wrote:
> > Yea, that seems possible.  I'm not really sure it's needed however? As
> > long as you're not teaching the locking mechanism new tricks that
> > influence the wait graph, why would the deadlock detector care? That's
> > quite different from the group locking case, where you explicitly needed
> > to teach it something fairly fundamental.
> 
> Well, you have to teach it that locks of certain types conflict even
> if they are in the same group, and that bleeds over pretty quickly
> into the whole area of deadlock detection, because lock waits are the
> edges in the graph that the deadlock detector processes.

Shouldn't this *theretically* be doable with changes mostly localized to
lock.c, by not using proc->lockGroupLeader but proc for lock types that
don't support group locking? I do see that deadlock.c largely looks at
->lockGroupLeader, but that kind of doesn't seem right to me.


> > It might still be a good idea independently to add the rule & enforce
> > that acquire heavyweight locks while holding certain classes of locks is
> > not allowed.
> 
> I think that's absolutely essential, if we're going to continue using
> the main lock manager for this. I remain somewhat unconvinced that
> doing so is the best way forward, but it is *a* way forward.

Seems like we should build this part independently of the lock.c/new
infra piece.


> > Right. For me that's *the* fundamental service that lock.c delivers. And
> > it's the fundamental bit this thread so far largely has been focusing
> > on.
> 
> For me, the deadlock detection is the far more complicated and problematic bit.
> 
> > Isn't that mostly true to varying degrees for the majority of lock types
> > in lock.c? Sure, perhaps historically that's a misuse of lock.c, but
> > it's been pretty ingrained by now.  I just don't see where leaving out
> > any of these features is going to give us fundamental advantages
> > justifying a different locking infrastructure.
> 
> I think the group locking + deadlock detection things are more
> fundamental than you might be crediting, but I agree that having
> parallel mechanisms has its own set of pitfalls.

It's possible. But I'm also hesitant to believe that we'll not need
other lock types that conflict between leader/worker, but that still
need deadlock detection.  The more work we want to parallelize, the more
likely that imo will become.

Greetings,

Andres Freund

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Tom Lane

Date:

17 February 2020, 00:12:25

Andres Freund <andres@anarazel.de> writes:
> On 2020-02-14 13:34:03 -0500, Robert Haas wrote:
>> I think the group locking + deadlock detection things are more
>> fundamental than you might be crediting, but I agree that having
>> parallel mechanisms has its own set of pitfalls.

> It's possible. But I'm also hesitant to believe that we'll not need
> other lock types that conflict between leader/worker, but that still
> need deadlock detection.  The more work we want to parallelize, the more
> likely that imo will become.

Yeah.  The concept that leader and workers can't conflict seems to me
to be dependent, in a very fundamental way, on the assumption that
we only need to parallelize read-only workloads.  I don't think that's
going to have a long half-life.

            regards, tom lane

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

19 February 2020, 08:42:18

On Mon, Feb 17, 2020 at 2:42 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Andres Freund <andres@anarazel.de> writes:
> > On 2020-02-14 13:34:03 -0500, Robert Haas wrote:
> >> I think the group locking + deadlock detection things are more
> >> fundamental than you might be crediting, but I agree that having
> >> parallel mechanisms has its own set of pitfalls.
>
> > It's possible. But I'm also hesitant to believe that we'll not need
> > other lock types that conflict between leader/worker, but that still
> > need deadlock detection.  The more work we want to parallelize, the more
> > likely that imo will become.
>
> Yeah.  The concept that leader and workers can't conflict seems to me
> to be dependent, in a very fundamental way, on the assumption that
> we only need to parallelize read-only workloads.  I don't think that's
> going to have a long half-life.
>

Surely, someday, we need to solve that problem.  But it is not clear
when because if we see the operations for which we want to solve the
relation extension lock problem doesn't require that.  For example,
for a parallel copy or further improving parallel vacuum to allow
multiple workers to scan and process the heap and individual index, we
don't need to change anything in group locking as far as I understand.

Now, for parallel deletes/updates, I think it will depend on how we
choose to parallelize those operations.  I mean if we decide that each
worker will work on an independent set of pages like we do for a
sequential scan, we again might not need to change the group locking
unless I am missing something which is possible.

I think till we know the real need for changing group locking, going
in the direction of what Tom suggested to use an array of LWLocks [1]
to address the problems in hand is a good idea.  It is not very clear
to me that are we thinking to give up on Tom's idea [1] and change
group locking even though it is not clear or at least nobody has
proposed an idea/patch which requires that?  Or are we thinking that
we can do what Tom suggested for relation extension lock and also plan
to change group locking for future parallel operations that might
require it?

[1] - https://www.postgresql.org/message-id/19443.1581435793%40sss.pgh.pa.us

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweightlock manager

From

Andres Freund

Date:

20 February 2020, 05:36:12

Hi,

On 2020-02-19 11:12:18 +0530, Amit Kapila wrote:
> I think till we know the real need for changing group locking, going
> in the direction of what Tom suggested to use an array of LWLocks [1]
> to address the problems in hand is a good idea.

-many

I think that building yet another locking subsystem is the entirely
wrong idea - especially when there's imo no convincing architectural
reasons to do so.

> It is not very clear to me that are we thinking to give up on Tom's
> idea [1] and change group locking even though it is not clear or at
> least nobody has proposed an idea/patch which requires that?  Or are
> we thinking that we can do what Tom suggested for relation extension
> lock and also plan to change group locking for future parallel
> operations that might require it?

What I'm advocating is that extension locks should continue to go
through lock.c. And yes, that requires some changes to group locking,
but I still don't see why they'd be complicated. And if there's concerns
about the cost of lock.c, I outlined a pretty long list of improvements
that'll help everyone, and I showed that the locking itself isn't
actually a large fraction of the scalability issues that extension has.

Regards,

Andres

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

24 February 2020, 13:08:47

On Thu, Feb 20, 2020 at 8:06 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2020-02-19 11:12:18 +0530, Amit Kapila wrote:
> > I think till we know the real need for changing group locking, going
> > in the direction of what Tom suggested to use an array of LWLocks [1]
> > to address the problems in hand is a good idea.
>
> -many
>
> I think that building yet another locking subsystem is the entirely
> wrong idea - especially when there's imo no convincing architectural
> reasons to do so.
>

Hmm, AFAIU, it will be done by having an array of LWLocks which we do
at other places as well (like BufferIO locks).  I am not sure if we
can call it as new locking subsystem, but if we decide to continue
using lock.c and change group locking then I think we can do that as
well, see my comments below regarding that.

>
> > It is not very clear to me that are we thinking to give up on Tom's
> > idea [1] and change group locking even though it is not clear or at
> > least nobody has proposed an idea/patch which requires that?  Or are
> > we thinking that we can do what Tom suggested for relation extension
> > lock and also plan to change group locking for future parallel
> > operations that might require it?
>
> What I'm advocating is that extension locks should continue to go
> through lock.c. And yes, that requires some changes to group locking,
> but I still don't see why they'd be complicated.
>

Fair position, as per initial analysis, I think if we do below three
things, it should work out without changing to a new way of locking
for relation extension or page type locks.
a. As per the discussion above, ensure in code we will never try to
acquire another heavy-weight lock after acquiring relation extension
or page type locks (probably by having Asserts in code or maybe some
other way).
b. Change lock.c so that group locking is not considered for these two
lock types. For ex. in LockCheckConflicts, along with the check (if
(proclock->groupLeader == MyProc && MyProc->lockGroupLeader == NULL)),
we also check lock->tag and call it a conflict for these two locks.
c. The deadlock detector can ignore checking these two types of locks
because point (a) ensures that those won't lead to deadlock.  One idea
could be that FindLockCycleRecurseMember just ignores these two types
of locks by checking the lock tag.

It is possible that I might be missing something or we could achieve
this some other way as well.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Mahendra Singh Thalor

Date:

04 March 2020, 09:15:12

On Mon, 24 Feb 2020 at 15:39, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Feb 20, 2020 at 8:06 AM Andres Freund <andres@anarazel.de> wrote:
> >
> > Hi,
> >
> > On 2020-02-19 11:12:18 +0530, Amit Kapila wrote:
> > > I think till we know the real need for changing group locking, going
> > > in the direction of what Tom suggested to use an array of LWLocks [1]
> > > to address the problems in hand is a good idea.
> >
> > -many
> >
> > I think that building yet another locking subsystem is the entirely
> > wrong idea - especially when there's imo no convincing architectural
> > reasons to do so.
> >
>
> Hmm, AFAIU, it will be done by having an array of LWLocks which we do
> at other places as well (like BufferIO locks).  I am not sure if we
> can call it as new locking subsystem, but if we decide to continue
> using lock.c and change group locking then I think we can do that as
> well, see my comments below regarding that.
>
> >
> > > It is not very clear to me that are we thinking to give up on Tom's
> > > idea [1] and change group locking even though it is not clear or at
> > > least nobody has proposed an idea/patch which requires that?  Or are
> > > we thinking that we can do what Tom suggested for relation extension
> > > lock and also plan to change group locking for future parallel
> > > operations that might require it?
> >
> > What I'm advocating is that extension locks should continue to go
> > through lock.c. And yes, that requires some changes to group locking,
> > but I still don't see why they'd be complicated.
> >
>
> Fair position, as per initial analysis, I think if we do below three
> things, it should work out without changing to a new way of locking
> for relation extension or page type locks.
> a. As per the discussion above, ensure in code we will never try to
> acquire another heavy-weight lock after acquiring relation extension
> or page type locks (probably by having Asserts in code or maybe some
> other way).
> b. Change lock.c so that group locking is not considered for these two
> lock types. For ex. in LockCheckConflicts, along with the check (if
> (proclock->groupLeader == MyProc && MyProc->lockGroupLeader == NULL)),
> we also check lock->tag and call it a conflict for these two locks.
> c. The deadlock detector can ignore checking these two types of locks
> because point (a) ensures that those won't lead to deadlock.  One idea
> could be that FindLockCycleRecurseMember just ignores these two types
> of locks by checking the lock tag.

Thanks Amit for summary.

Based on above 3 points, here attaching 2 patches for review.

1. v01_0001-Conflict-EXTENTION-lock-in-group-member.patch (Patch by Dilip Kumar)
Basically this patch is for point b and c.

2. v01_0002-Added-assert-to-verify-that-we-never-try-to-take-any.patch
(Patch by me)
This patch is for point a.

After applying both the patches, make check-world is passing.

We are testing both the patches and will post results.

Thoughts?

-- 
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

On Thu, 5 Mar 2020 at 13:54, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Mar 5, 2020 at 12:15 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Mar 4, 2020 at 11:45 AM Mahendra Singh Thalor
> > <mahi6run@gmail.com> wrote:
> > >
> > > On Mon, 24 Feb 2020 at 15:39, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund <andres@anarazel.de> wrote:
> > > > >
> > > > > What I'm advocating is that extension locks should continue to go
> > > > > through lock.c. And yes, that requires some changes to group locking,
> > > > > but I still don't see why they'd be complicated.
> > > > >
> > > >
> > > > Fair position, as per initial analysis, I think if we do below three
> > > > things, it should work out without changing to a new way of locking
> > > > for relation extension or page type locks.
> > > > a. As per the discussion above, ensure in code we will never try to
> > > > acquire another heavy-weight lock after acquiring relation extension
> > > > or page type locks (probably by having Asserts in code or maybe some
> > > > other way).
> > > > b. Change lock.c so that group locking is not considered for these two
> > > > lock types. For ex. in LockCheckConflicts, along with the check (if
> > > > (proclock->groupLeader == MyProc && MyProc->lockGroupLeader == NULL)),
> > > > we also check lock->tag and call it a conflict for these two locks.
> > > > c. The deadlock detector can ignore checking these two types of locks
> > > > because point (a) ensures that those won't lead to deadlock.  One idea
> > > > could be that FindLockCycleRecurseMember just ignores these two types
> > > > of locks by checking the lock tag.
> > >
> > > Thanks Amit for summary.
> > >
> > > Based on above 3 points, here attaching 2 patches for review.
> > >
> > > 1. v01_0001-Conflict-EXTENTION-lock-in-group-member.patch (Patch by Dilip Kumar)
> > > Basically this patch is for point b and c.
> > >
> > > 2. v01_0002-Added-assert-to-verify-that-we-never-try-to-take-any.patch
> > > (Patch by me)
> > > This patch is for point a.
> > >
> > > After applying both the patches, make check-world is passing.
> > >
> > > We are testing both the patches and will post results.
> > >
> >

Thanks Amit and Dilip for reviewing the patches.

> > I think we need to do detailed code review in the places where we are
> > taking Relation Extension Lock and see whether we are acquiring
> > another heavy-weight lock after that. It seems to me that in
> > brin_getinsertbuffer, after acquiring Relation Extension Lock, we
> > might again try to acquire the same lock.  See
> > brin_initialize_empty_new_buffer which is called after acquiring
> > Relation Extension Lock, in that function, we call
> > RecordPageWithFreeSpace and that can again try to acquire the same
> > lock if it needs to perform fsm_extend.  I think there will be similar
> > instances in the code.  I think it is fine if we again try to acquire
> > it, but the current assertion in your patch needs to be adjusted for
> > that.

I agree with you.  Dilip is doing code review and he will post
results.  As you mentioned that while holing Relation Extension Lock,
we might again try to acquire same Relation Extension Lock, so to
handle this in assert I did some changes in patch and attaching patch
for review. (I will test this scenario)

> >
> > Few other minor comments on
> > v01_0002-Added-assert-to-verify-that-we-never-try-to-take-any:
> > 1. Ideally, this should be the first patch as we first need to ensure
> > that we don't take any heavy-weight locks after acquiring a relation
> > extension lock.

Fixed.

> > 2. I think it is better to add an Assert after initial error checks
> > (after RecoveryInProgress().. check)

I am not getting your points. Can you explain me, that which type of
assert you are suggesting?

> > 3.
> > + Assert (locallock->tag.lock.locktag_type != LOCKTAG_RELATION_EXTEND ||
> > + locallock->nLocks == 0);
> >
> > I think it is not possible that we have an entry in
> > LockMethodLocalHash and its value is zero.  Do you see any such
> > possibility, if not, then we might want to remove it?

Yes, this condition is not needed. Fixed.

> >
> > 4. We already have a macro for LOCALLOCK_LOCKMETHOD, can we write
> > another one tag type?  This will make the check look a bit cleaner and
> > probably if we need to extend it in future for Page type locks, then
> > also it will be good.

Good point. I added macros in this version.

Here, attaching new patch set for review.

Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

On Sat, Mar 7, 2020 at 9:57 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Fri, Mar 6, 2020 at 9:47 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Mar 5, 2020 at 1:54 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Mar 5, 2020 at 12:15 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > 5. I have also tried to think of another way to check if we already
> > > hold lock type LOCKTAG_RELATION_EXTEND, but couldn't come up with a
> > > cheaper way than this. Basically, I think if we traverse the
> > > MyProc->myProcLocks queue, we will get this information, but that
> > > doesn't seem much cheaper than this.
> >
> > I think we can maintain a flag (rel_extlock_held). And, we can set
> > that true in LockRelationForExtension,
> > ConditionalLockRelationForExtension functions and we can reset it in
> > UnlockRelationForExtension or in the error path e.g. LockReleaseAll.
> >
>
> I think if we reset it in LockReleaseAll during the error path, then
> we need to find a way to reset it during LockReleaseCurrentOwner as
> that is called during Subtransaction Abort which can be tricky as we
> don't know if it belongs to the current owner. How about resetting in
> Abort(Sub)Transaction and CommitTransaction after we release locks via
> ResourceOwnerRelease.

I think instead of the flag we need to keep the counter because we can
acquire the same relation extension lock multiple times. So
basically, every time we acquire the lock we can increment the counter
and while releasing we can decrement it. During an error path, I
think it is fine to set it to 0 in CommitTransaction/AbortTransaction.
But, I am not sure that we can set to 0 or decrement it in
AbortSubTransaction because we are not sure whether we have acquired
the lock under this subtransaction or not.

Having said that, I think there should not be any case that we are
starting the sub-transaction while holding the relation extension
lock.

Right, this is exactly the point. I think we can mention this in comments to make it clear why setting it to zero is fine during subtransaction abort.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

07 March 2020, 12:56:19

On Sat, Mar 7, 2020 at 11:14 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Sat, Mar 7, 2020 at 9:57 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Fri, Mar 6, 2020 at 9:47 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Mar 5, 2020 at 1:54 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Mar 5, 2020 at 12:15 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > 5. I have also tried to think of another way to check if we already
> > > hold lock type LOCKTAG_RELATION_EXTEND, but couldn't come up with a
> > > cheaper way than this. Basically, I think if we traverse the
> > > MyProc->myProcLocks queue, we will get this information, but that
> > > doesn't seem much cheaper than this.
> >
> > I think we can maintain a flag (rel_extlock_held). And, we can set
> > that true in LockRelationForExtension,
> > ConditionalLockRelationForExtension functions and we can reset it in
> > UnlockRelationForExtension or in the error path e.g. LockReleaseAll.
> >
>
> I think if we reset it in LockReleaseAll during the error path, then
> we need to find a way to reset it during LockReleaseCurrentOwner as
> that is called during Subtransaction Abort which can be tricky as we
> don't know if it belongs to the current owner. How about resetting in
> Abort(Sub)Transaction and CommitTransaction after we release locks via
> ResourceOwnerRelease.

I think instead of the flag we need to keep the counter because we can
acquire the same relation extension lock multiple times. So
basically, every time we acquire the lock we can increment the counter
and while releasing we can decrement it. During an error path, I
think it is fine to set it to 0 in CommitTransaction/AbortTransaction.
But, I am not sure that we can set to 0 or decrement it in
AbortSubTransaction because we are not sure whether we have acquired
the lock under this subtransaction or not.

Having said that, I think there should not be any case that we are
starting the sub-transaction while holding the relation extension
lock.

Right, this is exactly the point. I think we can mention this in comments to make it clear why setting it to zero is fine during subtransaction abort.

Is there anything wrong with having an Assert during subtransaction start to indicate that we don't have a relation extension lock?

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dilip Kumar

Date:

07 March 2020, 17:24:11

On Sat, Mar 7, 2020 at 3:26 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sat, Mar 7, 2020 at 11:14 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Sat, Mar 7, 2020 at 9:57 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>>>
>>> On Fri, Mar 6, 2020 at 9:47 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> >
>>> > On Thu, Mar 5, 2020 at 1:54 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>>> > >
>>> > > On Thu, Mar 5, 2020 at 12:15 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> > > >
>>> > > >
>>> > > > 5. I have also tried to think of another way to check if we already
>>> > > > hold lock type LOCKTAG_RELATION_EXTEND, but couldn't come up with a
>>> > > > cheaper way than this.  Basically, I think if we traverse the
>>> > > > MyProc->myProcLocks queue, we will get this information, but that
>>> > > > doesn't seem much cheaper than this.
>>> > >
>>> > > I think we can maintain a flag (rel_extlock_held).  And, we can set
>>> > > that true in LockRelationForExtension,
>>> > > ConditionalLockRelationForExtension functions and we can reset it in
>>> > > UnlockRelationForExtension or in the error path e.g. LockReleaseAll.
>>> > >
>>> >
>>> > I think if we reset it in LockReleaseAll during the error path, then
>>> > we need to find a way to reset it during LockReleaseCurrentOwner as
>>> > that is called during Subtransaction Abort which can be tricky as we
>>> > don't know if it belongs to the current owner.  How about resetting in
>>> > Abort(Sub)Transaction and CommitTransaction after we release locks via
>>> > ResourceOwnerRelease.
>>>
>>> I think instead of the flag we need to keep the counter because we can
>>> acquire the same relation extension lock multiple times.  So
>>> basically, every time we acquire the lock we can increment the counter
>>> and while releasing we can decrement it.   During an error path, I
>>> think it is fine to set it to 0 in CommitTransaction/AbortTransaction.
>>> But, I am not sure that we can set to 0 or decrement it in
>>> AbortSubTransaction because we are not sure whether we have acquired
>>> the lock under this subtransaction or not.
>>>
>>> Having said that,  I think there should not be any case that we are
>>> starting the sub-transaction while holding the relation extension
>>> lock.
>>
>>
>> Right, this is exactly the point.  I think we can mention this in comments to make it clear why setting it to zero
isfine during subtransaction abort.
 
>
>
> Is there anything wrong with having an Assert during subtransaction start to indicate that we don't have a relation
extensionlock?
 

Yes, I was planning to do that.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Tom Lane

Date:

07 March 2020, 18:23:39

Dilip Kumar <dilipbalaut@gmail.com> writes:
> I think instead of the flag we need to keep the counter because we can
> acquire the same relation extension lock multiple times.

Uh ... what?  How would that not be broken usage on its face?

I continue to think that we'd be better off getting all of this
out of the heavyweight lock manager.  There is no reason why we
should need deadlock detection, or multiple holds of the same
lock, or pretty much anything that LWLocks don't give you.

            regards, tom lane

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dilip Kumar

Date:

07 March 2020, 18:49:01

On Sat, Mar 7, 2020 at 8:53 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Dilip Kumar <dilipbalaut@gmail.com> writes:
> > I think instead of the flag we need to keep the counter because we can
> > acquire the same relation extension lock multiple times.
>
> Uh ... what?  How would that not be broken usage on its face?

Basically,  if we can ensure that while holding the relation extension
lock we will not wait for any other lock then we can ignore in the
deadlock detection path so that we don't detect the false deadlock due
to the group locking mechanism.  So if we are already holding the
relation extension lock and trying to acquire the same lock-in same
mode then it can never wait so this is safe.

> I continue to think that we'd be better off getting all of this
> out of the heavyweight lock manager.  There is no reason why we
> should need deadlock detection, or multiple holds of the same
> lock, or pretty much anything that LWLocks don't give you.

Right, we never need deadlock detection for this lock.  But, I think
there are quite a few cases where we have multiple holds at the same
time.  e.g, during RelationAddExtraBlocks, while holding the relation
extension lock we try to update the block in FSM and FSM might need to
add extra FSM block which will again try to acquire the same lock.

But, I think the main reason for not converting it to an LWLocks is
because Andres has a concern about inventing new lock mechanism as
discuss upthread[1]

[1] https://www.postgresql.org/message-id/20200220023612.c44ggploywxtlvmx%40alap3.anarazel.de

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

08 March 2020, 05:28:21

On Mon, 24 Feb 2020 at 19:08, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Feb 20, 2020 at 8:06 AM Andres Freund <andres@anarazel.de> wrote:
> >
> > Hi,
> >
> > On 2020-02-19 11:12:18 +0530, Amit Kapila wrote:
> > > I think till we know the real need for changing group locking, going
> > > in the direction of what Tom suggested to use an array of LWLocks [1]
> > > to address the problems in hand is a good idea.
> >
> > -many
> >
> > I think that building yet another locking subsystem is the entirely
> > wrong idea - especially when there's imo no convincing architectural
> > reasons to do so.
> >
>
> Hmm, AFAIU, it will be done by having an array of LWLocks which we do
> at other places as well (like BufferIO locks).  I am not sure if we
> can call it as new locking subsystem, but if we decide to continue
> using lock.c and change group locking then I think we can do that as
> well, see my comments below regarding that.
>
> >
> > > It is not very clear to me that are we thinking to give up on Tom's
> > > idea [1] and change group locking even though it is not clear or at
> > > least nobody has proposed an idea/patch which requires that?  Or are
> > > we thinking that we can do what Tom suggested for relation extension
> > > lock and also plan to change group locking for future parallel
> > > operations that might require it?
> >
> > What I'm advocating is that extension locks should continue to go
> > through lock.c. And yes, that requires some changes to group locking,
> > but I still don't see why they'd be complicated.
> >
>
> Fair position, as per initial analysis, I think if we do below three
> things, it should work out without changing to a new way of locking
> for relation extension or page type locks.
> a. As per the discussion above, ensure in code we will never try to
> acquire another heavy-weight lock after acquiring relation extension
> or page type locks (probably by having Asserts in code or maybe some
> other way).

The current patch
(v02_0001-Added-assert-to-verify-that-we-never-try-to-take-any.patch)
doesn't check that acquiring a heavy-weight lock after page type lock,
is that right? There is the path doing that: ginInsertCleanup() holds
a page lock and insert the pending list items, which might hold a
relation extension lock.

Regards,

-- 
Masahiko Sawada            http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

09 March 2020, 07:26:43

On Sat, Mar 7, 2020 at 9:19 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:

On Sat, Mar 7, 2020 at 8:53 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Dilip Kumar <dilipbalaut@gmail.com> writes:
> > I think instead of the flag we need to keep the counter because we can
> > acquire the same relation extension lock multiple times.
>
> Uh ... what? How would that not be broken usage on its face?

Basically, if we can ensure that while holding the relation extension
lock we will not wait for any other lock then we can ignore in the
deadlock detection path so that we don't detect the false deadlock due
to the group locking mechanism. So if we are already holding the
relation extension lock and trying to acquire the same lock-in same
mode then it can never wait so this is safe.

> I continue to think that we'd be better off getting all of this
> out of the heavyweight lock manager. There is no reason why we
> should need deadlock detection, or multiple holds of the same
> lock, or pretty much anything that LWLocks don't give you.

Right, we never need deadlock detection for this lock. But, I think
there are quite a few cases where we have multiple holds at the same
time. e.g, during RelationAddExtraBlocks, while holding the relation
extension lock we try to update the block in FSM and FSM might need to
add extra FSM block which will again try to acquire the same lock.

But, I think the main reason for not converting it to an LWLocks is
because Andres has a concern about inventing new lock mechanism as
discuss upthread[1]

Right, that is one point and another is that if we go via the route of converting it to LWLocks, then we also need to think of some solution for page locks that are used in ginInsertCleanup. However, if we go with the approach being pursued [1] then the page locks will be handled in a similar way as relation extension locks.

[1] - https://www.postgresql.org/message-id/CAA4eK1%2BNjo%2BpnqSNi2ScKf0BcVBWWf37BrW-pykVSG0B0C5Qig%40mail.gmail.com

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

09 March 2020, 08:16:34

On Sun, Mar 8, 2020 at 7:58 AM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:

On Mon, 24 Feb 2020 at 19:08, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Feb 20, 2020 at 8:06 AM Andres Freund <andres@anarazel.de> wrote:
> >
> > Hi,
> >
> > On 2020-02-19 11:12:18 +0530, Amit Kapila wrote:
> > > I think till we know the real need for changing group locking, going
> > > in the direction of what Tom suggested to use an array of LWLocks [1]
> > > to address the problems in hand is a good idea.
> >
> > -many
> >
> > I think that building yet another locking subsystem is the entirely
> > wrong idea - especially when there's imo no convincing architectural
> > reasons to do so.
> >
>
> Hmm, AFAIU, it will be done by having an array of LWLocks which we do
> at other places as well (like BufferIO locks). I am not sure if we
> can call it as new locking subsystem, but if we decide to continue
> using lock.c and change group locking then I think we can do that as
> well, see my comments below regarding that.
>
> >
> > > It is not very clear to me that are we thinking to give up on Tom's
> > > idea [1] and change group locking even though it is not clear or at
> > > least nobody has proposed an idea/patch which requires that? Or are
> > > we thinking that we can do what Tom suggested for relation extension
> > > lock and also plan to change group locking for future parallel
> > > operations that might require it?
> >
> > What I'm advocating is that extension locks should continue to go
> > through lock.c. And yes, that requires some changes to group locking,
> > but I still don't see why they'd be complicated.
> >
>
> Fair position, as per initial analysis, I think if we do below three
> things, it should work out without changing to a new way of locking
> for relation extension or page type locks.
> a. As per the discussion above, ensure in code we will never try to
> acquire another heavy-weight lock after acquiring relation extension
> or page type locks (probably by having Asserts in code or maybe some
> other way).

The current patch
(v02_0001-Added-assert-to-verify-that-we-never-try-to-take-any.patch)
doesn't check that acquiring a heavy-weight lock after page type lock,
is that right?

No, it should do that.

There is the path doing that: ginInsertCleanup() holds
a page lock and insert the pending list items, which might hold a
relation extension lock.

Right, I could also see that, but do you see any problem with that? I agree that Assert should cover this case, but I don't see any fundamental problem with that.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

09 March 2020, 09:07:31

On Mon, 9 Mar 2020 at 14:16, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sun, Mar 8, 2020 at 7:58 AM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:
>>
>> On Mon, 24 Feb 2020 at 19:08, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> >
>> > On Thu, Feb 20, 2020 at 8:06 AM Andres Freund <andres@anarazel.de> wrote:
>> > >
>> > > Hi,
>> > >
>> > > On 2020-02-19 11:12:18 +0530, Amit Kapila wrote:
>> > > > I think till we know the real need for changing group locking, going
>> > > > in the direction of what Tom suggested to use an array of LWLocks [1]
>> > > > to address the problems in hand is a good idea.
>> > >
>> > > -many
>> > >
>> > > I think that building yet another locking subsystem is the entirely
>> > > wrong idea - especially when there's imo no convincing architectural
>> > > reasons to do so.
>> > >
>> >
>> > Hmm, AFAIU, it will be done by having an array of LWLocks which we do
>> > at other places as well (like BufferIO locks).  I am not sure if we
>> > can call it as new locking subsystem, but if we decide to continue
>> > using lock.c and change group locking then I think we can do that as
>> > well, see my comments below regarding that.
>> >
>> > >
>> > > > It is not very clear to me that are we thinking to give up on Tom's
>> > > > idea [1] and change group locking even though it is not clear or at
>> > > > least nobody has proposed an idea/patch which requires that?  Or are
>> > > > we thinking that we can do what Tom suggested for relation extension
>> > > > lock and also plan to change group locking for future parallel
>> > > > operations that might require it?
>> > >
>> > > What I'm advocating is that extension locks should continue to go
>> > > through lock.c. And yes, that requires some changes to group locking,
>> > > but I still don't see why they'd be complicated.
>> > >
>> >
>> > Fair position, as per initial analysis, I think if we do below three
>> > things, it should work out without changing to a new way of locking
>> > for relation extension or page type locks.
>> > a. As per the discussion above, ensure in code we will never try to
>> > acquire another heavy-weight lock after acquiring relation extension
>> > or page type locks (probably by having Asserts in code or maybe some
>> > other way).
>>
>> The current patch
>> (v02_0001-Added-assert-to-verify-that-we-never-try-to-take-any.patch)
>> doesn't check that acquiring a heavy-weight lock after page type lock,
>> is that right?
>
>
> No, it should do that.
>
>>
>> There is the path doing that: ginInsertCleanup() holds
>> a page lock and insert the pending list items, which might hold a
>> relation extension lock.
>
>
> Right, I could also see that, but do you see any problem with that?  I agree that Assert should cover this case, but
Idon't see any fundamental problem with that.
 

I think that could be a problem if we change the group locking so that
it doesn't consider page lock type.

Regards,

--
Masahiko Sawada            http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

09 March 2020, 09:50:45

On Mon, Mar 9, 2020 at 11:38 AM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:

On Mon, 9 Mar 2020 at 14:16, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sun, Mar 8, 2020 at 7:58 AM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:
>> >
>> > Fair position, as per initial analysis, I think if we do below three
>> > things, it should work out without changing to a new way of locking
>> > for relation extension or page type locks.
>> > a. As per the discussion above, ensure in code we will never try to
>> > acquire another heavy-weight lock after acquiring relation extension
>> > or page type locks (probably by having Asserts in code or maybe some
>> > other way).
>>
>> The current patch
>> (v02_0001-Added-assert-to-verify-that-we-never-try-to-take-any.patch)
>> doesn't check that acquiring a heavy-weight lock after page type lock,
>> is that right?
>
>
> No, it should do that.
>
>>
>> There is the path doing that: ginInsertCleanup() holds
>> a page lock and insert the pending list items, which might hold a
>> relation extension lock.
>
>
> Right, I could also see that, but do you see any problem with that? I agree that Assert should cover this case, but I don't see any fundamental problem with that.

I think that could be a problem if we change the group locking so that
it doesn't consider page lock type.

I might be missing something, but won't that be a problem only when if there is a case where we acquire page lock after acquiring a relation extension lock? Can you please explain the scenario you have in mind which can create a problem?

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Masahiko Sawada

Date:

09 March 2020, 11:38:31

On Mon, 9 Mar 2020 at 15:50, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Mar 9, 2020 at 11:38 AM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:
>>
>> On Mon, 9 Mar 2020 at 14:16, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> >
>> > On Sun, Mar 8, 2020 at 7:58 AM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:
>> >> >
>> >> > Fair position, as per initial analysis, I think if we do below three
>> >> > things, it should work out without changing to a new way of locking
>> >> > for relation extension or page type locks.
>> >> > a. As per the discussion above, ensure in code we will never try to
>> >> > acquire another heavy-weight lock after acquiring relation extension
>> >> > or page type locks (probably by having Asserts in code or maybe some
>> >> > other way).
>> >>
>> >> The current patch
>> >> (v02_0001-Added-assert-to-verify-that-we-never-try-to-take-any.patch)
>> >> doesn't check that acquiring a heavy-weight lock after page type lock,
>> >> is that right?
>> >
>> >
>> > No, it should do that.
>> >
>> >>
>> >> There is the path doing that: ginInsertCleanup() holds
>> >> a page lock and insert the pending list items, which might hold a
>> >> relation extension lock.
>> >
>> >
>> > Right, I could also see that, but do you see any problem with that?  I agree that Assert should cover this case,
butI don't see any fundamental problem with that.
 
>>
>> I think that could be a problem if we change the group locking so that
>> it doesn't consider page lock type.
>
>
> I might be missing something, but won't that be a problem only when if there is a case where we acquire page lock
afteracquiring a relation extension lock?
 

Yes, you're right.

Well I meant that the reason why we need to make Assert should cover
page locks case is the same as the reason for extension lock type
case. If we change the group locking so that it doesn't consider
extension lock and change deadlock so that it doesn't make a wait edge
for it, we need to ensure that the same backend doesn't acquire
heavy-weight lock after holding relation extension lock. These are
already done in the current patch. Similarly, if we did the similar
change for page lock in the group locking and deadlock , we need to
ensure the same things for page lock. But ISTM it doesn't necessarily
need to support page lock for now because currently we use it only for
cleanup pending list of gin index.


Regards,

-- 
Masahiko Sawada            http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

09 March 2020, 12:06:32

On Mon, Mar 9, 2020 at 2:09 PM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:

On Mon, 9 Mar 2020 at 15:50, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Mar 9, 2020 at 11:38 AM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:
>>
>> On Mon, 9 Mar 2020 at 14:16, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> >
>> > On Sun, Mar 8, 2020 at 7:58 AM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:
>> >> >
>> >> > Fair position, as per initial analysis, I think if we do below three
>> >> > things, it should work out without changing to a new way of locking
>> >> > for relation extension or page type locks.
>> >> > a. As per the discussion above, ensure in code we will never try to
>> >> > acquire another heavy-weight lock after acquiring relation extension
>> >> > or page type locks (probably by having Asserts in code or maybe some
>> >> > other way).
>> >>
>> >> The current patch
>> >> (v02_0001-Added-assert-to-verify-that-we-never-try-to-take-any.patch)
>> >> doesn't check that acquiring a heavy-weight lock after page type lock,
>> >> is that right?
>> >
>> >
>> > No, it should do that.
>> >
>> >>
>> >> There is the path doing that: ginInsertCleanup() holds
>> >> a page lock and insert the pending list items, which might hold a
>> >> relation extension lock.
>> >
>> >
>> > Right, I could also see that, but do you see any problem with that? I agree that Assert should cover this case, but I don't see any fundamental problem with that.
>>
>> I think that could be a problem if we change the group locking so that
>> it doesn't consider page lock type.
>
>
> I might be missing something, but won't that be a problem only when if there is a case where we acquire page lock after acquiring a relation extension lock?

Yes, you're right.

Well I meant that the reason why we need to make Assert should cover
page locks case is the same as the reason for extension lock type
case. If we change the group locking so that it doesn't consider
extension lock and change deadlock so that it doesn't make a wait edge
for it, we need to ensure that the same backend doesn't acquire
heavy-weight lock after holding relation extension lock. These are
already done in the current patch. Similarly, if we did the similar
change for page lock in the group locking and deadlock , we need to
ensure the same things for page lock.

Agreed.

But ISTM it doesn't necessarily
need to support page lock for now because currently we use it only for
cleanup pending list of gin index.

I agree, but I think it is better to have a patch for the same even if we want to review/commit that separately. That will help us to look at how the complete solution looks.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dilip Kumar

Date:

10 March 2020, 06:23:38

On Mon, Mar 9, 2020 at 2:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Mar 9, 2020 at 2:09 PM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:
>>
>> On Mon, 9 Mar 2020 at 15:50, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> >
>> > On Mon, Mar 9, 2020 at 11:38 AM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:
>> >>
>> >> On Mon, 9 Mar 2020 at 14:16, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> >> >
>> >> > On Sun, Mar 8, 2020 at 7:58 AM Masahiko Sawada <masahiko.sawada@2ndquadrant.com> wrote:
>> >> >> >
>> >> >> > Fair position, as per initial analysis, I think if we do below three
>> >> >> > things, it should work out without changing to a new way of locking
>> >> >> > for relation extension or page type locks.
>> >> >> > a. As per the discussion above, ensure in code we will never try to
>> >> >> > acquire another heavy-weight lock after acquiring relation extension
>> >> >> > or page type locks (probably by having Asserts in code or maybe some
>> >> >> > other way).
>> >> >>
>> >> >> The current patch
>> >> >> (v02_0001-Added-assert-to-verify-that-we-never-try-to-take-any.patch)
>> >> >> doesn't check that acquiring a heavy-weight lock after page type lock,
>> >> >> is that right?
>> >> >
>> >> >
>> >> > No, it should do that.
>> >> >
>> >> >>
>> >> >> There is the path doing that: ginInsertCleanup() holds
>> >> >> a page lock and insert the pending list items, which might hold a
>> >> >> relation extension lock.
>> >> >
>> >> >
>> >> > Right, I could also see that, but do you see any problem with that?  I agree that Assert should cover this
case,but I don't see any fundamental problem with that.
 
>> >>
>> >> I think that could be a problem if we change the group locking so that
>> >> it doesn't consider page lock type.
>> >
>> >
>> > I might be missing something, but won't that be a problem only when if there is a case where we acquire page lock
afteracquiring a relation extension lock?
 
>>
>> Yes, you're right.
>>
>> Well I meant that the reason why we need to make Assert should cover
>> page locks case is the same as the reason for extension lock type
>> case. If we change the group locking so that it doesn't consider
>> extension lock and change deadlock so that it doesn't make a wait edge
>> for it, we need to ensure that the same backend doesn't acquire
>> heavy-weight lock after holding relation extension lock. These are
>> already done in the current patch. Similarly, if we did the similar
>> change for page lock in the group locking and deadlock , we need to
>> ensure the same things for page lock.
>
>
> Agreed.
>
>>
>> But ISTM it doesn't necessarily
>> need to support page lock for now because currently we use it only for
>> cleanup pending list of gin index.
>>
>
> I agree, but I think it is better to have a patch for the same even if we want to review/commit that separately.
Thatwill help us to look at how the complete solution looks.
 

Please find the updated patch (summary of the changes)
- Instead of searching the lock hash table for assert, it maintains a counter.
- Also, handled the case where we can acquire the relation extension
lock while holding the relation extension lock on the same relation.
- Handled the error case.

In addition to that prepared a WIP patch for handling the PageLock.
First, I thought that we can use the same counter for the PageLock and
the RelationExtensionLock because in assert we just need to check
whether we are trying to acquire any other heavyweight lock while
holding any of these locks.  But, the exceptional case where we
allowed to acquire a relation extension lock while holding any of
these locks is a bit different.  Because, if we are holding a relation
extension lock then we allowed to acquire the relation extension lock
on the same relation but it can not be any other relation otherwise it
can create a cycle.  But, the same is not true with the PageLock,
i.e. while holding the PageLock you can acquire the relation extension
lock on any relation and that will be safe because the relation
extension lock guarantee that, it will never create the cycle.
However, I agree that we don't have any such cases where we want to
acquire a relation extension lock on the different relations while
holding the PageLock.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

On Wed, Mar 11, 2020 at 2:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Mar 10, 2020 at 8:53 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > Please find the updated patch (summary of the changes)
> > - Instead of searching the lock hash table for assert, it maintains a counter.
> > - Also, handled the case where we can acquire the relation extension
> > lock while holding the relation extension lock on the same relation.
> > - Handled the error case.
> >
> > In addition to that prepared a WIP patch for handling the PageLock.
> > First, I thought that we can use the same counter for the PageLock and
> > the RelationExtensionLock because in assert we just need to check
> > whether we are trying to acquire any other heavyweight lock while
> > holding any of these locks.  But, the exceptional case where we
> > allowed to acquire a relation extension lock while holding any of
> > these locks is a bit different.  Because, if we are holding a relation
> > extension lock then we allowed to acquire the relation extension lock
> > on the same relation but it can not be any other relation otherwise it
> > can create a cycle.  But, the same is not true with the PageLock,
> > i.e. while holding the PageLock you can acquire the relation extension
> > lock on any relation and that will be safe because the relation
> > extension lock guarantee that, it will never create the cycle.
> > However, I agree that we don't have any such cases where we want to
> > acquire a relation extension lock on the different relations while
> > holding the PageLock.
> >
>
> Right, today, we don't have such cases where after acquiring relation
> extension or page lock for a particular relation, we need to acquire
> any of those for other relation and I am not able to offhand think of
> many cases where we might have such a need in the future.  The one
> theoretical possibility is to include fork_num in the lock tag while
> acquiring extension lock for fsm/vm, but that will also have the same
> relation.  Similarly one might say it is valid to acquire extension
> lock in share mode after we have acquired it exclusive mode.  I am not
> sure how much futuristic we want to make these Asserts.
>
> I feel we should cover the current possible cases (which I think will
> make the asserts more strict then required) and if there is a need to
> relax them in the future for any particular use case, then we will
> consider those.  In general, if we consider the way Mahendra has
> written a patch which is to find the entry via the local hash table to
> check for an Assert condition, then it will be a bit easier to extend
> the checks if required in future as that way we have more information
> about the particular lock. However, it will make the check more
> expensive which might be okay considering that it is only for Assert
> enabled builds.
>
> One minor comment:
> /*
> + * We should not acquire any other lock if we are already holding the
> + * relation extension lock.  Only exception is that if we are trying to
> + * acquire the relation extension lock then we can hold the relation
> + * extension on the same relation.
> + */
> + Assert(!IsRelExtLockHeld() ||
> +    ((locktag->locktag_type == LOCKTAG_RELATION_EXTEND) && found));

I have fixed this in the attached patch set.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Attachment

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dilip Kumar

Date:

12 March 2020, 12:34:00

On Wed, Mar 11, 2020 at 2:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Mar 10, 2020 at 6:39 PM Robert Haas <robertmhaas@gmail.com> wrote:
> >
> > On Fri, Mar 6, 2020 at 11:27 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > I think instead of the flag we need to keep the counter because we can
> > > acquire the same relation extension lock multiple times.  So
> > > basically, every time we acquire the lock we can increment the counter
> > > and while releasing we can decrement it.   During an error path, I
> > > think it is fine to set it to 0 in CommitTransaction/AbortTransaction.
> > > But, I am not sure that we can set to 0 or decrement it in
> > > AbortSubTransaction because we are not sure whether we have acquired
> > > the lock under this subtransaction or not.
> >
> > I think that CommitTransaction, AbortTransaction, and friends have
> > *zero* business touching this. I think the counter - or flag - should
> > track whether we've got a PROCLOCK entry for a relation extension
> > lock. We either do, or we do not, and that does not change because of
> > anything have to do with the transaction state. It changes because
> > somebody calls LockRelease() or LockReleaseAll().
> >
>
> Do we want to have a special check in the LockRelease() to identify
> whether we are releasing relation extension lock?  If not, then how we
> will identify that relation extension is released and we can reset it
> during subtransaction abort due to error?  During success paths, we
> know when we have released RelationExtension or Page Lock (via
> UnlockRelationForExtension or UnlockPage).  During the top-level
> transaction end, we know when we have released all the locks, so that
> will imply that RelationExtension and or Page locks must have been
> released by that time.
>
> If we have no other choice, then I see a few downsides of adding a
> special check in the LockRelease() call:
>
> 1. Instead of resetting/decrement the variable from specific APIs like
> UnlockRelationForExtension or UnlockPage, we need to have it in
> LockRelease. It will also look odd, if set variable in
> LockRelationForExtension, but don't reset in the
> UnlockRelationForExtension variant.  Now, maybe we can allow to reset
> it at both places if it is a flag, but not if it is a counter
> variable.
>
> 2. One can argue that adding extra instructions in a generic path
> (like LockRelease) is not a good idea, especially if those are for an
> Assert. I understand this won't add anything which we can measure by
> standard benchmarks.

I have just written a WIP patch for relation extension lock where
instead of incrementing and decrementing the counter in
LockRelationForExtension and UnlockRelationForExtension respectively.
We can just set and reset the flag in LockAcquireExtended and
LockRelease.  So this patch appears simple to me as we are not
involving the transaction APIs to set and reset the flag.  However, we
need to add an extra check as you have already mentioned.  I think we
could measure the performance and see whether it has any impact or
not?

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Attachment

v5-0001-WIP-Add-assert-check-for-relation-extension-lock.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

12 March 2020, 14:57:59

On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have fixed this in the attached patch set.
>

I have modified your
v4-0003-Conflict-Extension-Page-lock-in-group-member patch.  The
modifications are (a) Change src/backend/storage/lmgr/README to
reflect new behaviour, (b) Introduce a new macro LOCK_LOCKTAG which
slightly simplifies the code, (c) moved the deadlock.c check a few
lines up and (d) changed a few comments.

It might be better if we can move the checks related to extension and
page lock in a separate API or macro.  What do you think?

I have also used an extension to test this patch.  This is the same
extension that I have used to test the group locking patch.  It will
allow backends to form a group as we do for parallel workers.  The
extension is attached to this email.

Test without patch:
Session-1
Create table t1(c1 int, c2 char(500));
Select become_lock_group_leader();

Insert into t1 values(generate_series(1,100),'aaa'); -- stop this
after acquiring relation extension lock via GDB.

Session-2
Select  become_lock_group_member();
Insert into t1 values(generate_series(101,200),'aaa');
- Debug LockAcquire and found that it doesn't generate conflict for
Relation Extension lock.

The above experiment has shown that without patch group members can
acquire relation extension lock if the group leader has that lock.
After patch the second session waits for the first session to release
the relation extension lock. I know this is not a perfect way to test,
but it is better than nothing.  I think we need to do some more
testing either using this extension or some other way for extension
and page locks.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

On Fri, Mar 13, 2020 at 8:37 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Mar 12, 2020 at 5:28 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > I have fixed this in the attached patch set.
> > >
> >
> > I have modified your
> > v4-0003-Conflict-Extension-Page-lock-in-group-member patch.  The
> > modifications are (a) Change src/backend/storage/lmgr/README to
> > reflect new behaviour, (b) Introduce a new macro LOCK_LOCKTAG which
> > slightly simplifies the code, (c) moved the deadlock.c check a few
> > lines up and (d) changed a few comments.
>
> Changes look fine to me.
>

Today, while looking at this patch again, I realized that there is a
where we sometimes allow group members to jump the wait queue.  This
is primarily to avoid creating deadlocks (see ProcSleep).  Now,
ideally, we don't need this for relation extension or page locks as
those can never lead to deadlocks.  However, the current code will
give group members more priority to acquire relation extension or page
locks if any one of the members has held those locks.  Now, if we want
we can prevent giving group members priority for these locks, but I am
not sure how important is that case.  So, I have left that as it is by
adding a few comments.  What do you think?

Additionally, I have changed/added a few more sentences in README.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Attachment

0002-Allow-relation-extension-and-page-locks-to-conflict-.v2.patch

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dilip Kumar

Date:

13 March 2020, 13:11:56

On Fri, Mar 13, 2020 at 2:32 PM Kuntal Ghosh <kuntalghosh.2007@gmail.com> wrote:
>
> On Fri, Mar 13, 2020 at 8:29 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Mar 12, 2020 at 7:50 PM Kuntal Ghosh <kuntalghosh.2007@gmail.com> wrote:
> > > I think moving them inside a macro is a good idea. Also, I think we
> > > should move all the Assert related code inside some debugging macro
> > > similar to this:
> > > #ifdef LOCK_DEBUG
> > > ....
> > > #endif
> > >
> > If we move it under some macro, then those Asserts will be only
> > enabled when that macro is defined.  I think we want there Asserts to
> > be enabled always in assert enabled build, these will be like any
> > other Asserts in the code.  What is the advantage of doing those under
> > macro?
> >
> My concern is related to performance regression. We're using two
> static variables in hot-paths only for checking a few asserts. So, I'm
> not sure whether we should enable the same by default, specially when
> asserts are itself disabled.
> -ResetRelExtLockHeldCount()
> +ResetRelExtPageLockHeldCount()
>  {
>   RelationExtensionLockHeldCount = 0;
> + PageLockHeldCount = 0;
> +}
> Also, we're calling this method from frequently used functions like
> Commit/AbortTransaction. So, it's better these two static variables
> share the same cache line and reinitalize them with a single
> instruction.

In the recent version of the patch, instead of a counter, we have done
with a flag.  So I think now we can just keep a single variable and we
can just reset the bit in a single instruction.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dilip Kumar

Date:

13 March 2020, 16:32:01

On Fri, Mar 13, 2020 at 11:16 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Fri, Mar 13, 2020 at 11:08 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Mar 12, 2020 at 3:04 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Wed, Mar 11, 2020 at 2:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > >
> > > > If we have no other choice, then I see a few downsides of adding a
> > > > special check in the LockRelease() call:
> > > >
> > > > 1. Instead of resetting/decrement the variable from specific APIs like
> > > > UnlockRelationForExtension or UnlockPage, we need to have it in
> > > > LockRelease. It will also look odd, if set variable in
> > > > LockRelationForExtension, but don't reset in the
> > > > UnlockRelationForExtension variant.  Now, maybe we can allow to reset
> > > > it at both places if it is a flag, but not if it is a counter
> > > > variable.
> > > >
> > > > 2. One can argue that adding extra instructions in a generic path
> > > > (like LockRelease) is not a good idea, especially if those are for an
> > > > Assert. I understand this won't add anything which we can measure by
> > > > standard benchmarks.
> > >
> > > I have just written a WIP patch for relation extension lock where
> > > instead of incrementing and decrementing the counter in
> > > LockRelationForExtension and UnlockRelationForExtension respectively.
> > > We can just set and reset the flag in LockAcquireExtended and
> > > LockRelease.  So this patch appears simple to me as we are not
> > > involving the transaction APIs to set and reset the flag.  However, we
> > > need to add an extra check as you have already mentioned.  I think we
> > > could measure the performance and see whether it has any impact or
> > > not?
> > >
> >
> > LockAcquireExtended()
> > {
> > ..
> > + if (locktag->locktag_type == LOCKTAG_RELATION_EXTEND)
> > + IsRelationExtensionLockHeld = true;
> > ..
> > }
> >
> > Can we move this check inside a function (CheckAndSetLockHeld or
> > something like that) as we need to add a similar thing for page lock?
>
> ok

Done

>
> > Also, how about moving the set and reset of these flags to
> > GrantLockLocal and RemoveLocalLock as that will further reduce the
> > number of places where we need to add such a check.
>
> Make sense to me.

Done
>

>  Another thing is
> > to see if it makes sense to have a macro like LOCALLOCK_LOCKMETHOD to
> > get the lock tag.
>
> ok

Done

Apart from that, I have also extended the solution for the page lock.
And, I have also broken down the 3rd patch in two parts for relation
extension and for the page lock.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Attachment

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dilip Kumar

Date:

13 March 2020, 16:32:40

On Fri, Mar 13, 2020 at 3:39 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Mar 13, 2020 at 8:37 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Mar 12, 2020 at 5:28 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, Mar 12, 2020 at 11:15 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > I have fixed this in the attached patch set.
> > > >
> > >
> > > I have modified your
> > > v4-0003-Conflict-Extension-Page-lock-in-group-member patch.  The
> > > modifications are (a) Change src/backend/storage/lmgr/README to
> > > reflect new behaviour, (b) Introduce a new macro LOCK_LOCKTAG which
> > > slightly simplifies the code, (c) moved the deadlock.c check a few
> > > lines up and (d) changed a few comments.
> >
> > Changes look fine to me.
> >
>
> Today, while looking at this patch again, I realized that there is a
> where we sometimes allow group members to jump the wait queue.  This
> is primarily to avoid creating deadlocks (see ProcSleep).  Now,
> ideally, we don't need this for relation extension or page locks as
> those can never lead to deadlocks.  However, the current code will
> give group members more priority to acquire relation extension or page
> locks if any one of the members has held those locks.  Now, if we want
> we can prevent giving group members priority for these locks, but I am
> not sure how important is that case.  So, I have left that as it is by
> adding a few comments.  What do you think?
>
> Additionally, I have changed/added a few more sentences in README.

I have included all your changes in the latest patch set.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

14 March 2020, 17:09:43

On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> Apart from that, I have also extended the solution for the page lock.
> And, I have also broken down the 3rd patch in two parts for relation
> extension and for the page lock.
>

Thanks, I have made a number of cosmetic changes and written
appropriate commit messages for all patches.  See the attached patch
series and let me know your opinion? BTW, did you get a chance to test
page locks by using the extension which I have posted above or by some
other way?  I think it is important to test page-lock related patches
now.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Attachment

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dilip Kumar

Date:

15 March 2020, 10:45:19

On Sat, Mar 14, 2020 at 7:39 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > Apart from that, I have also extended the solution for the page lock.
> > And, I have also broken down the 3rd patch in two parts for relation
> > extension and for the page lock.
> >
>
> Thanks, I have made a number of cosmetic changes and written
> appropriate commit messages for all patches.  See the attached patch
> series and let me know your opinion? BTW, did you get a chance to test
> page locks by using the extension which I have posted above or by some
> other way?  I think it is important to test page-lock related patches
> now.

I have reviewed the updated patches and looks fine to me.  Apart from
this I have done testing for the Page Lock using group locking
extension.

--Setup
create table gin_test_tbl(i int4[]) with (autovacuum_enabled = off);
create index gin_test_idx on gin_test_tbl using gin (i);
create table gin_test_tbl1(i int4[]) with (autovacuum_enabled = off);
create index gin_test_idx1 on gin_test_tbl1 using gin (i);

--session1:
select become_lock_group_leader();
select gin_clean_pending_list('gin_test_idx');

--session2:
select become_lock_group_member(session1_pid);
select gin_clean_pending_list('gin_test_idx1');

--session3:
select become_lock_group_leader();
select gin_clean_pending_list('gin_test_idx1');

--session4:
select become_lock_group_member(session3_pid);
select gin_clean_pending_list('gin_test_idx');

ERROR:  deadlock detected
DETAIL:  Process 61953 waits for ExclusiveLock on page 0 of relation
16399 of database 13577; blocked by process 62197.
Process 62197 waits for ExclusiveLock on page 0 of relation 16400 of
database 13577; blocked by process 61953.
HINT:  See server log for query details.

Session1 and Session3 acquire the PageLock on two different index's
meta-pages and blocked in gdb,  meanwhile, their member tries to
acquire the page lock as shown in the above example and it detects the
deadlock which is solved after applying the patch.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dilip Kumar

Date:

15 March 2020, 14:04:28

On Sun, Mar 15, 2020 at 1:15 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Sat, Mar 14, 2020 at 7:39 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, Mar 13, 2020 at 7:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > Apart from that, I have also extended the solution for the page lock.
> > > And, I have also broken down the 3rd patch in two parts for relation
> > > extension and for the page lock.
> > >
> >
> > Thanks, I have made a number of cosmetic changes and written
> > appropriate commit messages for all patches.  See the attached patch
> > series and let me know your opinion? BTW, did you get a chance to test
> > page locks by using the extension which I have posted above or by some
> > other way?  I think it is important to test page-lock related patches
> > now.
>
> I have reviewed the updated patches and looks fine to me.  Apart from
> this I have done testing for the Page Lock using group locking
> extension.
>
> --Setup
> create table gin_test_tbl(i int4[]) with (autovacuum_enabled = off);
> create index gin_test_idx on gin_test_tbl using gin (i);
> create table gin_test_tbl1(i int4[]) with (autovacuum_enabled = off);
> create index gin_test_idx1 on gin_test_tbl1 using gin (i);
>
> --session1:
> select become_lock_group_leader();
> select gin_clean_pending_list('gin_test_idx');
>
> --session2:
> select become_lock_group_member(session1_pid);
> select gin_clean_pending_list('gin_test_idx1');
>
> --session3:
> select become_lock_group_leader();
> select gin_clean_pending_list('gin_test_idx1');
>
> --session4:
> select become_lock_group_member(session3_pid);
> select gin_clean_pending_list('gin_test_idx');
>
> ERROR:  deadlock detected
> DETAIL:  Process 61953 waits for ExclusiveLock on page 0 of relation
> 16399 of database 13577; blocked by process 62197.
> Process 62197 waits for ExclusiveLock on page 0 of relation 16400 of
> database 13577; blocked by process 61953.
> HINT:  See server log for query details.
>
>
> Session1 and Session3 acquire the PageLock on two different index's
> meta-pages and blocked in gdb,  meanwhile, their member tries to
> acquire the page lock as shown in the above example and it detects the
> deadlock which is solved after applying the patch.

I have modified 0001 and 0002 slightly,  Basically, instead of two
function CheckAndSetLockHeld and CheckAndReSetLockHeld, I have created
a one function.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

On Sun, Mar 15, 2020 at 6:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have modified 0001 and 0002 slightly,  Basically, instead of two
> > function CheckAndSetLockHeld and CheckAndReSetLockHeld, I have created
> > a one function.
> >
>
> +CheckAndSetLockHeld(LOCALLOCK *locallock, bool value)
>
> Can we rename the parameter as lock_held, acquired or something like
> that so that it indicates what it intends to do and probably add a
> comment for that variable atop of function?

Done

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

On Mon, Mar 16, 2020 at 8:57 AM Masahiko Sawada
<masahiko.sawada@2ndquadrant.com> wrote:
>
> On Mon, 16 Mar 2020 at 00:54, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Sun, Mar 15, 2020 at 6:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Sun, Mar 15, 2020 at 4:34 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > I have modified 0001 and 0002 slightly,  Basically, instead of two
> > > > function CheckAndSetLockHeld and CheckAndReSetLockHeld, I have created
> > > > a one function.
> > > >
> > >
> > > +CheckAndSetLockHeld(LOCALLOCK *locallock, bool value)
> > >
> > > Can we rename the parameter as lock_held, acquired or something like
> > > that so that it indicates what it intends to do and probably add a
> > > comment for that variable atop of function?
> >
> > Done
> >
>
> I've looked at the patches and ISTM these work as expected.

Thanks for verifying.

> IsRelationExtensionLockHeld and IsPageLockHeld are used only when
> assertion is enabled. So how about making CheckAndSetLockHeld work
> only if USE_ASSERT_CHECKING to avoid overheads?

That makes sense to me so updated the patch.


-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

On Mon, Mar 16, 2020 at 11:56 AM Kuntal Ghosh
<kuntalghosh.2007@gmail.com> wrote:
>
> On Mon, Mar 16, 2020 at 9:43 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > On Mon, Mar 16, 2020 at 8:57 AM Masahiko Sawada
> > <masahiko.sawada@2ndquadrant.com> wrote:
> > > IsRelationExtensionLockHeld and IsPageLockHeld are used only when
> > > assertion is enabled. So how about making CheckAndSetLockHeld work
> > > only if USE_ASSERT_CHECKING to avoid overheads?
> >
> > That makes sense to me so updated the patch.
> +1
>
> In v10-0001-Assert-that-we-don-t-acquire-a-heavyweight-lock-.patch,
>
> + * Indicate that the lock is released for a particular type of locks.
> s/lock is/locks are

Done

> + /* Indicate that the lock is acquired for a certain type of locks. */
> s/lock is/locks are

Done

>
> In v10-0002-*.patch,
>
> + * Flag to indicate if the page lock is held by this backend.  We don't
> + * acquire any other heavyweight lock while holding the page lock except for
> + * relation extension.  However, these locks are never taken in reverse order
> + * which implies that page locks will also never participate in the deadlock
> + * cycle.
> s/while holding the page lock except for relation extension/while
> holding the page lock except for relation extension and page lock

Done

> + * We don't acquire any other heavyweight lock while holding the page lock
> + * except for relation extension lock.
> Same as above

Done

>
> Other than that, the patches look good to me. I've also done some
> testing after applying the Test-group-deadlock patch provided by Amit
> earlier in the thread. It works as expected.

Thanks for testing.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Attachment

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

17 March 2020, 14:44:35

On Mon, Mar 16, 2020 at 3:24 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>

+
+ /*
+ * Indicate that the lock is released for certain types of locks
+ */
+#ifdef USE_ASSERT_CHECKING
+ CheckAndSetLockHeld(locallock, false);
+#endif
 }

 /*
@@ -1618,6 +1666,11 @@ GrantLockLocal(LOCALLOCK *locallock, ResourceOwner owner)
  locallock->numLockOwners++;
  if (owner != NULL)
  ResourceOwnerRememberLock(owner, locallock);
+
+ /* Indicate that the lock is acquired for certain types of locks. */
+#ifdef USE_ASSERT_CHECKING
+ CheckAndSetLockHeld(locallock, true);
+#endif
 }

There is no need to sprinkle USE_ASSERT_CHECKING at so many places,
having inside the new function is sufficient.  I have changed that,
added few more comments and
made minor changes.  See, what you think about attached?

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Attachment

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Dilip Kumar

Date:

17 March 2020, 16:02:24

On Tue, Mar 17, 2020 at 5:14 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Mar 16, 2020 at 3:24 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
>
> +
> + /*
> + * Indicate that the lock is released for certain types of locks
> + */
> +#ifdef USE_ASSERT_CHECKING
> + CheckAndSetLockHeld(locallock, false);
> +#endif
>  }
>
>  /*
> @@ -1618,6 +1666,11 @@ GrantLockLocal(LOCALLOCK *locallock, ResourceOwner owner)
>   locallock->numLockOwners++;
>   if (owner != NULL)
>   ResourceOwnerRememberLock(owner, locallock);
> +
> + /* Indicate that the lock is acquired for certain types of locks. */
> +#ifdef USE_ASSERT_CHECKING
> + CheckAndSetLockHeld(locallock, true);
> +#endif
>  }
>
> There is no need to sprinkle USE_ASSERT_CHECKING at so many places,
> having inside the new function is sufficient.  I have changed that,
> added few more comments and
> made minor changes.  See, what you think about attached?

Your changes look fine to me.  I have also verified all the test and
everything works fine.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

From

Amit Kapila

Date:

18 March 2020, 06:28:52

On Tue, Mar 17, 2020 at 6:32 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> Your changes look fine to me.  I have also verified all the test and
> everything works fine.
>

I have pushed the first patch.  I will push the others in coming days.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com