Re: lockup in parallel hash join on dikkop (freebsd 14.0-current) - Mailing list pgsql-hackers

From Tomas Vondra
Subject Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)
Date
Msg-id 2ca4d9b5-ebcd-cc6b-8535-3edbd9dcf630@enterprisedb.com
Whole thread Raw
In response to Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)  (Alexander Lakhin <exclusion@gmail.com>)
Responses Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)
List pgsql-hackers

On 9/1/23 10:00, Alexander Lakhin wrote:
> Hello Thomas,
> 
> 31.08.2023 14:15, Thomas Munro wrote:
> 
>> We have a signal that is pending and not blocked, so I don't
>> immediately know why poll() hasn't returned control.
> 
> When I worked at the Postgres Pro company, we observed a similar lockup
> under rather specific conditions (we used Elbrus CPU and the specific
> Elbrus
> compiler (lcc) based on edg).
> I managed to reproduce that lockup and Anton Voloshin investigated it.
> The issue was caused by the compiler optimization in WaitEventSetWait():
>     waiting = true;
> ...
>     while (returned_events == 0)
>     {
> ...
>         if (set->latch && set->latch->is_set)
>         {
> ...
>             break;
>         }
> 
> In that case, compiler decided that it may place the read
> "set->latch->is_set" before the write "waiting = true".
> (Placing "pg_compiler_barrier();" just after "waiting = true;" fixed the
> issue for us.)
> I can't provide more details for now, but maybe you could look at the
> binary
> code generated on the target platform to confirm or reject my guess.
> 

Hmmm, I'm not very good at reading the binary code, but here's what
objdump produced for WaitEventSetWait. Maybe someone will see what the
issue is.

I thought about maybe just adding the barrier in the code, but then how
would we know it's the issue and this fixed it? It happens so rarely we
can't make any conclusions from a couple runs of tests.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachment

pgsql-hackers by date:

Previous
From: Peter Eisentraut
Date:
Subject: Re: Move bki file pre-processing from initdb to bootstrap
Next
From: "Hayato Kuroda (Fujitsu)"
Date:
Subject: RE: [PoC] pg_upgrade: allow to upgrade publisher node