Thread: PANIC: wrong buffer passed to visibilitymap_clear

On 7/22/22 10:22, 王伟(学弈) wrote:
> Hi,
> I recently find this problem while testing PG14 with sysbench.
> Then I look through the emails from pgsql-hackers and find a previous
> similary bug which
> is https://www.postgresql.org/message-id/flat/2247102.1618008027%40sss.pgh.pa.us
> <https://www.postgresql.org/message-id/flat/2247102.1618008027%40sss.pgh.pa.us>.
> But the bugfix commit(34f581c39e97e2ea237255cf75cccebccc02d477) is
> already patched to PG14.
> 

Which PG14 version / commit is this, exactly? What sysbench parameters
did you use, how likely is hitting the issue?


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

------------------------------------------------------------------
发件人：Tomas Vondra<tomas.vondra@enterprisedb.com>
日　期：2022年07月22日 18:06:21
收件人：王伟(学弈)<rogers.ww@alibaba-inc.com>; pgsql-hackers<pgsql-hackers@lists.postgresql.org>
主　题：Re: PANIC: wrong buffer passed to visibilitymap_clear



On 7/22/22 10:22, 王伟(学弈) wrote:
> Hi,
> I recently find this problem while testing PG14 with sysbench.
> Then I look through the emails from pgsql-hackers and find a previous
> similary bug which
> is https://www.postgresql.org/message-id/flat/2247102.1618008027%40sss.pgh.pa.us
> <https://www.postgresql.org/message-id/flat/2247102.1618008027%40sss.pgh.pa.us>.
> But the bugfix commit(34f581c39e97e2ea237255cf75cccebccc02d477) is
> already patched to PG14.
> 

Which PG14 version / commit is this, exactly? What sysbench parameters
did you use, how likely is hitting the issue?


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 7/22/22 14:17, 王伟(学弈) wrote:
> On 7/22/22 18:06, Tomas Vondra wrote:
>> Which PG14 version / commit is this, exactly? What sysbench parameters
>> did you use, how likely is hitting the issue?
> PG_VERSION is '14beta2'.
> The head commit id is 'e1c1c30f635390b6a3ae4993e8cac213a33e6e3f'.

Why not current REL_14_STABLE? 14beta2 is pretty old, and while I
haven't checked, perhaps this was already fixed since then.

> I have run these sysbench commands for  couple of days, but only two times to hit the issue.
> These sysbench commands are:
> prepare:
>
sysbench --tables=10 --table-size=1000000 --db-ps-mode=auto --pgsql-password=* --time=72000 --db-driver=pgsql --pgsql-port=*
>
--threads=50 --thread-init-timeout=3000 --report-interval=5 --pgsql-user=* --pgsql-host=* --pgsql-db=* --events=0 --pgsql-ignore-errors="PX000,58M01"
> ./src/lua/oltp_insert.lua prepare
> parallel execution: 
>
sysbench --tables=10 --table-size=1000000 --db-ps-mode=auto --pgsql-password=* --time=72000 --db-driver=pgsql --pgsql-port=*
> --threads=50 --thread-init-timeout=3000 --report-interval=5 --pgsql-user=* --pgsql-host=*
> --pgsql-db=* --events=0 --pgsql-ignore-errors="PX000,58M01"
> ./src/lua/oltp_insert.lua run
>
sysbench --tables=10 --table-size=1000000 --db-ps-mode=auto --pgsql-password=* --time=72000 --db-driver=pgsql --pgsql-port=*
>
--threads=50 --thread-init-timeout=3000 --report-interval=5 --pgsql-user=* --pgsql-host=*--pgsql-db=* --events=0 --pgsql-ignore-errors="PX000,58M01"
> ./src/lua/oltp_delete.lua run
>
sysbench --tables=10 --table-size=1000000 --db-ps-mode=auto --pgsql-password=* --time=72000 --db-driver=pgsql --pgsql-port=*
> --threads=50 --thread-init-timeout=3000 --report-interval=5 --pgsql-user=* --pgsql-host=*
> --pgsql-db=* --events=0 --pgsql-ignore-errors="PX000,58M01" ./src/lua/oltp_read_write.lua run
>
sysbench --tables=10 --table-size=1000000 --db-ps-mode=auto --pgsql-password=* --time=72000 --db-driver=pgsql --pgsql-port=*
> --threads=50 --thread-init-timeout=3000 --report-interval=5 --pgsql-user=* --pgsql-host=*
> --pgsql-db=* --events=0 --pgsql-ignore-errors="PX000,58M01" ./src/lua/oltp_update_index.lua run
> 

Thanks. Not sure I'll be able to do such long sysbench runs, though. Can
you try reproducing this with current REL_14_STABLE? I wonder if dumping
the WAL (using pg_waldump) might tell us more about what happened.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Jul 22, 2022 at 1:22 AM 王伟(学弈) <rogers.ww@alibaba-inc.com> wrote:
> I recently find this problem while testing PG14 with sysbench.

The line numbers from your stack trace don't match up with
REL_14_STABLE. Is this actually a fork of Postgres 14? (Oh, looks like
it's an old beta release.)

> Then I look through the emails from pgsql-hackers and find a previous similary bug which is
https://www.postgresql.org/message-id/flat/2247102.1618008027%40sss.pgh.pa.us.But the bugfix
commit(34f581c39e97e2ea237255cf75cccebccc02d477)is already patched to PG14. 

It does seem possible that there is another similar bug somewhere --
another case where we were protected by the fact that VACUUM acquired
a full cleanup lock (not just an exclusive buffer lock) during its
second heap pass. That changed in Postgres 14 (commit 8523492d4e). But
I really don't know -- almost anything is possible.

> I'm wondering whether there's another code path to lead this problem happened. Since, I take a deep dig via gdb which
turnsout that newbuffer is not euqal to buffer. In other words, the function RelationGetBufferForTuple must have been
calledjust now. 
> Besides, why didn't we re-check the flag after RelationGetBufferForTuple was called?

Recheck what flag? And at what point? It's not easy to figure this out
from your stack trace, because of the line number issues.

It would also be helpful if you told us about the specific table
involved. Though the important thing (the essential thing) is to test
today's REL_14_STABLE. There have been *lots* of bug fixes since
Postgres 14 beta2 was current.

--
Peter Geoghegan

Peter Geoghegan <pg@bowt.ie> writes:
> It would also be helpful if you told us about the specific table
> involved. Though the important thing (the essential thing) is to test
> today's REL_14_STABLE. There have been *lots* of bug fixes since
> Postgres 14 beta2 was current.

Yeah.  To be blunt, you're wasting your time and ours by testing
a year-old beta version.  The odds are respectable that the problem
is already fixed.  Even if it's not, the version skew between
what you are looking at and what we are looking at creates lots of
confusion.

            regards, tom lane