Re: Use streaming read API in ANALYZE - Mailing list pgsql-hackers
From | Nazir Bilal Yavuz |
---|---|
Subject | Re: Use streaming read API in ANALYZE |
Date | |
Msg-id | CAN55FZ016mLb_g6APBDAozZRUN4L3GN6wrqYd4PJ-7FVPruGKw@mail.gmail.com Whole thread Raw |
In response to | Re: Use streaming read API in ANALYZE (Heikki Linnakangas <hlinnaka@iki.fi>) |
Responses |
Re: Use streaming read API in ANALYZE
Re: Use streaming read API in ANALYZE |
List | pgsql-hackers |
Hi, Thank you for looking into this! On Wed, 3 Apr 2024 at 20:17, Heikki Linnakangas <hlinnaka@iki.fi> wrote: > > On 03/04/2024 13:31, Nazir Bilal Yavuz wrote: > > Streaming API has been committed but the committed version has a minor > > change, the read_stream_begin_relation function takes Relation instead > > of BufferManagerRelation now. So, here is a v5 which addresses this > > change. > > I'm getting a repeatable segfault / assertion failure with this: > > postgres=# CREATE TABLE tengiga (i int, filler text) with (fillfactor=10); > CREATE TABLE > postgres=# insert into tengiga select g, repeat('x', 900) from > generate_series(1, 1400000) g; > INSERT 0 1400000 > postgres=# set default_statistics_target = 10; ANALYZE tengiga; > SET > ANALYZE > postgres=# set default_statistics_target = 100; ANALYZE tengiga; > SET > ANALYZE > postgres=# set default_statistics_target =1000; ANALYZE tengiga; > SET > server closed the connection unexpectedly > This probably means the server terminated abnormally > before or while processing the request. > > TRAP: failed Assert("BufferIsValid(hscan->rs_cbuf)"), File: > "heapam_handler.c", Line: 1079, PID: 262232 > postgres: heikki postgres [local] > ANALYZE(ExceptionalCondition+0xa8)[0x56488a0de9d8] > postgres: heikki postgres [local] > ANALYZE(heapam_scan_analyze_next_block+0x63)[0x5648899ece34] > postgres: heikki postgres [local] ANALYZE(+0x2d3f34)[0x564889b6af34] > postgres: heikki postgres [local] ANALYZE(+0x2d2a3a)[0x564889b69a3a] > postgres: heikki postgres [local] ANALYZE(analyze_rel+0x33e)[0x564889b68fa9] > postgres: heikki postgres [local] ANALYZE(vacuum+0x4b3)[0x564889c2dcc0] > postgres: heikki postgres [local] ANALYZE(ExecVacuum+0xd6f)[0x564889c2d7fe] > postgres: heikki postgres [local] > ANALYZE(standard_ProcessUtility+0x901)[0x564889f0b8b9] > postgres: heikki postgres [local] > ANALYZE(ProcessUtility+0x136)[0x564889f0afb1] > postgres: heikki postgres [local] ANALYZE(+0x6728c8)[0x564889f098c8] > postgres: heikki postgres [local] ANALYZE(+0x672b3b)[0x564889f09b3b] > postgres: heikki postgres [local] ANALYZE(PortalRun+0x320)[0x564889f09015] > postgres: heikki postgres [local] ANALYZE(+0x66b2c6)[0x564889f022c6] > postgres: heikki postgres [local] > ANALYZE(PostgresMain+0x80c)[0x564889f06fd7] > postgres: heikki postgres [local] ANALYZE(+0x667876)[0x564889efe876] > postgres: heikki postgres [local] > ANALYZE(postmaster_child_launch+0xe6)[0x564889e1f4b3] > postgres: heikki postgres [local] ANALYZE(+0x58e68e)[0x564889e2568e] > postgres: heikki postgres [local] ANALYZE(+0x58b7f0)[0x564889e227f0] > postgres: heikki postgres [local] > ANALYZE(PostmasterMain+0x152b)[0x564889e2214d] > postgres: heikki postgres [local] ANALYZE(+0x4444b4)[0x564889cdb4b4] > /lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f7d83b6724a] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f7d83b67305] > postgres: heikki postgres [local] ANALYZE(_start+0x21)[0x564889971a61] > 2024-04-03 20:15:49.157 EEST [262101] LOG: server process (PID 262232) > was terminated by signal 6: Aborted I realized the same error while working on Jakub's benchmarking results. Cause: I was using the nblocks variable to check how many blocks will be returned from the streaming API. But I realized that sometimes the number returned from BlockSampler_Init() is not equal to the number of blocks that BlockSampler_Next() will return as BlockSampling algorithm decides how many blocks to return on the fly by using some random seeds. There are a couple of solutions I thought of: 1- Use BlockSampler_HasMore() instead of nblocks in the main loop in the acquire_sample_rows(): Streaming API uses this function to prefetch block numbers. BlockSampler_HasMore() will reach to the end first as it is used while prefetching, so it will start to return false while there are still buffers to return from the streaming API. That will cause some buffers at the end to not be processed. 2- Expose something (function, variable etc.) from the streaming API to understand if the read is finished and there is no buffer to return: I think this works but I am not sure if the streaming API allows something like that. 3- Check every buffer returned from the streaming API, if it is invalid stop the main loop in the acquire_sample_rows(): This solves the problem but there will be two if checks for each buffer returned, - in heapam_scan_analyze_next_block() to check if the returned buffer is invalid - to break main loop in acquire_sample_rows() if heapam_scan_analyze_next_block() returns false One of the if cases can be bypassed by moving heapam_scan_analyze_next_block()'s code to the main loop in the acquire_sample_rows(). I implemented the third solution, here is v6. -- Regards, Nazir Bilal Yavuz Microsoft
Attachment
pgsql-hackers by date: