Hi Jakub
On Tue, 10 Dec 2024 at 22:36, Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> Yay, reflink=0, that's pretty old fs ?!
This particular filesystem was created on Centos 7, and retained when
the system was upgraded to RL9. So yes probably pretty old!
> Could you get us maybe those below commands too? (or from any other directory exhibiting such errors)
>
> stat pg_tblspc/16401/PG_16_202307071/17643/
> ls -1 pg_tblspc/16401/PG_16_202307071/17643/ | wc -l
> time ls -1 pg_tblspc/16401/PG_16_202307071/17643/ | wc -l # to assess timing of getdents() call as that may something
aboutthat directory indirectly
# stat pg_tblspc/16402/PG_16_202307071/49163/
File: pg_tblspc/16402/PG_16_202307071/49163/
Size: 5177344 Blocks: 14880 IO Block: 4096 directory
Device: fd02h/64770d Inode: 4299946593 Links: 2
Access: (0700/drwx------) Uid: ( 26/postgres) Gid: ( 26/postgres)
Access: 2024-12-11 09:39:42.467802419 +0900
Modify: 2024-12-11 09:51:19.813948673 +0900
Change: 2024-12-11 09:51:19.813948673 +0900
Birth: 2024-11-25 17:37:11.812374672 +0900
# time ls -1 pg_tblspc/16402/PG_16_202307071/49163/ | wc -l
179000
real 0m0.474s
user 0m0.439s
sys 0m0.038s
> 3. Maybe somehow there is a bigger interaction between posix_fallocate() and delayed XFS's dynamic speculative
preallocationfrom many processes all writing into different partitions ? Maybe try "allocsize=1m" mount option for that
/fsand see if that helps. I'm going to speculate about XFS speculative :) pre allocations, but if we have fdcache and
are*not* closing fds, how XFS might know to abort its own speculation about streaming write ? (multiply that up to
potentiallythe number of opened fds to get an avalanche of "preallocations").
I will try to organize that. They are production systems so it might
take some time.
> 4. You can also try compiling with patch from Alvaro from [2] "0001-Add-some-debugging-around-mdzeroextend.patch", so
wemight end up having more clarity in offsets involved. If not then you could use 'strace -e fallocate -p <pid>' to get
theexact syscall.
I'll take a look at Alvaro's patch. strace sounds good, but how to
arrange to start it on the correct PG backends? There will be a
large-ish number of PG backends going at a time, only some of which
are performing imports, and they will be coming and going every so
often as the ETL application scales up and down with the load.
> 5. Another idea could be catching the kernel side stacktrace of fallocate() when it is hitting ENOSPC. E.g. with XFS
fsand attached bpftrace eBPF tracer I could get the source of the problem in my artificial reproducer, e.g
OK, I will look into that also.
Cheers
Mike
