pg_upgrade cannot create btrfs clones on linux kernel 6.8.0 - Mailing list pgsql-bugs

From Michael Misiewicz
Subject pg_upgrade cannot create btrfs clones on linux kernel 6.8.0
Date
Msg-id 63b5aaafb0f89a714be6415d33844de21d682770@hey.com
Whole thread Raw
Responses Re: pg_upgrade cannot create btrfs clones on linux kernel 6.8.0
List pgsql-bugs
Hello all, 

I am attempting to upgrade a postgres cluster from 16 to 17. I'm running ubuntu 24.04 on x86_64, with postgres installed from the postgres apt repositories. 

I'm invoking pg_upgrade like so:
/usr/lib/postgresql/17/bin/pg_upgrade --clone -d /var/lib/postgresql/16/main -b /usr/lib/postgresql/16/bin -B  /usr/lib/postgresql/17/bin -D /var/lib/postgresql/17/main -o '-c config_file=/etc/postgresql/16/main/postgresql.conf' -O '-c config_file=/etc/postgresql/17/main/postgresql.conf'

and it fails with:
could not clone file between old and new data directories: Invalid argument
Failure, exiting

I ran the following strace command to find out where the problem is happening:
sudo -u postgres strace -f -e trace=copy_file_range,clone,ioctl,%file -s 2000 -v /usr/lib/postgresql/17/bin/pg_upgrade --clone -d /var/lib/postgresql/16/main -b /usr/lib/postgresql/16/bin -B  /usr/lib/postgresql/17/bin -D /var/lib/postgresql/17/main -o '-c config_file=/etc/postgresql/16/main/postgresql.conf' -O '-c config_file=/etc/postgresql/17/main/postgresql.conf' 2>/tmp/pg_strace.log

And grepping through the log I found what looks like the problem here:
--
[pid 1153372] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 8
[pid 1153372] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpq.so.5", O_RDONLY|O_CLOEXEC) = 8
[pid 1153261] unlink("/var/lib/postgresql/17/main/PG_VERSION.clonetest") = 0
[pid 1153261] openat(AT_FDCWD, "/var/lib/postgresql/16/main/PG_VERSION", O_RDONLY) = 4
[pid 1153261] openat(AT_FDCWD, "/var/lib/postgresql/17/main/PG_VERSION.clonetest", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
[pid 1153261] ioctl(5, BTRFS_IOC_CLONE or FICLONE, 4) = -1 EINVAL (Invalid argument)

Which I believe can be traced back to this commit 3a769d8239afdc003c91a56d2d8d5adfadacda5d

I have validated that my filesystem supports reflinks:
$  dd of=tempfile if=/dev/random bs=1M count=24
24+0 records in
24+0 records out
25165824 bytes (25 MB, 24 MiB) copied, 0.0806355 s, 312 MB/s
$  cp --reflink=always tempfile tempfile.reflink
$  btrfs filesystem du -s tempfile tempfile.reflink     Total   Exclusive  Set shared  Filename  24.00MiB       0.00B    24.00MiB  tempfile  24.00MiB       0.00B    24.00MiB  tempfile.reflink

And there's nothing interesting in `dmesg`. 

I ran this same command today on a macOS system (using apfs) and it worked great. I have no idea how to fix this problem and I'm curious if anyone has any pointers. 

Thanks,
Michael

pgsql-bugs by date:

Previous
From: Tom Lane
Date:
Subject: Re: BUG #18756: Postgres corruption
Next
From: Tomas Vondra
Date:
Subject: Re: pg_upgrade cannot create btrfs clones on linux kernel 6.8.0