Re: Direct I/O - Mailing list pgsql-hackers

Andrew Dunstan <andrew@dunslane.net> writes:

On 2023-04-12 We 10:23, Dagfinn Ilmari Mannsåker wrote:

Andrew Dunstan<andrew@dunslane.net>  writes:

On 2023-04-12 We 01:48, Thomas Munro wrote:

On Wed, Apr 12, 2023 at 3:04 PM Thomas Munro<thomas.munro@gmail.com>   wrote:

On Wed, Apr 12, 2023 at 2:56 PM Christoph Berg<myon@debian.org>   wrote:

I'm hitting a panic in t_004_io_direct. The build is running on
overlayfs on tmpfs/ext4 (upper/lower) which is probably a weird
combination but has worked well for building everything over the last
decade. On Debian unstable:

PANIC:  could not open file "pg_wal/000000010000000000000001": Invalid argument

... I have a new idea:  perhaps it is possible to try
to open a file with O_DIRECT from perl, and if it fails like that,
skip the test.  Looking into that now.

I think I have that working OK.  Any Perl hackers want to comment on
my use of IO::File (copied from examples on the internet that showed
how to use O_DIRECT)?  I am not much of a perl hacker but according to
my package manager, IO/File.pm came with perl itself.  And the Fcntl
eval trick that I copied from File::stat, and the perl-critic
suppression that requires?

I think you can probably replace a lot of the magic here by simply saying


if (Fcntl->can("O_DIRECT")) ...

Fcntl->can() is true for all constants that Fcntl knows about, whether
or not they are defined for your OS. `defined &O_DIRECT` is the simplest
check, see my other reply to Thomas.

My understanding was that Fcntl only exported constants known to the
OS. That's certainly what its docco suggests, e.g.:

    By default your system's F_* and O_* constants (eg, F_DUPFD and
O_CREAT)
    and the FD_CLOEXEC constant are exported into your namespace.

It's a bit more magical than that (this is Perl after all).  They are
all exported (which implicitly creates stubs visible to `->can()`,
similarly to forward declarations like `sub O_FOO;`), but only the
defined ones (`#ifdef O_FOO` is true) are defined (`defined &O_FOO` is
true).  The rest fall through to an AUTOLOAD¹ function that throws an
exception for undefined ones.

Here's an example (Fcntl knows O_RAW, but Linux does not define it):
    $ perl -E '        use strict; use Fcntl;        say "can", main->can("O_RAW") ? "" : "not";        say defined &O_RAW ? "" : "not ", "defined";        say O_RAW;'    can    not defined    Your vendor has not defined Fcntl macro O_RAW, used at -e line 4

While O_DIRECT is defined:
    $ perl -E '        use strict; use Fcntl;        say "can", main->can("O_DIRECT") ? "" : "not";        say defined &O_DIRECT ? "" : "not ", "defined";        say O_DIRECT;'    can    defined    16384

And O_FOO is unknown to Fcntl (the parens on `O_FOO()q are to make it
not a bareword, which would be a compile error under `use strict;` when
the sub doesn't exist at all):
    $ perl -E '        use strict; use Fcntl;        say "can", main->can("O_FOO") ? "" : "not";        say defined &O_FOO ? "" : "not ", "defined";        say O_FOO();'    cannot    not defined    Undefined subroutine &main::O_FOO called at -e line 4.

*grumble* a bit too magical for my taste. Thanks for the correction.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com