Re: RFC: Additional Directory for Extensions - Mailing list pgsql-hackers

On Nov 11, 2024, at 02:16, Peter Eisentraut <peter@eisentraut.org> wrote:

> I implemented a patch along the lines Craig had suggested.

Oh, nice, thank you.

> It's a new GUC variable that is a path for extension control files.  It's called extension_control_path, and it works
exactlythe same way as dynamic_library_path.  Except that the magic token is called $system instead of $libdir. 

I assume we can bikeshed these names later. :-)

> In fact, most of the patch is refactoring the routines in dfmgr.c to not hardcode dynamic_library_path but allow
searchingfor any file in any path.  Once a control file is found, the other extension support files (script files and
auxiliarycontrol files) are looked for in the same directory. 

What about shared libraries files?

> This works pretty much fine for the use cases that have been presented here, including installing extensions outside
ofthe core installation tree (for CNPG and Postgres.app) and for testing uninstalled extensions (for Debian). 

If I understand correctly, shared modules still lie in dynamic_library_path, yes? That makes things a bit more
complicated,as the CNPG use case has to set up multiple persistent volumes to persist files put into various
directories,notably sharedir and pkglibdir. 

> - You can install extensions into alternative directories using PGXS like
>
>    make install datadir=/else/where/share pkglibdir=/else/where/lib
>
> This works.  I was hoping it would work to use
>
>    make install prefix=/else/where
>
> but that doesn't because of some details in Makefile.global.  I think we can tweak that a little bit to make that
worktoo. 

In the broader extension organization RFC I’ve been working up[1], I propose a new Makefile prefix for the destination
directoryfor an extension. It hinges on the idea that an extension has all of its files organized in a directory with
theextension name, rather than separate params for data, pkglib, bin, etc. 

> - With the current patch, if you install into datadir=/else/where/share, then you need to set
extension_control_path=/else/where/share/extension. This is a bit confusing.  Maybe we want to make the "extension"
partimplicit. 

+1

> - The biggest problem is that many extensions set in their control file
>
>    module_pathname = '$libdir/foo'
>
> This disables the use of dynamic_library_path, so this whole idea of installing an extension elsewhere won't work
thatway.  The obvious solution is that extensions change this to just 'foo'.  But this will require a lot updating work
formany extensions, or a lot of patching by packagers. 

Since that’s set at build/install time, couldn’t the definition of `$libdir` here be changed to mean “the directory
intowhich it’s being installed right now?”. Doesn’t seem necessary to search a path if the specific location is set at
installtime. 

> Maybe we could devise some sort of rule that if the extension control file is found via the path outside of $system,
thenthe leading $libdir is ignored. 
> <v0-0001-extension_control_path.patch>

This is what I propose,[1] yes. If we redefine the organization of extension files to live in a single directory named
foran extension, then once you’ve found the control files all the other files are there, too. But `$libdir` is
presumablystill meaningful, since the first time you call an extension function in a new connection, it just tries to
loadthat location, it doesn’t go through the control file search process, AFAIK. 

Perhaps I misunderstand, but I would like to talk through the implications of a more radical rethinking of extension
filelocation along the lines of the other thread[2] and the RFC I’m working up based on them both[1], especially since
thereare a few other use cases that inform it. 

A notable one is the idea Gabriele shared with me in Athens to be able to add an extension to a running CNPG pod by
simplymounting a read-only volume with all the files it requires. 

Best,

David

[1]: https://github.com/theory/justatheory/pull/7/files