RFC: Extension Packaging & Lookup - Mailing list pgsql-hackers

Hackers,

Back at the end of August, I promised[1]:

> I’ll try to put some thought into a more formal proposal in a new thread next week. Unless your Gabriele beats me to
it😂. 

I guess I should get off my butt and do it. So let’s do this. Here’s what I propose.

*   When an extension is installed, all of its files should live in a single directory. These include:

    *   The Control file in directory describes extension
    *   Subdirectories for SQL, shared libraries, docs, binaries
        (also locales and tsearch dictionaries?)

*   Next, there should be an extension lookup path. The first item in the path is the compile-time default, and ideally
wouldinclude only core extensions. Subsequent paths would be set by a GUC, similar to dynamic_library_path, but only
forextensions (including their shared libraries). 

*   Modify PGXS (or create a new installer CLI used by PGXS?) to install an extension according to this pattern. Allow
thespecification of a prefix. This should differ from the current `PREFIX`, in that the values of `sharedir`,
`pkglibdir`,etc. would not be fully-duplicated under the prefix, but point to a directory used in the extension path.
Forexample, when installing an extension need “pair", something like 

        make install BASE_DIR=/opt/pg/extension

    Would create `/opt/pg/extension/pair`, rather than `/opt/pg/extension/$(pg_config --sharedir)/extension/pair`.

*   Perhaps there could also be an option to symlink binary files or man pages to keep paths simple.

*   For CREATE EXTENSION, Postgres would look for an extension on the file system by directory name in each of the
extensionpaths instead of control file name. It would then find the control file in that directory and the necessary
SQLand shared library files in the `sql` and `lib` subdirectories of that directory. 

*   Binary-only extensions might also be installed here; the difference is they have no control file. The LOAD command
andshared_preload_libraries would need to know to look here, too. 

The basic idea, then, is three-fold:

1.  This pattern is more like a packaging pattern than CREATE EXTENSION-specific, since it includes other types of
extensions

2.  All the files for a given extension live within a single directory, making it easier to reason about what’s
installedand what’s not. 

3.  These extension packages can live in multiple paths.

Some examples. Core extensions, like citext, would live in, say, $(pg_config --extensiondir)/citext), and have a
structuresuch as: 

```
citext
├── citext.control
├── lib
│   ├── citext.dylib
│   └── bitcode
│   ├── citext
│   │   └── citext.bc
│   └── citext.index.bc
└── sql
    ├── citext--1.0--1.1.sql
    ├── citext--1.1--1.2.sql
    ├── citext--1.2--1.3.sql
    ├── citext--1.3--1.4.sql
    ├── citext--1.4--1.5.sql
    ├── citext--1.4.sql
    └── citext--1.5--1.6.sql
```

Third-party extensions would live in one or more other directories on the file system, unknown at compile time, but set
inthe extension path GUC and accessible to/owned by the Postgres system user. Let’s say we set `/opt/pgxn` as one of
thepaths. Within that directory, we might have a directory for a pure SQL extension in a a directory named “pair” that
lookslike this: 

```
pair
├── LICENSE.md
├── README.md
├── pair.control
├── doc
│   ├── html
│   │   └── pair.html
│   └── pair.md
└── sql
├── pair--1.0--1.1.sql
└── pair--1.1.sql
```

A binary application like pg_top would live in the pg_top directory, structured something like:

```
pg_top
├── HISTORY.rst
├── INSTALL.rst
├── LICENSE
├── README.rst
├── bin
|   └── pg_top
└── doc
    └── man
        └── man3
            └── pg_top.3
```

And a C extension like semver would live in the semver directory and be structured something like:

```
semver
├── LICENSE
├── README.md
├── semver.control
├── doc
│   └── semver.md
├── lib
│   ├── semver.dylib
│   ├── bitcode
│   └── semver
│   │   └── semver.bc
│   └── semver.index.bc
└── sql
    ├── semver--1.0--1.1.sql
    └── semver--1.1.sql
```

Thoughts?

Best,

David

[1]: https://www.postgresql.org/message-id/D30A91FA-A6D4-4737-941F-0BBB2984B730%40justatheory.com