Thread: Guiding principle for dropping LLVM versions?

Hi,

Currently we claim to support all versions of LLVM from 3.9 up.  It's
now getting quite inconvenient to test changes on older releases with
single digit major versions, because they aren't available through
usual package channels on current distributions, and frankly it feels
like pointless busy-work to build those older versions from source
(not to mention that it takes hoooouuurrs to compile that much C++).
At the other end of the window, we've also been back-patching support
for the latest LLVM versions into all supported releases, which might
make slightly more sense, but I don't know.

For the trailing end of the window, would it make sense to say that
when PostgreSQL 17 ships, it doesn't need to support any LLVM versions
that are no longer available in the default package repositories of
current major distros?

I'm trying to understand the practical constraints.  Perhaps a package
maintainer could correct me if I have this wrong.  Distros typically
support a range of releases from the past few years, and then bless
one as 'default' by making it the one you get if you install a meta
package eg 'llvm' without a number (for example, on Debian 12 this is
LLVM 14, though LLVM 13 is still available).  Having a default
encourages sharing, eg one LLVM library can be used by many different
things.  The maintainer of the PostgreSQL package then chooses which
one to link against, and it's usually the default one unless we can't
use that one yet for technical reasons (a situation that might arise
from time to time in bleeding edge distros).  So if we just knew the
*oldest default* on every live distro at release time, I assume no
package maintainer would get upset if we ripped out support for
everything older, and that'd let us vacuum a lot of old versions out
of our tree.

A more conservative horizon would be: which is the *oldest* LLVM you
can still get through the usual channels on every relevant distro, for
the benefit of people compiling from source, who for some reason want
to use a version older then the default on their distro?  I don't know
what the motivation would be.

What reason could there be to be more conservative than that?

I wonder if there is a good way to make this sort of thing more
systematic.  If we could agree on a guiding principle vaguely like the
above, then perhaps we just need a wiki page that lists relevant
distributions, versions and EOL dates, that could be used to reduce
the combinations of stuff we have to consider and make the pruning
decisions into no-brainers.

Hi,

On Thu, 2023-09-21 at 10:54 +1200, Thomas Munro wrote:
> I'm trying to understand the practical constraints.  Perhaps a package
> maintainer could correct me if I have this wrong.  Distros typically
> support a range of releases from the past few years, and then bless
> one as 'default' by making it the one you get if you install a meta
> package eg 'llvm' without a number (for example, on Debian 12 this is
> LLVM 14, though LLVM 13 is still available).  Having a default
> encourages sharing, eg one LLVM library can be used by many different
> things.  The maintainer of the PostgreSQL package then chooses which
> one to link against, and it's usually the default one unless we can't
> use that one yet for technical reasons (a situation that might arise
> from time to time in bleeding edge distros).  So if we just knew the
> *oldest default* on every live distro at release time, I assume no
> package maintainer would get upset if we ripped out support for
> everything older, and that'd let us vacuum a lot of old versions out
> of our tree.

RPM packager speaking:

Even though older LLVM versions exist on both RHEL and Fedora, they
don't provide older Clang packages, which means we have to link to the
latest release anyway (like currently Fedora 38 packages are waiting for
LLVM 16 patch, as they cannot be linked against LLVM 15)

Regards,

Regards,
--
Devrim Gündüz
Open Source Solution Architect, PostgreSQL Major Contributor
Twitter: @DevrimGunduz , @DevrimGunduzTR

Thomas Munro <thomas.munro@gmail.com> writes:
> I wonder if there is a good way to make this sort of thing more
> systematic.  If we could agree on a guiding principle vaguely like the
> above, then perhaps we just need a wiki page that lists relevant
> distributions, versions and EOL dates, that could be used to reduce
> the combinations of stuff we have to consider and make the pruning
> decisions into no-brainers.

FWIW, I think "compile older Postgres on newer infrastructure"
is a more common and more defensible scenario than "compile
newer Postgres on older infrastructure".  We've spent a ton of
effort on the latter scenario (and I've helped lead the charge
in many cases), but I think the real-world demand for it isn't
truly that high once you get beyond a year or two back.  On the
other hand, if you have an app that depends on PG 11 behavioral
details and you don't want to update it right now, you might
nonetheless need to put that server onto recent infrastructure
for practical reasons.

Thus, I think it's worthwhile to spend effort on back-patching
new-LLVM compatibility fixes into old PG branches, but I agree
that newer PG branches can drop compatibility with obsolete
LLVM versions.

LLVM is maybe not the poster child for these concerns -- for
either direction of compatibility problems, someone could build
without JIT support and not really be dead in the water.

In any case, I agree with your prior decision to not touch v11
for this.  With that branch's next release being its last,
I think the odds of introducing a bug we can't fix later
outweigh any arguable portability gain.

            regards, tom lane

> On 21 Sep 2023, at 07:28, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Thomas Munro <thomas.munro@gmail.com> writes:
>> I wonder if there is a good way to make this sort of thing more
>> systematic.  If we could agree on a guiding principle vaguely like the
>> above, then perhaps we just need a wiki page that lists relevant
>> distributions, versions and EOL dates, that could be used to reduce
>> the combinations of stuff we have to consider and make the pruning
>> decisions into no-brainers.

As someone who on occasion poke at OpenSSL compat code I would very much like a
more structured approach around dealing with dependencies.

> Thus, I think it's worthwhile to spend effort on back-patching
> new-LLVM compatibility fixes into old PG branches, but I agree
> that newer PG branches can drop compatibility with obsolete
> LLVM versions.

+1

> LLVM is maybe not the poster child for these concerns -- for
> either direction of compatibility problems, someone could build
> without JIT support and not really be dead in the water.

Right, OpenSSL on the other hand might be better example since removing TLS
support is likely a no-show.  I can see both the need to use an old OpenSSL
version in a backbranch due to certifications etc, as well as a requirement in
other cases to use the latest version due to CVE's.

> In any case, I agree with your prior decision to not touch v11
> for this.  With that branch's next release being its last,
> I think the odds of introducing a bug we can't fix later
> outweigh any arguable portability gain.

Agreed.

--
Daniel Gustafsson

On Thu, Sep 21, 2023 at 12:27 PM Devrim Gündüz <devrim@gunduz.org> wrote:
> Even though older LLVM versions exist on both RHEL and Fedora, they
> don't provide older Clang packages, which means we have to link to the
> latest release anyway (like currently Fedora 38 packages are waiting for
> LLVM 16 patch, as they cannot be linked against LLVM 15)

That's quite interesting, because it means that RHEL doesn't act as
the "lanterne route" for this, ie the most conservative relevant
distribution.

If we used Debian as a yardstick, PostgreSQL 17 wouldn't need anything
older than LLVM 14 AFAICS.  Who else do we need to ask?  Where could
we find this sort of information in machine-readable form (that is
feedback I got discussing the wiki page idea with people, ie that it
would be bound to become stale and abandoned)?

Fresh from doing battle with this stuff, I wanted to see what it would
look like if we dropped 3.9...13 in master:

 11 files changed, 12 insertions(+), 367 deletions(-)

I noticed in passing that the LLVMOrcRegisterJITEventListener
configure probes are not present in meson.

*rouge

We could go further.  With LLVM 14 as the minimum we can just use
opaque pointers everywhere, and delete more conditional code in
master.  Tested on 14-18.

I explored using the new pass manager everywhere too.  It almost
worked, but I couldn't see how to override the inlining threshold
before LLVM 16[1], even in C++, so we couldn't fix that with a
llvmjit_wrap.cpp hack.

I like this.  How can I find out if someone would shout at me for
dropping LLVM 13?

[1] https://github.com/llvm/llvm-project/commit/4fa328074efd7eefdbb314b8f6e9f855e443ca20

Rebased.  I also noticed this woefully out of date line:

-  PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-7
llvm-config-6.0 llvm-config-5.0 llvm-config-4.0 llvm-config-3.9)
+  PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-17
llvm-config-16 llvm-config-15 llvm-config-14)

On Sun, Oct 22, 2023 at 3:46 PM Xing Guo <higuoxing@gmail.com> wrote:
> Can we also check if the clang's version is compatible with llvm's version in llvm.m4? I have multiple llvm
toolchainsinstalled on my system and I have to specify the $CLANG and $LLVM_CONFIG variables each time I build the
serveragainst a toolchain that is not present in $PATH. If one of the variables is missing, the build system will pick
upa default one whose version might not be compatible with the other. E.g., If we use clang-16 and llvm-config-15,
therewill be issues when creating indexes for bitcodes at the end of installation. 

Hmm.  Problems that occur to me:

1.  We need to decide if our rule is that clang must be <= llvm, or
==.  I think this question has been left unanswered in the past when
it has come up.  So far I think <= would be enough to avoid the error
you showed but can we find where this policy (ie especially
commitments for future releases) is written down in LLVM literature?
2.  Apple's clang lies about its version (I don't know the story
behind that, but my wild guess is that someone from marketing wanted
the compiler's version numbers to align with xcode's version numbers?
they're off by 1 or something like that).

Another idea could be to produce some bitcode with clang, and then
check if a relevant LLVM tool can deal with it.

Hi,

On October 21, 2023 7:46:17 PM PDT, Xing Guo <higuoxing@gmail.com> wrote:
>Can we also check if the clang's version is compatible with llvm's version
>in llvm.m4? I have multiple llvm toolchains installed on my system and I
>have to specify the $CLANG and $LLVM_CONFIG variables each time I build the
>server against a toolchain that is not present in $PATH. If one of the
>variables is missing, the build system will pick up a default one whose
>version might not be compatible with the other. E.g., If we use clang-16
>and llvm-config-15, there will be issues when creating indexes for bitcodes
>at the end of installation.

It's unfortunately not that obvious to figure out what is compatible and what not. Older clang versions work, except if
tooold. Newer versions sometimes work. We could perhaps write a script that will find many, but not all,
incompatibilities. 

For the meson build I made it just use clang belonging to the llvm install - but that's very painful when building
againstan assert enabled llvm, clang is slower by an order of magnitude or so. 

I wonder if we should change the search order to 1) CLANG, iff explicitly specified, 2) use explicitly specified or
inferredllvm-config, 3) only if that didn't find clang, search path.  

>wrote:
>
>> Rebased.  I also noticed this woefully out of date line:
>>
>> -  PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-7
>> llvm-config-6.0 llvm-config-5.0 llvm-config-4.0 llvm-config-3.9)
>> +  PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-17
>> llvm-config-16 llvm-config-15 llvm-config-14)
>>

It's outdated, but not completely absurd - back then often no llvm-config -> llvm-config-XY was installed, but these
daysthere pretty much always is. 


Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Here are some systematic rules I'd like to propose to anchor this
stuff to reality and avoid future doubt and litigation:

1.  Build farm animals testing LLVM determine the set of OSes and LLVM
versions we consider.
2.  We exclude OSes that will be out of full vendor support when a
release ships.
3.  We exclude OSes that don't bless an LLVM release (eg macOS running
an arbitrarily picked version), and animals running only to cover
ancient LLVM compiled from source for coverage (Andres's sid
menagerie).

By these rules we can't require LLVM 14 for another year, because
Ubuntu and Amazon Linux are standing in the way*:

    animal     |  arch   | llvm_version |   os   | os_release | end_of_support
---------------+---------+--------------+--------+------------+----------------
 branta        | s390x   | 10.0.0       | Ubuntu | 20.04      | 2025-04-01
 splitfin      | aarch64 | 10.0.0       | Ubuntu | 20.04      | 2025-04-01
 urutau        | s390x   | 10.0.0       | Ubuntu | 20.04      | 2025-04-01
 massasauga    | aarch64 | 11.1.0       | Amazon | 2          | 2025-06-30
 snakefly      | aarch64 | 11.1.0       | Amazon | 2          | 2025-06-30
 sarus         | s390x   | 14.0.0       | Ubuntu | 22.04      | 2027-06-01
 shiner        | aarch64 | 14.0.0       | Ubuntu | 22.04      | 2027-06-01
 turbot        | aarch64 | 14.0.0       | Ubuntu | 22.04      | 2027-06-01
 lora          | s390x   | 15.0.7       | RHEL   | 9          | 2027-05-31
 mamushi       | s390x   | 15.0.7       | RHEL   | 9          | 2027-05-31
 nicator       | ppc64le | 15.0.7       | Alma   | 9          | 2027-05-31
 oystercatcher | aarch64 | 15.0.7       | Alma   | 9          | 2027-05-31

Ideally more distros would be present in this vacuum-horizon decision
table, but I don't think it'd change the conclusion: 10 is the
trailing edge.  Therefore the attached patch scales back its ambition
to that release.  Tested on LLVM 10-18.

If I pushed this we'd need to disable or upgrade the following to
avoid failure in configure on master:

   animal    |        arch        | llvm_version |   os   | os_release
| end_of_support
-------------+--------------------+--------------+--------+------------+----------------
 dragonet    | x86_64             | 3.9.1        | Debian | sid        |
 phycodurus  | x86_64             | 3.9.1        | Debian | sid        |
 desmoxytes  | x86_64             | 4.0.1        | Debian | sid        |
 petalura    | x86_64             | 4.0.1        | Debian | sid        |
 mantid      | x86_64             | 5.0.1        | CentOS | 7
| 2019-08-06
 idiacanthus | x86_64             | 5.0.2        | Debian | sid        |
 pogona      | x86_64             | 5.0.2        | Debian | sid        |
 cotinga     | s390x              | 6.0.0        | Ubuntu | 18.04
| 2023-06-01
 vimba       | aarch64            | 6.0.0        | Ubuntu | 18.04
| 2023-06-01
 komodoensis | x86_64             | 6.0.1        | Debian | sid        |
 topminnow   | mips64el; -mabi=32 | 6.0.1        | Debian | 8
| 2018-06-17
 xenodermus  | x86_64             | 6.0.1        | Debian | sid        |
 alimoche    | aarch64            | 7.0.1        | Debian | 10
| 2022-09-10
 blackneck   | aarch64            | 7.0.1        | Debian | 10
| 2022-09-10
 bonito      | ppc64le            | 7.0.1        | Fedora | 29
| 2019-11-26

*Some distros announce EOL date by month without saying which day, so
in my data collecting operation I just punched in the first of the
month, *shrug*

Thomas Munro <thomas.munro@gmail.com> writes:
> Here are some systematic rules I'd like to propose to anchor this
> stuff to reality and avoid future doubt and litigation:

> 1.  Build farm animals testing LLVM determine the set of OSes and LLVM
> versions we consider.
> 2.  We exclude OSes that will be out of full vendor support when a
> release ships.
> 3.  We exclude OSes that don't bless an LLVM release (eg macOS running
> an arbitrarily picked version), and animals running only to cover
> ancient LLVM compiled from source for coverage (Andres's sid
> menagerie).

Seems generally reasonable.  Maybe rephrase 3 as "We consider only
an OS release's default LLVM version"?  Or a bit more forgivingly,
"... only LLVM versions available from the OS vendor"?  Also,
what's an OS vendor?  You rejected macOS which is fine, but
I think the packages available from MacPorts or Homebrew should
be considered.

You could imagine somebody trying to game the system by standing up a
buildfarm animal running some really arbitrary combination of versions
--- but what would be the point?  I think we can deal with that
when/if it happens.  But "macOS running an LLVM version available
from MacPorts" doesn't seem arbitrary.

            regards, tom lane

On Wed, Oct 25, 2023 at 7:12 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > 3.  We exclude OSes that don't bless an LLVM release (eg macOS running
> > an arbitrarily picked version), and animals running only to cover
> > ancient LLVM compiled from source for coverage (Andres's sid
> > menagerie).
>
> Seems generally reasonable.  Maybe rephrase 3 as "We consider only
> an OS release's default LLVM version"?  Or a bit more forgivingly,
> "... only LLVM versions available from the OS vendor"?  Also,
> what's an OS vendor?  You rejected macOS which is fine, but
> I think the packages available from MacPorts or Homebrew should
> be considered.

OK.  For me the key differences are that they are independent of OS
releases and time lines, they promptly add new releases, they have a
wide back-catalogue of the old releases and they let the user decide
which to use.  So I don't think they constrain us and it makes no
sense to try to apply 'end of support' logic to them.

https://formulae.brew.sh/formula/llvm
https://ports.macports.org/search/?q=llvm&name=on

(Frustratingly, the ancient releases of clang don't actually seem to
work on MacPorts at least on aarch64, and they tell you so when you
try to install them.)

The BSDs may be closer to macOS in that respect too, since they have
ports separate from OS releases and they offer a rolling and wide
range of LLVMs and generally default to a very new one, so I don't
think they constrain us either.  It's really Big Linux that is
drawing the lines in the sand here, though (unusually) not
RHEL-and-frenemies as they've opted for rolling to the latest in every
minor release as Devrim explained.

Hi,

On Thu, 2023-10-19 at 08:13 +1300, Thomas Munro wrote:
> If we used Debian as a yardstick, PostgreSQL 17 wouldn't need anything
> older than LLVM 14 AFAICS.  Who else do we need to ask? 

LLVM 15 is the minimum one for the platforms that I build the packages
on. So LLVM >= 14 is great for HEAD.

Regards,
--
Devrim Gündüz
Open Source Solution Architect, PostgreSQL Major Contributor
Twitter: @DevrimGunduz , @DevrimGunduzTR

So it sounds like we're in agreement that it is time to require LLVM
10+ in master.  Could the owners (CC'd) of the following animals
please turn off --with-llvm on master (and future 17+ branches), or
consider upgrading to a modern OS release?  Otherwise they'll turn
red.

   animal    |        arch        | llvm_version |   os   | os_release
| end_of_support
-------------+--------------------+--------------+--------+------------+----------------
 mantid      | x86_64             | 5.0.1        | CentOS | 7
| 2019-08-06
 cotinga     | s390x              | 6.0.0        | Ubuntu | 18.04
| 2023-06-01
 vimba       | aarch64            | 6.0.0        | Ubuntu | 18.04
| 2023-06-01
 topminnow   | mips64el; -mabi=32 | 6.0.1        | Debian | 8
| 2018-06-17
 alimoche    | aarch64            | 7.0.1        | Debian | 10
| 2022-09-10
 blackneck   | aarch64            | 7.0.1        | Debian | 10
| 2022-09-10

And of course Andres would need to do the same for his coverage
animals in that range:

   animal    |        arch        | llvm_version |   os   | os_release
| end_of_support
-------------+--------------------+--------------+--------+------------+----------------
 dragonet    | x86_64             | 3.9.1        | Debian | sid        |
 phycodurus  | x86_64             | 3.9.1        | Debian | sid        |
 desmoxytes  | x86_64             | 4.0.1        | Debian | sid        |
 petalura    | x86_64             | 4.0.1        | Debian | sid        |
 idiacanthus | x86_64             | 5.0.2        | Debian | sid        |
 pogona      | x86_64             | 5.0.2        | Debian | sid        |
 komodoensis | x86_64             | 6.0.1        | Debian | sid        |
 xenodermus  | x86_64             | 6.0.1        | Debian | sid        |

Hi,

On 2023-11-02 10:46:52 +1300, Thomas Munro wrote:
> So it sounds like we're in agreement that it is time to require LLVM
> 10+ in master.  Could the owners (CC'd) of the following animals
> please turn off --with-llvm on master (and future 17+ branches), or
> consider upgrading to a modern OS release?  Otherwise they'll turn
> red.
>
> And of course Andres would need to do the same for his coverage
> animals in that range:
> 
>    animal    |        arch        | llvm_version |   os   | os_release
> | end_of_support
> -------------+--------------------+--------------+--------+------------+----------------
>  dragonet    | x86_64             | 3.9.1        | Debian | sid        |
>  phycodurus  | x86_64             | 3.9.1        | Debian | sid        |
>  desmoxytes  | x86_64             | 4.0.1        | Debian | sid        |
>  petalura    | x86_64             | 4.0.1        | Debian | sid        |
>  idiacanthus | x86_64             | 5.0.2        | Debian | sid        |
>  pogona      | x86_64             | 5.0.2        | Debian | sid        |
>  komodoensis | x86_64             | 6.0.1        | Debian | sid        |
>  xenodermus  | x86_64             | 6.0.1        | Debian | sid        |

Would you want me to do this now or just before you apply the patch?

I think I should stand up a few more replacement animals to cover older llvm
versions with assertions enabled...

Greetings,

Andres Freund

On 25.10.23 07:47, Thomas Munro wrote:
> Ideally more distros would be present in this vacuum-horizon decision
> table, but I don't think it'd change the conclusion: 10 is the
> trailing edge.  Therefore the attached patch scales back its ambition
> to that release.  Tested on LLVM 10-18.

This patch and the associated reasoning look good to me.  I think this 
is good to go for PG17.

2024-01 Commitfest.

Hi, This patch has a CF status of "Ready for Committer", but it looks
like it failed when the CFbot test for it was last run [1]. Please
have a look and post an updated version..

======
[1] https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/46/4640

Kind Regards,
Peter Smith.

Thanks all for the discussion.  Pushed.  A few build farm animals will
now fail in the configure step as discussed, and need some adjustment
(ie disable LLVM or upgrade to LLVM 10+ for the master branch).

Next year I think we should be able to do a much bigger cleanup, by
moving to LLVM 14+.

On Thu, Jan 25, 2024 at 4:44 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> ... A few build farm animals will
> now fail in the configure step as discussed, and need some adjustment
> (ie disable LLVM or upgrade to LLVM 10+ for the master branch).

Owners pinged.

On 1/25/24 13:41, Thomas Munro wrote:
> On Thu, Jan 25, 2024 at 4:44 PM Thomas Munro <thomas.munro@gmail.com> wrote:
>> ... A few build farm animals will
>> now fail in the configure step as discussed, and need some adjustment
>> (ie disable LLVM or upgrade to LLVM 10+ for the master branch).
> 
> Owners pinged.

I think I fixed up the 4 or 6 under my name...

Regards,
Mark