Thread: PG vs LLVM 12 on seawasp, next round
Hi Since seawasp's bleeding edge LLVM installation moved to "trunk 20201114 c8f4e06b 12.0.0" ~16 days ago, it has been red. Further updates didn't help it and it's now on "trunk 20201127 6ee22ca6 12.0.0". I wonder if there is something in Fabien's scripting that needs to be tweaked, perhaps a symlink name or similar. I don't follow LLVM development but I found my way to a commit[1] around the right time that mentions breaking up the OrcJIT library, so *shrug* maybe that's a clue. +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git: cannot open shared object file: No such file or directory [1] https://github.com/llvm/llvm-project/commit/1d0676b54c4e3a517719220def96dfdbc26d8048
Hi, On 2020-12-01 17:35:49 +1300, Thomas Munro wrote: > Since seawasp's bleeding edge LLVM installation moved to "trunk > 20201114 c8f4e06b 12.0.0" ~16 days ago, it has been red. Further > updates didn't help it and it's now on "trunk 20201127 6ee22ca6 > 12.0.0". I wonder if there is something in Fabien's scripting that > needs to be tweaked, perhaps a symlink name or similar. I don't > follow LLVM development but I found my way to a commit[1] around the > right time that mentions breaking up the OrcJIT library, so *shrug* > maybe that's a clue. > > +ERROR: could not load library > "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": > libLLVMOrcJIT.so.12git: cannot open shared object file: No such file > or directory It's a change in how LLVM dependencies are declared internally. Previously the 'native' component was - unintentionally - transitively included via the 'orcjit' component, but now that's not the case anymore. The attached patch should fix it, I think? Greetings, Andres Freund
Attachment
Hello Thomas, > Since seawasp's bleeding edge LLVM installation moved to "trunk > 20201114 c8f4e06b 12.0.0" ~16 days ago, it has been red. Further > updates didn't help it and it's now on "trunk 20201127 6ee22ca6 > 12.0.0". I wonder if there is something in Fabien's scripting that > needs to be tweaked, perhaps a symlink name or similar. The compiler compilation script is quite straightforward (basically, get sources, configure and compile), even for a such a moving target… The right approach is to wait for some time before looking at the issue, typically one week for the next recompilation, in case the problem evaporates, so you were right not to jump on it right away:-) Andres investigated a few days ago, managed to reproduce the issue locally, and has one line patch. I'm unsure if it should be prevently back-patched, though. -- Fabien.
Hi, On 2020-12-01 21:04:44 +0100, Fabien COELHO wrote: > Andres investigated a few days ago, managed to reproduce the issue locally, > and has one line patch. I'm unsure if it should be prevently back-patched, > though. I see no reason not to backpatch - it's more correct for past versions of LLVM as well. Greetings, Andres Freund
Hi, On 2020-12-01 12:08:10 -0800, Andres Freund wrote: > On 2020-12-01 21:04:44 +0100, Fabien COELHO wrote: > > Andres investigated a few days ago, managed to reproduce the issue locally, > > and has one line patch. I'm unsure if it should be prevently back-patched, > > though. > > I see no reason not to backpatch - it's more correct for past versions > of LLVM as well. I pushed that now. - Andres
Hi, On 2020-12-07 19:38:19 -0800, Andres Freund wrote: > On 2020-12-01 12:08:10 -0800, Andres Freund wrote: > > On 2020-12-01 21:04:44 +0100, Fabien COELHO wrote: > > > Andres investigated a few days ago, managed to reproduce the issue locally, > > > and has one line patch. I'm unsure if it should be prevently back-patched, > > > though. > > > > I see no reason not to backpatch - it's more correct for past versions > > of LLVM as well. > > I pushed that now. I hadn't checked that before, but for the last few days there's been a different failure than the one I saw earlier: +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git:cannot open shared object file: No such file or directory whereas what I fixed is about: +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": /home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so: undefinedsymbol: LLVMInitializeX86Target Changed somewhere between https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp&dt=2020-11-20%2009%3A17%3A10 and https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp&dt=2020-11-21%2023%3A17%3A11 The "no such file" error seems more like a machine local issue to me. Greetings, Andres Freund
Hello Andres, > I hadn't checked that before, but for the last few days there's been a > different failure than the one I saw earlier: > > +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git:cannot open shared object file: No such file or directory > > whereas what I fixed is about: > > +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": /home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so: undefinedsymbol: LLVMInitializeX86Target > > Changed somewhere between > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp&dt=2020-11-20%2009%3A17%3A10 > and > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp&dt=2020-11-21%2023%3A17%3A11 > > The "no such file" error seems more like a machine local issue to me. I'll look into it when I have time, which make take some time. Hopefully over the holidays. -- Fabien.
On 2020-Dec-11, Fabien COELHO wrote: > > I hadn't checked that before, but for the last few days there's been a > > different failure than the one I saw earlier: > > > > +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git:cannot open shared object file: No such file or directory > > The "no such file" error seems more like a machine local issue to me. > > I'll look into it when I have time, which make take some time. Hopefully > over the holidays. This is still happening ... Any chance you can have a look at it? -- Álvaro Herrera 39°49'30"S 73°17'W
Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> On 2020-Dec-11, Fabien COELHO wrote:
>> I'll look into it when I have time, which make take some time. Hopefully
>> over the holidays.
> This is still happening ... Any chance you can have a look at it?
If you don't have time to debug it, perhaps you could just disable
the buildfarm animal till you do. It's cluttering the buildfarm
failure report without providing useful info ...
regards, tom lane
Hello Alvaro, >>> The "no such file" error seems more like a machine local issue to me. >> >> I'll look into it when I have time, which make take some time. Hopefully >> over the holidays. > > This is still happening ... Any chance you can have a look at it? Indeed. I'll try to look (again) into it soon. I had a look but did not find anything obvious in the short time frame I had. Last two months were a little overworked for me so I let slip quite a few things. If you want to disable the animal as Tom suggests, do as you want. -- Fabien.
On Mon, Jan 18, 2021 at 09:29:53PM +0100, Fabien COELHO wrote: > >>>The "no such file" error seems more like a machine local issue to me. > >> > >>I'll look into it when I have time, which make take some time. Hopefully > >>over the holidays. > > > >This is still happening ... Any chance you can have a look at it? > > Indeed. I'll try to look (again) into it soon. I had a look but did not find > anything obvious in the short time frame I had. Last two months were a > little overworked for me so I let slip quite a few things. If you want to > disable the animal as Tom suggests, do as you want. Perhaps he was suggesting that you (buildfarm owner) disable the cron job that initiates new runs. That's what I do when one of my animals needs my intervention.
Noah Misch <noah@leadboat.com> writes:
> On Mon, Jan 18, 2021 at 09:29:53PM +0100, Fabien COELHO wrote:
>> ... Last two months were a
>> little overworked for me so I let slip quite a few things. If you want to
>> disable the animal as Tom suggests, do as you want.
> Perhaps he was suggesting that you (buildfarm owner) disable the cron job that
> initiates new runs. That's what I do when one of my animals needs my
> intervention.
Indeed. I'm not sure there even is a provision to block an animal on the
buildfarm-server side. If there is, you'd have to request that it be
manually undone after you get around to fixing the animal. Frankly,
if I were the BF admin, I would be in about as much hurry to do that
as you've been to fix it.
regards, tom lane
On Sat, Dec 12, 2020 at 8:45 AM Fabien COELHO <coelho@cri.ensmp.fr> wrote: > > +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git:cannot open shared object file: No such file or directory Bonjour Fabien, Here is the creation of llvmjit.so: g++ ... -o llvmjit.so ... -L/home/fabien/clgtk/lib ... -lLLVMOrcJIT ... That'd be from llvm-config --ldflags or similar, from this binary: checking for llvm-config... (cached) /home/fabien/clgtk/bin/llvm-config So what does ls -slap /home/fabien/clgtk/lib show? What does ldd /home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so show? What do you have in seawap's LD_LIBRARY_PATH, and is the problem that you need to add /home/fabien/clgtk/lib to it (or is it supposed to be making it into the rpath, or is it in your ld.so.conf)? PS Could you try blowing away the accache directory so we can rule out bad cached configure stuff?
Hello Thomas,
Thanks for looking at this, I'm currently far behind on many things and
not very responsive:-(
> Here is the creation of llvmjit.so:
>
> g++ ... -o llvmjit.so ... -L/home/fabien/clgtk/lib ... -lLLVMOrcJIT ...
>
> That'd be from llvm-config --ldflags or similar, from this binary:
>
> checking for llvm-config... (cached) /home/fabien/clgtk/bin/llvm-config
>
> So what does ls -slap /home/fabien/clgtk/lib show?
Plenty files and links, eg:
0 lrwxrwxrwx 1 fabien fabien 21 janv. 23 09:40 /home/fabien/clgtk/lib/libLLVMMCJIT.so ->
libLLVMMCJIT.so.12git
2140 -rw-r--r-- 1 fabien fabien 2190824 janv. 23 09:28 /home/fabien/clgtk/lib/libLLVMMCJIT.so.12git
0 lrwxrwxrwx 1 fabien fabien 22 janv. 23 09:40 /home/fabien/clgtk/lib/libLLVMOrcJIT.so ->
libLLVMOrcJIT.so.12git
40224 -rw-r--r-- 1 fabien fabien 41187208 janv. 23 09:34 /home/fabien/clgtk/lib/libLLVMOrcJIT.so.12git
Hmmm, clang recompilation has been failing for some weeks. Argh, this is
due to the recent "master" to "main" branch renaming.
> What does ldd
>
/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so
No such file of directory:-) because the directory is cleaned up.
> show? What do you have in seawap's LD_LIBRARY_PATH, and is the
> problem that you need to add /home/fabien/clgtk/lib to it
Argh. Would it be so stupid? :-( I thought the configuration stuff would
manage the link path automatically, which may be quite naïve, indeed.
> PS Could you try blowing away the accache directory so we can rule out
> bad cached configure stuff?
Hmmm. I've tried that before. I can do it again.
I've added an explicit LD_LIBRARY_PATH, which will be triggered at some
point later.
--
Fabien Coelho - CRI, MINES ParisTech
Hi, On 2021-02-15 10:05:32 +0100, Fabien COELHO wrote: > > show? What do you have in seawap's LD_LIBRARY_PATH, and is the > > problem that you need to add /home/fabien/clgtk/lib to it > > Argh. Would it be so stupid? :-( I thought the configuration stuff would > manage the link path automatically, which may be quite naïve, indeed. The only way to do that is to add an rpath annotation, and e.g. distros don't like that - there's some security implications. > I've added an explicit LD_LIBRARY_PATH, which will be triggered at some > point later. You can also do something like LDFLAGS="$LDFLAGS -Wl,-rpath,$(llvm-config --libdir)" or such. Greetings, Andres Freund
>> I've added an explicit LD_LIBRARY_PATH, which will be triggered at some >> point later. This seems to have fixed the issue. I'm sorry for the noise and quite baffled anyway, because according to my change logs it does not seem that I modified anything from my side about the dynamic library path when compiling with clang. At least I do not see a trace of that. > You can also do something like LDFLAGS="$LDFLAGS -Wl,-rpath,$(llvm-config --libdir)" > or such. I've resorted to just hardcode LD_LIBRARY_PATH alongside PATH when compiling with clang in my buildfarm script. Thanks for the tip anyway. And thanks Thomas for pointing out the fix! -- Fabien.
On 1/24/21 11:06 PM, Tom Lane wrote: > Noah Misch <noah@leadboat.com> writes: >> On Mon, Jan 18, 2021 at 09:29:53PM +0100, Fabien COELHO wrote: >>> ... Last two months were a >>> little overworked for me so I let slip quite a few things. If you want to >>> disable the animal as Tom suggests, do as you want. >> Perhaps he was suggesting that you (buildfarm owner) disable the cron job that >> initiates new runs. That's what I do when one of my animals needs my >> intervention. > Indeed. I'm not sure there even is a provision to block an animal on the > buildfarm-server side. If there is, you'd have to request that it be > manually undone after you get around to fixing the animal. Frankly, > if I were the BF admin, I would be in about as much hurry to do that > as you've been to fix it. > > It's actually very easy, but it's something I usually reserve for people who are very unresponsive to emails. As noted, disabling the crontab entry on the client side is a preferable solution. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com