Thread: another look at macOS SIP
I have figured out another solution to the problem that macOS SIP defeats the use of DYLD_LIBRARY_PATH for running the temp-install regression tests. It's not without problems either, but it might show a path forward. First of all, I think I now know the exact mechanism by which this breakage happens. The precise issue is that /bin/sh filters out DYLD_* environment variables (presumably all, but at least the ones we care about) when it starts. If you use a shell other than /bin/sh (say, a Homebrew installation of bash or dash), there is no problem. But /bin/sh is hardcoded in the system() library call, so in order to fix that, you need to override that library call. Attached is a patch that shows how this could be done. It uses the DYLD_INSERT_LIBRARIES environment variable (equivalent to LD_PRELOAD) to substitute another version of system(), which I hacked to allow overriding /bin/sh with another shell using the environment variable PG_REGRESS_SHELL. That works. There are also some other places where PostgreSQL code itself hardcodes /bin/sh as part of system()-like functionality. These have to be fixed up similarly, but that's easier. The problem now is that DYLD_INSERT_LIBRARIES requires the "flat namespace", which isn't the default. You can either build PostgreSQL with -Wl,-flat_namespace, which works, but it's probably weird as a proper solution, or you can set the environment variable DYLD_FORCE_FLAT_NAMESPACE at run time, which also works but makes everything brutally slow. I think the way forward here is to get rid of all uses of system() for calling between PostgreSQL programs. There are only a handful of those, and we already have well-tested replacement code like spawn_process() in pg_regress.c that could be used. (Perhaps we could also use that opportunity to get rid of the need for shell quoting?) There is a minor second issue, namely that /usr/bin/perl also filters out DYLD_* environment variables. This can be worked around again by using a third-party installation of Perl. You just need to make sure that the "prove" program calls that installation instead of the system one. (I just manually edited the shebang line. There is probably a proper way to do it.) -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachment
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> I have figured out another solution to the problem that macOS SIP
> defeats the use of DYLD_LIBRARY_PATH for running the temp-install
> regression tests. It's not without problems either, but it might show a
> path forward.
> ...
> The precise issue is that /bin/sh filters out DYLD_* environment
> variables (presumably all, but at least the ones we care about) when it
> starts.
Yeah, that was pretty much what we'd speculated.
> I think the way forward here is to get rid of all uses of system() for
> calling between PostgreSQL programs.
We could do that perhaps, but how are you going to get make to not use
/bin/sh while spawning subprocesses? I don't think we want to also
reimplement make ...
> There is a minor second issue, namely that /usr/bin/perl also filters
> out DYLD_* environment variables. This can be worked around again by
> using a third-party installation of Perl.
This is not sounding better than just turning off SIP :-(
We could, however, probably fix things so that our Perl test scripts
re-establish those environment variables internally. We don't need
the perl processes themselves to load test libraries, just their
descendants.
Maybe a similar workaround is possible for the "make" issue?
I have a feeling it would be less flexible than what we have
today, though.
regards, tom lane
On 2019-09-10 19:26, Tom Lane wrote: >> I think the way forward here is to get rid of all uses of system() for >> calling between PostgreSQL programs. > > We could do that perhaps, but how are you going to get make to not use > /bin/sh while spawning subprocesses? I don't think we want to also > reimplement make ... make is not a problem if the DYLD_* assignments are in a makefile rule (as currently), because then make just calls a shell with a string "DYLD_*=foo some command", which is not affected by any filtering. It would be a problem if you do the variable assignments in a makefile outside a rule or outside a makefile. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hi, On 2019-09-10 19:14:19 +0200, Peter Eisentraut wrote: > I think the way forward here is to get rid of all uses of system() for > calling between PostgreSQL programs. There are only a handful of those, > and we already have well-tested replacement code like spawn_process() in > pg_regress.c that could be used. (Perhaps we could also use that > opportunity to get rid of the need for shell quoting?) Yea, I think that'd be good, regardless of SIP. > There is a minor second issue, namely that /usr/bin/perl also filters > out DYLD_* environment variables. This can be worked around again by > using a third-party installation of Perl. You just need to make sure > that the "prove" program calls that installation instead of the system > one. (I just manually edited the shebang line. There is probably a > proper way to do it.) Hm, could we just have perl code set DYLD_* again? I assume we don't need prove itself to have it set, and for the testscripts we could just set it in TestLib.pm or such? Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes:
> On 2019-09-10 19:14:19 +0200, Peter Eisentraut wrote:
>> There is a minor second issue, namely that /usr/bin/perl also filters
>> out DYLD_* environment variables. This can be worked around again by
>> using a third-party installation of Perl.
> Hm, could we just have perl code set DYLD_* again? I assume we don't
> need prove itself to have it set, and for the testscripts we could just
> set it in TestLib.pm or such?
Yeah, that's what I was suggesting. "Use another copy of Perl" doesn't
seem like an acceptable answer, or at least it's hardly better than
"turn off SIP".
regards, tom lane
On 2019-09-17 21:43, Tom Lane wrote: > Yeah, that's what I was suggesting. "Use another copy of Perl" doesn't > seem like an acceptable answer, or at least it's hardly better than > "turn off SIP". In my mind, the Perl aspect of this is the most trivial part of the problem. "brew install perl" is probably faster than writing out this email. But I suppose everyone has their own workflows. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 2019-09-17 21:43, Tom Lane wrote:
>> Yeah, that's what I was suggesting. "Use another copy of Perl" doesn't
>> seem like an acceptable answer, or at least it's hardly better than
>> "turn off SIP".
> In my mind, the Perl aspect of this is the most trivial part of the
> problem. "brew install perl" is probably faster than writing out this
> email. But I suppose everyone has their own workflows.
There's a not-insignificant contingent that don't wish to install
either homebrew or macports.
regards, tom lane
On Tue, Sep 17, 2019 at 1:52 PM Andres Freund <andres@anarazel.de> wrote: > On 2019-09-10 19:14:19 +0200, Peter Eisentraut wrote: > > I think the way forward here is to get rid of all uses of system() for > > calling between PostgreSQL programs. There are only a handful of those, > > and we already have well-tested replacement code like spawn_process() in > > pg_regress.c that could be used. (Perhaps we could also use that > > opportunity to get rid of the need for shell quoting?) > > Yea, I think that'd be good, regardless of SIP. +1, and making some progress on the SIP issue would be good, too, even if we don't fix everything right away. It seems entirely possible that Apple will make this even more annoying to disable than it already is. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company