Thread: another look at macOS SIP

another look at macOS SIP

From
Peter Eisentraut
Date:
I have figured out another solution to the problem that macOS SIP
defeats the use of DYLD_LIBRARY_PATH for running the temp-install
regression tests.  It's not without problems either, but it might show a
path forward.

First of all, I think I now know the exact mechanism by which this
breakage happens.

The precise issue is that /bin/sh filters out DYLD_* environment
variables (presumably all, but at least the ones we care about) when it
starts.  If you use a shell other than /bin/sh (say, a Homebrew
installation of bash or dash), there is no problem.

But /bin/sh is hardcoded in the system() library call, so in order to
fix that, you need to override that library call.  Attached is a patch
that shows how this could be done.  It uses the DYLD_INSERT_LIBRARIES
environment variable (equivalent to LD_PRELOAD) to substitute another
version of system(), which I hacked to allow overriding /bin/sh with
another shell using the environment variable PG_REGRESS_SHELL.  That works.

There are also some other places where PostgreSQL code itself hardcodes
/bin/sh as part of system()-like functionality.  These have to be fixed
up similarly, but that's easier.

The problem now is that DYLD_INSERT_LIBRARIES requires the "flat
namespace", which isn't the default.  You can either build PostgreSQL
with -Wl,-flat_namespace, which works, but it's probably weird as a
proper solution, or you can set the environment variable
DYLD_FORCE_FLAT_NAMESPACE at run time, which also works but makes
everything brutally slow.

I think the way forward here is to get rid of all uses of system() for
calling between PostgreSQL programs.  There are only a handful of those,
and we already have well-tested replacement code like spawn_process() in
pg_regress.c that could be used.  (Perhaps we could also use that
opportunity to get rid of the need for shell quoting?)

There is a minor second issue, namely that /usr/bin/perl also filters
out DYLD_* environment variables.  This can be worked around again by
using a third-party installation of Perl.  You just need to make sure
that the "prove" program calls that installation instead of the system
one.  (I just manually edited the shebang line.  There is probably a
proper way to do it.)

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Attachment

Re: another look at macOS SIP

From
Tom Lane
Date:
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> I have figured out another solution to the problem that macOS SIP
> defeats the use of DYLD_LIBRARY_PATH for running the temp-install
> regression tests.  It's not without problems either, but it might show a
> path forward.
> ...
> The precise issue is that /bin/sh filters out DYLD_* environment
> variables (presumably all, but at least the ones we care about) when it
> starts.

Yeah, that was pretty much what we'd speculated.

> I think the way forward here is to get rid of all uses of system() for
> calling between PostgreSQL programs.

We could do that perhaps, but how are you going to get make to not use
/bin/sh while spawning subprocesses?  I don't think we want to also
reimplement make ...

> There is a minor second issue, namely that /usr/bin/perl also filters
> out DYLD_* environment variables.  This can be worked around again by
> using a third-party installation of Perl.

This is not sounding better than just turning off SIP :-(

We could, however, probably fix things so that our Perl test scripts
re-establish those environment variables internally.  We don't need
the perl processes themselves to load test libraries, just their
descendants.

Maybe a similar workaround is possible for the "make" issue?
I have a feeling it would be less flexible than what we have
today, though.

            regards, tom lane



Re: another look at macOS SIP

From
Peter Eisentraut
Date:
On 2019-09-10 19:26, Tom Lane wrote:
>> I think the way forward here is to get rid of all uses of system() for
>> calling between PostgreSQL programs.
> 
> We could do that perhaps, but how are you going to get make to not use
> /bin/sh while spawning subprocesses?  I don't think we want to also
> reimplement make ...

make is not a problem if the DYLD_* assignments are in a makefile rule
(as currently), because then make just calls a shell with a string
"DYLD_*=foo some command", which is not affected by any filtering.  It
would be a problem if you do the variable assignments in a makefile
outside a rule or outside a makefile.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



Re: another look at macOS SIP

From
Andres Freund
Date:
Hi,

On 2019-09-10 19:14:19 +0200, Peter Eisentraut wrote:
> I think the way forward here is to get rid of all uses of system() for
> calling between PostgreSQL programs.  There are only a handful of those,
> and we already have well-tested replacement code like spawn_process() in
> pg_regress.c that could be used.  (Perhaps we could also use that
> opportunity to get rid of the need for shell quoting?)

Yea, I think that'd be good, regardless of SIP.


> There is a minor second issue, namely that /usr/bin/perl also filters
> out DYLD_* environment variables.  This can be worked around again by
> using a third-party installation of Perl.  You just need to make sure
> that the "prove" program calls that installation instead of the system
> one.  (I just manually edited the shebang line.  There is probably a
> proper way to do it.)

Hm, could we just have perl code set DYLD_* again? I assume we don't
need prove itself to have it set, and for the testscripts we could just
set it in TestLib.pm or such?


Greetings,

Andres Freund



Re: another look at macOS SIP

From
Tom Lane
Date:
Andres Freund <andres@anarazel.de> writes:
> On 2019-09-10 19:14:19 +0200, Peter Eisentraut wrote:
>> There is a minor second issue, namely that /usr/bin/perl also filters
>> out DYLD_* environment variables.  This can be worked around again by
>> using a third-party installation of Perl.

> Hm, could we just have perl code set DYLD_* again? I assume we don't
> need prove itself to have it set, and for the testscripts we could just
> set it in TestLib.pm or such?

Yeah, that's what I was suggesting.  "Use another copy of Perl" doesn't
seem like an acceptable answer, or at least it's hardly better than
"turn off SIP".

            regards, tom lane



Re: another look at macOS SIP

From
Peter Eisentraut
Date:
On 2019-09-17 21:43, Tom Lane wrote:
> Yeah, that's what I was suggesting.  "Use another copy of Perl" doesn't
> seem like an acceptable answer, or at least it's hardly better than
> "turn off SIP".

In my mind, the Perl aspect of this is the most trivial part of the
problem.  "brew install perl" is probably faster than writing out this
email.  But I suppose everyone has their own workflows.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



Re: another look at macOS SIP

From
Tom Lane
Date:
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 2019-09-17 21:43, Tom Lane wrote:
>> Yeah, that's what I was suggesting.  "Use another copy of Perl" doesn't
>> seem like an acceptable answer, or at least it's hardly better than
>> "turn off SIP".

> In my mind, the Perl aspect of this is the most trivial part of the
> problem.  "brew install perl" is probably faster than writing out this
> email.  But I suppose everyone has their own workflows.

There's a not-insignificant contingent that don't wish to install
either homebrew or macports.

            regards, tom lane



Re: another look at macOS SIP

From
Robert Haas
Date:
On Tue, Sep 17, 2019 at 1:52 PM Andres Freund <andres@anarazel.de> wrote:
> On 2019-09-10 19:14:19 +0200, Peter Eisentraut wrote:
> > I think the way forward here is to get rid of all uses of system() for
> > calling between PostgreSQL programs.  There are only a handful of those,
> > and we already have well-tested replacement code like spawn_process() in
> > pg_regress.c that could be used.  (Perhaps we could also use that
> > opportunity to get rid of the need for shell quoting?)
>
> Yea, I think that'd be good, regardless of SIP.

+1, and making some progress on the SIP issue would be good, too, even
if we don't fix everything right away.  It seems entirely possible
that Apple will make this even more annoying to disable than it
already is.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company