Thread: Question regarding dynamic_library_path
Why does postgres maintain a loader logic of its own? I can understand that the dynamic_library_path is necessary in order to configure everything in one single place. But why not just merge it with the LD_LIBRARY_PATH (or PATH on Windows) and then let dlopen do the rest using a stripped filename? The reason I ask is because I run into problems as soon as I have a module that in turn depends on other shared libraries. It doesn't help much that they are reachable through the dynamic_library_path since that path never is made known to the OS loader mechanisms. I'll be happty to submit some code to do the actual path merging (i.e omitting duplicates etc.). Kind regards, Thomas Hallgren
Thomas Hallgren wrote: > Why does postgres maintain a loader logic of its own? I can understand that > the dynamic_library_path is necessary in order to configure everything in > one single place. But why not just merge it with the LD_LIBRARY_PATH (or > PATH on Windows) and then let dlopen do the rest using a stripped filename? > > The reason I ask is because I run into problems as soon as I have a module > that in turn depends on other shared libraries. It doesn't help much that > they are reachable through the dynamic_library_path since that path never is > made known to the OS loader mechanisms. > > I'll be happty to submit some code to do the actual path merging (i.e > omitting duplicates etc.). I think the idea is that you want to specify the path in the config file, after the app has already started. I don't think you can modify the environment variable after the app has started, and even if you can, it seems simpler to just do it in our code and specify the exact path rather than having it poke around in whatever LD_LIBRARY_PATH is set to. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian wrote: >Thomas Hallgren wrote: > > >>Why does postgres maintain a loader logic of its own? I can understand that >>the dynamic_library_path is necessary in order to configure everything in >>one single place. But why not just merge it with the LD_LIBRARY_PATH (or >>PATH on Windows) and then let dlopen do the rest using a stripped filename? >> >>The reason I ask is because I run into problems as soon as I have a module >>that in turn depends on other shared libraries. It doesn't help much that >>they are reachable through the dynamic_library_path since that path never is >>made known to the OS loader mechanisms. >> >>I'll be happty to submit some code to do the actual path merging (i.e >>omitting duplicates etc.). >> >> > >I think the idea is that you want to specify the path in the config >file, after the app has already started. I don't think you can modify >the environment variable after the app has started, and even if you can, >it seems simpler to just do it in our code and specify the exact path >rather than having it poke around in whatever LD_LIBRARY_PATH is set to. > > > And many people just hate playing games with LD_LIBRARY_PATH, and even more hate apps that do it. cheers andrew
"Thomas Hallgren" <thhal@mailblocks.com> writes:
> Why does postgres maintain a loader logic of its own? I can understand that
> the dynamic_library_path is necessary in order to configure everything in
> one single place. But why not just merge it with the LD_LIBRARY_PATH (or
> PATH on Windows) and then let dlopen do the rest using a stripped filename?
What LD_LIBRARY_PATH? The above statement is so full of system-specific
assumptions that it seems hopeless.
regards, tom lane
> "Thomas Hallgren" <thhal@mailblocks.com> writes: > > Why does postgres maintain a loader logic of its own? I can understand that > > the dynamic_library_path is necessary in order to configure everything in > > one single place. But why not just merge it with the LD_LIBRARY_PATH (or > > PATH on Windows) and then let dlopen do the rest using a stripped filename? > > What LD_LIBRARY_PATH? The above statement is so full of system-specific > assumptions that it seems hopeless. > > regards, tom lane > The LD_LIBRARY_PATH or PATH depending on system (Posix or Windows) that is effective when the dlopen function is called. All OS'es where shared libraries are possible have something similar. The general idea is to let the OS find the shared library rather than have the backend do it by itself. There's a flaw in the current design. IMHO, it would be a good thing to improve it. regards, Thomas Hallgren
From: "Bruce Momjian" <pgman@candle.pha.pa.us> > > I think the idea is that you want to specify the path in the config > file, after the app has already started. I don't think you can modify > the environment variable after the app has started, and even if you can, > it seems simpler to just do it in our code and specify the exact path > rather than having it poke around in whatever LD_LIBRARY_PATH is set to. > Ok, that makes sense. But I think most systems have the ability to change the environment of a running process (setenv on Posix systems) and the current design, simple as it is, doesn't work in all cases. regards, Thomas Hallgren
"Thomas Hallgren" <thhal@mailblocks.com> writes:
> The LD_LIBRARY_PATH or PATH depending on system (Posix or Windows) that is
> effective when the dlopen function is called. All OS'es where shared
> libraries are possible have something similar.
The variations among platforms are great enough that I don't think you
can make such an assertion.
(To take one example, HPUX does have a variable like this --- they call
it SHLIB_PATH --- but it *is not used* unless a flag enabling it was
provided when the shlib was linked, and I think most people avoid doing
that. Certainly we don't enable SHLIB_PATH in the shlibs we build in
the standard Postgres distribution. Furthermore, AFAICT shl_load()
requires an exact path spec for the initial shared library; it won't
do a path search for that, only for dependencies.)
> The general idea is to let the OS find the shared library rather than have
> the backend do it by itself. There's a flaw in the current design. IMHO, it
> would be a good thing to improve it.
I think we'd just be buying into a lot of pain and platform-specific
behavior. Up to now we have not depended on knowing how to
environmentally influence the dynamic linker, but with your proposal
Postgres would be immediately broken anywhere that we didn't have that
right. And there are places we *couldn't* get it right, see HPUX.
I'm much more concerned about whether shared library loading works at
all than about whether it's possible for one shlib to autoload another.
If you want $libdir to be part of LD_LIBRARY_PATH on your platform, why
don't you just set that up in the postmaster start script?
regards, tom lane
"Andrew Dunstan" <andrew@dunslane.net> wrote: > And many people just hate playing games with LD_LIBRARY_PATH, and even > more hate apps that do it. > The current design forces me and anyone else who wants a module that depends on shared libraries to play these games. My suggestion is that we hide this with a better implementation. I do *not* want to expose LD_LIBRARY_PATH nor force anyone to use it. I simply want the dynamic_library_path propagated to the loader in a correct fasion. regards, Thomas Hallgren
> The variations among platforms are great enough that I don't think you > can make such an assertion. > > (To take one example, HPUX does have a variable like this --- they call > it SHLIB_PATH --- but it *is not used* unless a flag enabling it was > provided when the shlib was linked, and I think most people avoid doing > that. Certainly we don't enable SHLIB_PATH in the shlibs we build in > the standard Postgres distribution. Furthermore, AFAICT shl_load() > requires an exact path spec for the initial shared library; it won't > do a path search for that, only for dependencies.) > Here's an excerpt from the HPUX manual pages on dlopen(3C). "First, any directories specified by the environment variable LD_LIBRARY_PATH are searched. Next, any directories specified by SHLIB_PATH are searched. Then, any directories specified by a DT_RPATH entry in the .dynamic section of the original program object are searched. Finally, the directories /usr/lib/hpux32 in 32-bit mode and /usr/lib/hpux64 in 64-bit mode are searched." > > The general idea is to let the OS find the shared library rather than > have > > the backend do it by itself. There's a flaw in the current design. IMHO, > it > > would be a good thing to improve it. > > I think we'd just be buying into a lot of pain and platform-specific > behavior. Up to now we have not depended on knowing how to > environmentally influence the dynamic linker, but with your proposal > Postgres would be immediately broken anywhere that we didn't have that > right. And there are places we *couldn't* get it right, see HPUX. > I'm much more concerned about whether shared library loading works at > all than about whether it's possible for one shlib to autoload another. > I don't agree. Propagating the dynamic_library_path to the loader will make it work the way people would expect it to. Failing to do so is confusing since the path you specified appears to be incorrect. The concept of shared libs that are dependent on other shared libs is not uncommon. I think that following the platform specific behavior is what PostgreSQL *should* do in this case. And HPUX, as you can see, is no obstacle. I actually sincerely doubt that you'll find a platform where shared libraries are enabled and this cannot be done. > If you want $libdir to be part of LD_LIBRARY_PATH on your platform, why > don't you just set that up in the postmaster start script? Yes, there's always a workaround for the problem and this is what I do today. My suggestion is based on the assumption that it would be nice if the dynamic_library_path actually worked for libraries loading libraries. The discussion is pointless if that assumption is incorrect. Kind regards, Thomas Hallgren
"Thomas Hallgren" <thhal@mailblocks.com> writes:
>> ... Furthermore, AFAICT shl_load()
>> requires an exact path spec for the initial shared library; it won't
>> do a path search for that, only for dependencies.)
>>
> Here's an excerpt from the HPUX manual pages on dlopen(3C).
HPUX which? There is no dlopen() in HPUX 10.20, which is what
I'm running here; nor do we use dlopen() in any HPUX version
given the current state of port/dynloader/hpux.c.
regards, tom lane
Oh, sorry. This HP-UX 11.x. But you can get the same using shl_load (HP-UX 10.x) and pass the flag DYNAMIC_PATH provided the executable is linked with +s. So it's still no showstopper. If you do find that it is impossible on some older OS (HP-UX 11 arrived 4 years ago), then why not just disable this feature on that particular version of the OS and make a note for the user that in order to get it, it's time to upgrade? IMO it's not a good policy to prevent new good features just to keep customers sticking to old stuff from feeling bad not getting them. You need to assure backward compatibility to keep everyone happy. Kind regards, Thomas Hallgren ----- Original Message ----- From: "Tom Lane" <tgl@sss.pgh.pa.us> To: "Thomas Hallgren" <thhal@mailblocks.com> Cc: <pgsql-hackers@postgresql.org> Sent: Wednesday, June 09, 2004 7:24 AM Subject: Re: [HACKERS] Question regarding dynamic_library_path > "Thomas Hallgren" <thhal@mailblocks.com> writes: > >> ... Furthermore, AFAICT shl_load() > >> requires an exact path spec for the initial shared library; it won't > >> do a path search for that, only for dependencies.) > >> > > Here's an excerpt from the HPUX manual pages on dlopen(3C). > > HPUX which? There is no dlopen() in HPUX 10.20, which is what > I'm running here; nor do we use dlopen() in any HPUX version > given the current state of port/dynloader/hpux.c. > > regards, tom lane >
Hi, in linux you can change LD_LIBRARY_PATH in a running process, but it does not help. The library-loader initializes himself at process-startup and changing LD_LIBRARY_PATH do not change the actual path, the process is using for dlopen. Tommi Mäkitalo Am Dienstag, 8. Juni 2004 19:14 schrieb Thomas Hallgren: > From: "Bruce Momjian" <pgman@candle.pha.pa.us> > > > I think the idea is that you want to specify the path in the config > > file, after the app has already started. I don't think you can modify > > the environment variable after the app has started, and even if you can, > > it seems simpler to just do it in our code and specify the exact path > > rather than having it poke around in whatever LD_LIBRARY_PATH is set to. > > Ok, that makes sense. But I think most systems have the ability to change > the environment of a running process (setenv on Posix systems) and the > current design, simple as it is, doesn't work in all cases. > > regards, > > Thomas Hallgren > > > > ---------------------------(end of broadcast)--------------------------- > TIP 7: don't forget to increase your free space map settings
"Tommi Maekitalo" <t.maekitalo@epgmbh.de> wrote: > Hi, > > in linux you can change LD_LIBRARY_PATH in a running process, but it does not > help. The library-loader initializes himself at process-startup and changing > LD_LIBRARY_PATH do not change the actual path, the process is using for > dlopen. > That's bad. My nice suggestion just scattered into pieces :-) But then again, perhaps not. Isn't the fact that "dynamic_library_path" can be changed at any time by a user a serious security flaw? It enables the user to load a module from an arbitrary place and then execute what's in that module with postmaster priviligies. I.e. there's nothing preventing such a user to access all data in the database using low-level C-routines and bypass the security imposed by PostgreSQL. IMHO, the DB-admin should be able to limit what's loaded by defining loader constraints. Regardless of the previous discussion I would suggest the following: 1. Prohibit use of paths containing a directory component in SQL. 2. Make dynamic_library_path a PGC_POSTMASTER variable. This would close the security hole and give the DB Admin full control over what a user can and cannot load. Now, back to my original suggestion. Using the current desing, the postmaster would still not be able to set the LD_LIBRARY_PATH from the dynamic_library_path on Unix since all it does when a new connection is established is a fork. So on a Unix machine with Linux behavior the postmaster would need to parse the config file on startup, merge dynamic_library_path with LD_LIBRARY_PATH (or whatever is applicable), set the environment, and then do re-exec using some flag indicating "stage2". Stage 2 would do exactly what's done today but ignore the dynamic_library_path setting completely. I realize that this is stretching it a little :-). The part concerning the security-leak is important though. Kind regards, Thomas Hallgren
Thomas Hallgren wrote: >Isn't the fact that "dynamic_library_path" can >be changed at any time by a user a serious security flaw? > > It's not "a user". Only the superuser can change it. cheers andrew
And only the super user can use directory components in a module name? regards, Thomas Hallgren "Andrew Dunstan" <andrew@dunslane.net> wrote in message news:40C6FADD.3000300@dunslane.net... > > > Thomas Hallgren wrote: > > >Isn't the fact that "dynamic_library_path" can > >be changed at any time by a user a serious security flaw? > > > > > It's not "a user". Only the superuser can change it. > > cheers > > andrew > > ---------------------------(end of broadcast)--------------------------- > TIP 2: you can get off all lists at once with the unregister command > (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) >
"Thomas Hallgren" <thhal@mailblocks.com> writes:
> And only the super user can use directory components in a module name?
Only superusers can create C functions in the first place, so quibbling
about what paths they can use doesn't seem that useful ...
regards, tom lane
I agree. I wasn't aware of that restriction. regards, Thomas Hallgren ----- Original Message ----- From: "Tom Lane" <tgl@sss.pgh.pa.us> To: "Thomas Hallgren" <thhal@mailblocks.com> Cc: <pgsql-hackers@postgresql.org> Sent: Wednesday, June 09, 2004 15:56 Subject: Re: [HACKERS] Question regarding dynamic_library_path > "Thomas Hallgren" <thhal@mailblocks.com> writes: > > And only the super user can use directory components in a module name? > > Only superusers can create C functions in the first place, so quibbling > about what paths they can use doesn't seem that useful ... > > regards, tom lane >