Thread: [PATCH] Add session_preload_libraries configuration parameter
This is like shared_preload_libraries except that it takes effect at backend start and can be changed without a full postmaster restart. It is like local_preload_libraries except that it is still only settable by a superuser. This can be a better way to load modules such as auto_explain. Since there are now three preload parameters, regroup the documentation a bit. Put all parameters into one section, explain common functionality only once, update the descriptions to reflect current and future realities. --- This is a reboot of the ideas discussed in http://www.postgresql.org/message-id/1358251492.401.5.camel@vanquo.pezone.net and http://www.postgresql.org/message-id/515357F4.6000307@gmx.net during the 2013-01 commit fest. doc/src/sgml/auto-explain.sgml | 5 +-doc/src/sgml/config.sgml | 281 ++++++++++++++++++++++---------------src/backend/tcop/postgres.c | 2 +-src/backend/utils/init/miscinit.c | 6 +-src/backend/utils/misc/guc.c | 19 ++-src/include/miscadmin.h | 3 +-src/include/utils/guc_tables.h | 1 +7 files changed, 193 insertions(+), 124 deletions(-) diff --git a/doc/src/sgml/auto-explain.sgml b/doc/src/sgml/auto-explain.sgml index 03b2309..cd0d6d8 100644 --- a/doc/src/sgml/auto-explain.sgml +++ b/doc/src/sgml/auto-explain.sgml @@ -24,7 +24,8 @@ <title>auto_explain</title></programlisting> (You must be superuser to do that.) More typical usage isto preload - it into all sessions by including <literal>auto_explain</> in + it into some or all sessions by including <literal>auto_explain</> in + <xref linkend="guc-session-preload-libraries"> or <xref linkend="guc-shared-preload-libraries"> in <filename>postgresql.conf</>. Then you can track unexpectedly slow queries no matter when they happen. Of course thereis a price in overhead for @@ -185,7 +186,7 @@ <title>Configuration Parameters</title><programlisting># postgresql.conf -shared_preload_libraries = 'auto_explain' +session_preload_libraries = 'auto_explain'auto_explain.log_min_duration = '3s'</programlisting> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index c7d84b5..b8e766d 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -1252,66 +1252,6 @@ <title>Kernel Resource Usage</title> </para> </listitem> </varlistentry> - - <varlistentry id="guc-shared-preload-libraries" xreflabel="shared_preload_libraries"> - <term><varname>shared_preload_libraries</varname> (<type>string</type>)</term> - <indexterm> - <primary><varname>shared_preload_libraries</> configuration parameter</primary> - </indexterm> - <listitem> - <para> - This variable specifies one or more shared libraries - to be preloaded at server start. For example, - <literal>'$libdir/mylib'</literal> would cause - <literal>mylib.so</> (or on some platforms, - <literal>mylib.sl</>) to be preloaded from the installation's - standard library directory. - All library names are converted to lower case unless double-quoted. - If more than one library is to be loaded, separate their names - with commas. This parameter can only be set at server start. - </para> - - <para> - <productname>PostgreSQL</productname> procedural language - libraries can be preloaded in this way, typically by using the - syntax <literal>'$libdir/plXXX'</literal> where - <literal>XXX</literal> is <literal>pgsql</>, <literal>perl</>, - <literal>tcl</>, or <literal>python</>. - </para> - - <para> - By preloading a shared library, the library startup time is avoided - when the library is first used. However, the time to start each new - server process might increase slightly, even if that process never - uses the library. So this parameter is recommended only for - libraries that will be used in most sessions. - </para> - - <note> - <para> - On Windows hosts, preloading a library at server start will not reduce - the time required to start each new server process; each server process - will re-load all preload libraries. However, <varname>shared_preload_libraries - </varname> is still useful on Windows hosts because some shared libraries may - need to perform certain operations that only take place at postmaster start - (for example, a shared library may need to reserve lightweight locks - or shared memory and you can't do that after the postmaster has started). - </para> - </note> - <para> - If a specified library is not found, - the server will fail to start. - </para> - - <para> - Every PostgreSQL-supported library has a <quote>magic - block</> that is checked to guarantee compatibility. - For this reason, non-PostgreSQL libraries cannot be - loaded in this way. - </para> - </listitem> - </varlistentry> - </variablelist> </sect2> @@ -5494,6 +5434,171 @@ <title>Locale and Formatting</title> </variablelist> </sect2> + + <sect2 id="runtime-config-client-preload"> + <title>Shared Library Preloading</title> + + <para> + Several settings are available for preloading shared libraries into the + server, in order to load additional functionality or achieve performance + benefits. For example, a setting of + <literal>'$libdir/mylib'</literal> would cause + <literal>mylib.so</> (or on some platforms, + <literal>mylib.sl</>) to be preloaded from the installation's standard + library directory. The differences between the settings are when they + take effect and what privileges are required to change them. + </para> + + <para> + <productname>PostgreSQL</productname> procedural language libraries can + be preloaded in this way, typically by using the + syntax <literal>'$libdir/plXXX'</literal> where + <literal>XXX</literal> is <literal>pgsql</>, <literal>perl</>, + <literal>tcl</>, or <literal>python</>. + </para> + + <para> + For each parameter, if more than one library is to be loaded, separate + their names with commas. All library names are converted to lower case + unless double-quoted. + </para> + + <para> + Only shared libraries specifically intended to be used with PostgreSQL + can be loaded this way. Every PostgreSQL-supported library has + a <quote>magic block</> that is checked to guarantee compatibility. For + this reason, non-PostgreSQL libraries cannot be loaded in this way. You + might be able to use operating-system facilities such + as <envar>LD_PRELOAD</envar> for that. + </para> + + <para> + In general, refer to the documentation of a specific module for the + recommended way to load that module. + </para> + + <variablelist> + <varlistentry id="guc-local-preload-libraries" xreflabel="local_preload_libraries"> + <term><varname>local_preload_libraries</varname> (<type>string</type>)</term> + <indexterm> + <primary><varname>local_preload_libraries</> configuration parameter</primary> + </indexterm> + <indexterm> + <primary><filename>$libdir/plugins</></primary> + </indexterm> + <listitem> + <para> + This variable specifies one or more shared libraries that are to be + preloaded at connection start. This parameter cannot be changed after + the start of a particular session. If a specified library is not + found, the connection attempt will fail. + </para> + + <para> + This option can be set by any user. Because of that, the libraries + that can be loaded are restricted to those appearing in the + <filename>plugins</> subdirectory of the installation's + standard library directory. (It is the database administrator's + responsibility to ensure that only <quote>safe</> libraries + are installed there.) Entries in <varname>local_preload_libraries</> + can specify this directory explicitly, for example + <literal>$libdir/plugins/mylib</literal>, or just specify + the library name — <literal>mylib</literal> would have + the same effect as <literal>$libdir/plugins/mylib</literal>. + </para> + + <para> + Unless a module is specifically designed to be used in this way by + non-superusers, this is usually not the right setting to use. Look + at <xref linkend="guc-session-preload-libraries"> instead. + </para> + </listitem> + </varlistentry> + + + <varlistentry id="guc-session-preload-libraries" xreflabel="session_preload_libraries"> + <term><varname>session_preload_libraries</varname> (<type>string</type>)</term> + <indexterm> + <primary><varname>session_preload_libraries</> configuration parameter</primary> + </indexterm> + <listitem> + <para> + This variable specifies one or more shared libraries that are to be + preloaded at connection start. Only superusers can change this setting. + The parameter value only takes effect at the start of the connection. + Subsequent changes have no effect. If a specified library is not + found, the connection attempt will fail. + </para> + + <para> + The intent of this feature is to allow debugging or + performance-measurement libraries to be loaded into specific sessions + without an explicit + <command>LOAD</> command being given. For + example, <xref linkend="auto-explain"> could be enabled for all + sessions under a given user name by setting this parameter + with <command>ALTER ROLE SET</>. Also, this parameter can be changed + without restarting the server (but changes only take effect when a new + session is started), so it is easier to add new modules this way, even + if they should apply to all sessions. + </para> + + <para> + Unlike <xref linkend="guc-shared-preload-libraries">, there is no large + performance advantage to loading a library at session start rather than + when it is first used. There is some advantage, however, when + connection pooling is used. + </para> + </listitem> + </varlistentry> + + <varlistentry id="guc-shared-preload-libraries" xreflabel="shared_preload_libraries"> + <term><varname>shared_preload_libraries</varname> (<type>string</type>)</term> + <indexterm> + <primary><varname>shared_preload_libraries</> configuration parameter</primary> + </indexterm> + <listitem> + <para> + This variable specifies one or more shared libraries to be preloaded at + server start. with commas. This parameter can only be set at server + start. If a specified library is not found, the server will fail to + start. + </para> + + <para> + Some libraries need to perform certain operations that can only take + place at postmaster start, such as allocating shared memory, reserving + light-weight locks, or starting background workers. Those libraries + must be loaded at server start through this parameter. See the + documentation of each library for details. + </para> + + <para> + Other libraries can also be preloaded. By preloading a shared library, + the library startup time is avoided when the library is first used. + However, the time to start each new server process might increase + slightly, even if that process never uses the library. So this + parameter is recommended only for libraries that will be used in most + sessions. Also, changing this parameter requires a server restart, so + this is not the right setting to use for short-term debugging tasks, + say. Use <xref linkend="guc-session-preload-libraries"> for that + instead. + </para> + + <note> + <para> + On Windows hosts, preloading a library at server start will not reduce + the time required to start each new server process; each server process + will re-load all preload libraries. However, <varname>shared_preload_libraries + </varname> is still useful on Windows hosts for libraries that need to + perform operations at postmaster start time.. + </para> + </note> + </listitem> + </varlistentry> + </variablelist> + </sect2> + <sect2 id="runtime-config-client-other"> <title>Other Defaults</title> @@ -5565,62 +5670,6 @@ <title>Other Defaults</title> </listitem> </varlistentry> - <varlistentry id="guc-local-preload-libraries" xreflabel="local_preload_libraries"> - <term><varname>local_preload_libraries</varname> (<type>string</type>)</term> - <indexterm> - <primary><varname>local_preload_libraries</> configuration parameter</primary> - </indexterm> - <indexterm> - <primary><filename>$libdir/plugins</></primary> - </indexterm> - <listitem> - <para> - This variable specifies one or more shared libraries that are - to be preloaded at connection start. If more than one library - is to be loaded, separate their names with commas. All library - names are converted to lower case unless double-quoted. - This parameter cannot be changed after the start of a particular - session. - </para> - - <para> - Because this is not a superuser-only option, the libraries - that can be loaded are restricted to those appearing in the - <filename>plugins</> subdirectory of the installation's - standard library directory. (It is the database administrator's - responsibility to ensure that only <quote>safe</> libraries - are installed there.) Entries in <varname>local_preload_libraries</> - can specify this directory explicitly, for example - <literal>$libdir/plugins/mylib</literal>, or just specify - the library name — <literal>mylib</literal> would have - the same effect as <literal>$libdir/plugins/mylib</literal>. - </para> - - <para> - Unlike <xref linkend="guc-shared-preload-libraries">, there is no - performance advantage to loading a library at session - start rather than when it is first used. Rather, the intent of - this feature is to allow debugging or performance-measurement - libraries to be loaded into specific sessions without an explicit - <command>LOAD</> command being given. For example, debugging could - be enabled for all sessions under a given user name by setting - this parameter with <command>ALTER ROLE SET</>. - </para> - - <para> - If a specified library is not found, - the connection attempt will fail. - </para> - - <para> - Every PostgreSQL-supported library has a <quote>magic - block</> that is checked to guarantee compatibility. - For this reason, non-PostgreSQL libraries cannot be - loaded in this way. - </para> - </listitem> - </varlistentry> - </variablelist> </sect2> </sect1> diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 31ea313..da0c5c9 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3738,7 +3738,7 @@ * process any libraries that should be preloaded at backend start (this * likewise can't bedone until GUC settings are complete) */ - process_local_preload_libraries(); + process_session_preload_libraries(); /* * Send this backend's cancellation info to the frontend. diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c index cb78caf..ed514f6 100644 --- a/src/backend/utils/init/miscinit.c +++ b/src/backend/utils/init/miscinit.c @@ -1222,6 +1222,7 @@ * GUC variables: lists of library names to be preloaded at postmaster * start and at backend start*/ +char *session_preload_libraries_string = NULL;char *shared_preload_libraries_string = NULL;char *local_preload_libraries_string= NULL; @@ -1318,8 +1319,11 @@ * process any libraries that should be preloaded at backend start */void -process_local_preload_libraries(void) +process_session_preload_libraries(void){ + load_libraries(session_preload_libraries_string, + "session_preload_libraries", + false); load_libraries(local_preload_libraries_string, "local_preload_libraries", true); diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index ea16c64..f05b801 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -589,6 +589,8 @@ static char *config_enum_get_options(struct config_enum * record, gettext_noop("Client ConnectionDefaults / Statement Behavior"), /* CLIENT_CONN_LOCALE */ gettext_noop("Client Connection Defaults / Localeand Formatting"), + /* CLIENT_CONN_PRELOAD */ + gettext_noop("Client Connection Defaults / Shared Library Preloading"), /* CLIENT_CONN_OTHER */ gettext_noop("ClientConnection Defaults / Other Defaults"), /* LOCK_MANAGEMENT */ @@ -2745,7 +2747,18 @@ static char *config_enum_get_options(struct config_enum * record, }, { - {"shared_preload_libraries", PGC_POSTMASTER, RESOURCES_KERNEL, + {"session_preload_libraries", PGC_SUSET, CLIENT_CONN_PRELOAD, + gettext_noop("Lists shared libraries to preload into each backend."), + NULL, + GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY + }, + &session_preload_libraries_string, + "", + NULL, NULL, NULL + }, + + { + {"shared_preload_libraries", PGC_POSTMASTER, CLIENT_CONN_PRELOAD, gettext_noop("Lists shared librariesto preload into server."), NULL, GUC_LIST_INPUT | GUC_LIST_QUOTE | GUC_SUPERUSER_ONLY @@ -2756,8 +2769,8 @@ static char *config_enum_get_options(struct config_enum * record, }, { - {"local_preload_libraries", PGC_BACKEND, CLIENT_CONN_OTHER, - gettext_noop("Lists shared libraries to preload into each backend."), + {"local_preload_libraries", PGC_BACKEND, CLIENT_CONN_PRELOAD, + gettext_noop("Lists unprivileged shared libraries to preload into each backend."), NULL, GUC_LIST_INPUT | GUC_LIST_QUOTE }, diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index be3add9..77861ea 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -402,6 +402,7 @@ extern void BaseInit(void);/* in utils/init/miscinit.c */extern bool IgnoreSystemIndexes;extern PGDLLIMPORTbool process_shared_preload_libraries_in_progress; +extern char *session_preload_libraries_string;extern char *shared_preload_libraries_string;extern char *local_preload_libraries_string; @@ -437,7 +438,7 @@ extern void TouchSocketLockFiles(void);extern void AddToDataDirLockFile(int target_line, const char *str);externvoid ValidatePgVersion(const char *path);extern void process_shared_preload_libraries(void); -extern void process_local_preload_libraries(void); +extern void process_session_preload_libraries(void);extern void pg_bindtextdomain(const char *domain);extern bool has_rolreplication(Oidroleid); diff --git a/src/include/utils/guc_tables.h b/src/include/utils/guc_tables.h index 8dcdd4b..4f1f6e0 100644 --- a/src/include/utils/guc_tables.h +++ b/src/include/utils/guc_tables.h @@ -88,6 +88,7 @@ enum config_group CLIENT_CONN, CLIENT_CONN_STATEMENT, CLIENT_CONN_LOCALE, + CLIENT_CONN_PRELOAD, CLIENT_CONN_OTHER, LOCK_MANAGEMENT, COMPAT_OPTIONS, -- 1.7.10.4
Hi, Peter Eisentraut <peter_e@gmx.net> writes: > This is like shared_preload_libraries except that it takes effect at > backend start and can be changed without a full postmaster restart. It > is like local_preload_libraries except that it is still only settable by > a superuser. This can be a better way to load modules such as > auto_explain. I had a pretty hard time to get my head around that new one, and I'm not sure I totally did. The important things to me are: - No need to manually copy the lib into the plugin directory, - ALTER ROLE support. So basically it's a very good solution for auto_explain and any other module you want to load eagerly but not for everyone and when not using shared memory. I have a feeling that something simpler could be made, but I will have to continue thinking about it. I found it strange that those two paras read differently for saying the same thing? > + preloaded at connection start. This parameter cannot be changed after > + the start of a particular session. If a specified library is not > + The parameter value only takes effect at the start of the connection. > + Subsequent changes have no effect. If a specified library is not Will review the code in more details, wanted to get back on the context where this patch is useful first, and try to understand better the trade offs involved. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On 12 June 2013 22:12, Peter Eisentraut <peter_e@gmx.net> wrote:
This is like shared_preload_libraries except that it takes effect at
backend start and can be changed without a full postmaster restart. It
is like local_preload_libraries except that it is still only settable by
a superuser. This can be a better way to load modules such as
auto_explain.
Since there are now three preload parameters, regroup the documentation
a bit. Put all parameters into one section, explain common
functionality only once, update the descriptions to reflect current and
future realities.
---
Hi,
Did some basic checks on this patch. List-wise feedback below.
- Documentation Updated: Yes
- All tests pass: Yes - Removed unnecessary extra-lines: Yes
- Do we want it?: ???
- Any compiler warnings: No
- Others:
Number of new lines added not covered by tests: -107 (Wierd but, it seems to have reduced line coverage). But correct me if am not seeing the elephant in the room.
--
Robins Tharakan
Number of new lines added not covered by tests: -107 (Wierd but, it seems to have reduced line coverage). But correct me if am not seeing the elephant in the room.
My apologies.
Number of new untested lines: 108
Robins
On Fri, 2013-06-21 at 11:51 +0200, Dimitri Fontaine wrote: > I found it strange that those two paras read differently for saying > the > same thing? > > > + preloaded at connection start. This parameter cannot be > changed after > > + the start of a particular session. If a specified library > is not > > > + The parameter value only takes effect at the start of the > connection. > > + Subsequent changes have no effect. If a specified library > is not The first says that changing the parameter after connection start will result in an error. The second says that you can change the parameter after connection start, but it won't have any effect. This is just a result of the various GUC context mechanisms.
Peter Eisentraut <peter_e@gmx.net> writes: > This is like shared_preload_libraries except that it takes effect at > backend start and can be changed without a full postmaster restart. It > is like local_preload_libraries except that it is still only settable by > a superuser. This can be a better way to load modules such as > auto_explain. I finally took enough time to actually understand (I think) your proposal. The key point seems to be around ALTER ROLE SET and such like dynamic setting of a local_preload_libraries. Would it make sense to review the local_preload_libraries semantics instead? It looks like session_preload_libraries only adds flexibility, and that current usages of local_preload_libraries would be served as well by the new setting. Did you figure out a case where a local_preload_libraries setting would stop working if made into a session_preload_libraries setting? Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On Sun, 2013-07-07 at 22:50 +0200, Dimitri Fontaine wrote: > Would it make sense to review the local_preload_libraries semantics > instead? It looks like session_preload_libraries only adds > flexibility, > and that current usages of local_preload_libraries would be served as > well by the new setting. > > Did you figure out a case where a local_preload_libraries setting > would > stop working if made into a session_preload_libraries setting? local_preload_libraries allows unprivileged users to preload whitelisted libraries. session_preload_libraries allows superusers to preload any library. It's hard to see how to consolidate those, at least without adding another setting that whitelists the libraries somehow, at which point you haven't really gained anything in terms of complexity. I don't know of any actual legitimate uses of local_preload_libraries. I recall that the plpgsql debugger was meant to use it, but doesn't anymore. So it's hard to judge what to do about this, without any actual use cases.
Peter Eisentraut <peter_e@gmx.net> writes: > local_preload_libraries allows unprivileged users to preload whitelisted > libraries. session_preload_libraries allows superusers to preload any > library. It's hard to see how to consolidate those, at least without > adding another setting that whitelists the libraries somehow, at which > point you haven't really gained anything in terms of complexity. Good point. > I don't know of any actual legitimate uses of local_preload_libraries. > I recall that the plpgsql debugger was meant to use it, but doesn't > anymore. So it's hard to judge what to do about this, without any > actual use cases. Well there's my preprepare thing at https://github.com/dimitri/preprepare I don't think that the whitelisting is actually used in a way to allow for non superusers to load modules in the field, because the only way to do that with local_preload_libraries that I know of is to edit the postgresql.conf file and reload. alter role dim set local_preload_libraries = 'auto_explain'; ERROR: 55P02: parameter "local_preload_libraries" cannot beset after connection start That said, I think we could go with your current patch and revise the local_preload_libraries down the road in the 9.4 development cycle. Having both for a while is a good way to better understand if they are useful to have together. Going to mark your patch ready for commiter, thanks! Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On 7/8/13 4:37 AM, Dimitri Fontaine wrote: >> I don't know of any actual legitimate uses of local_preload_libraries. >> I recall that the plpgsql debugger was meant to use it, but doesn't >> anymore. So it's hard to judge what to do about this, without any >> actual use cases. > > Well there's my preprepare thing at > > https://github.com/dimitri/preprepare preprepare has an SQL function as entry point, so you don't need to preload it. > I don't think that the whitelisting is actually used in a way to allow > for non superusers to load modules in the field, because the only way to > do that with local_preload_libraries that I know of is to edit the > postgresql.conf file and reload. > > alter role dim set local_preload_libraries = 'auto_explain'; > ERROR: 55P02: parameter "local_preload_libraries" cannot be set after connection start I think the idea was to use PGOPTIONS to load it, controlled by the client side.
Peter Eisentraut <peter_e@gmx.net> writes: > preprepare has an SQL function as entry point, so you don't need to > preload it. Well the whole goal of that extension is to autoload itself entirely server side at connection time. The function API had been made to cope with 8.3 where it wasn't possible to get a snapshot from within the init function of a module. > I think the idea was to use PGOPTIONS to load it, controlled by the > client side. Oh. Never done that, I don't know if that's common in the field. If it's in use, then the new session_preload_libraries GUC looks the only way forward then. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support