Thread: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
On Monday, January 14, 2013 6:48 PM Boszormenyi Zoltan wrote:
2013-01-14 07:47 keltezéssel, Amit kapila írta:
> On Sunday, January 13, 2013 2:45 PM Boszormenyi Zoltan wrote:
> 2013-01-13 05:49 keltezéssel, Amit kapila írta:
>> On Sunday, January 13, 2013 12:41 AM Tom Lane wrote:
>> Boszormenyi Zoltan <zb@cybertec.at> writes:
>>>> I can think of below ways to generate tmp file
>>>> 1. Generate session specific tmp file name during operation. This will
be similar to what is
>>>> currently being done in OpenTemporaryFileinTablespace() to
generate a tempfilename.
>>>> 2. Generate global tmp file name. For this we need to maintain global
counter.
>>>> 3. Always generate temp file with same name "postgresql.auto.conf.tmp",
as at a time only one
>>>> session is allowed to operate.
>>> What's wrong with mkstemp("config_dir/postgresql.auto.conf.XXXXXX")
>>> that returns a file descriptor for an already created file with a
unique name?
>> I think for Windows exactly same behavior API might not exist, we might
need to use GetTempFileName.
>> It will generate some different kind of name, which means cleanup also
need to take care of different kind of names.
>> So if this is right, I think it might not be very appropriate to use it.
> In that case (and also because the LWLock still serialize the whole
procedure)
> you can use GetTempFileName() on Windows and tempnam(3) for non-Windows.
> GetTempFileName() and tempnam() return the constructed temporary file name
> out of the directory and the filename prefix components.
On using mktemp, linux compilation gives below warning
warning: the use of `mktemp' is dangerous, better use `mkstemp'
So I planned to use mkstemp.
In Windows, there is an api _mktemp_s, followed by open call, behaves in a similar way as mkstemp available in linux.
The code is changed to use of mkstemp functionality to generate a unique temporary file name instead of .lock file.
Please check this part of code, I am not very comfortable with using this API.
Removed the code which is used for deleting the .lock file in case of backend crash or during server startup.
Added a new function RemoveAutoConfTmpFiles used for deleting the temp files left out any during previous
postmaster session in the server startup.
In SendDir(), used sizeof() -1
With Regards,
Amit Kapila.
Attachment
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
Hi, comments below. 2013-01-18 11:05 keltezéssel, Amit kapila írta: > On using mktemp, linux compilation gives below warning > warning: the use of `mktemp' is dangerous, better use `mkstemp' > > So I planned to use mkstemp. Good. > In Windows, there is an api _mktemp_s, followed by open call, behaves in a similar way as mkstemp available in linux. > The code is changed to use of mkstemp functionality to generate a unique temporary file name instead of .lock file. > Please check this part of code, I am not very comfortable with using this API. I read the documentation of _mktemp_s() at http://msdn.microsoft.com/en-us/library/t8ex5e91%28v=vs.80%29.aspx The comment says it's only for C++, but I can compile your patch using the MinGW cross-compiler under Fedora 18. So either the MSDN comment is false, or MinGW provides this function outside C++. More research is needed on this. However, I am not sure whether Cygwin provides the mkstemp() call or not. Searching... Found bugzilla reports against mkstemp on Cygwin. So, yes, it does and your code would clash with it. In port/dirmod.c: ----8<--------8<--------8<--------8<--------8<---- #if defined(WIN32) || defined(__CYGWIN__) ... /* * pgmkstemp */ int pgmkstemp (char *TmpFileName) { int err; err = _mktemp_s(TmpFileName, strlen(TmpFileName) + 1); if (err != 0) return -1; return open(TmpFileName, O_CREAT | O_RDWR | O_EXCL, S_IRUSR | S_IWUSR); } /* We undefined these above; now redefine for possible use below */ #define rename(from, to) pgrename(from, to) #define unlink(path) pgunlink(path) #define mkstemp(tempfilename) pgmkstemp(tempfilename) #endif /* defined(WIN32) || defined(__CYGWIN__) */ ----8<--------8<--------8<--------8<--------8<---- pgmkstemp() should be defined in its own block inside #if defined(WIN32) && !defined(__CYGWIN__) ... #endif Same thing applies to src/include/port.h: ----8<--------8<--------8<--------8<--------8<---- #if defined(WIN32) || defined(__CYGWIN__) /* * Win32 doesn't have reliable rename/unlink during concurrent access. */ extern int pgrename(const char *from, const char *to); extern int pgunlink(const char *path); extern int pgmkstemp (char *TmpFileName); /* Include this first so later includes don't see these defines */ #ifdef WIN32_ONLY_COMPILER #include <io.h> #endif #define rename(from, to) pgrename(from, to) #define unlink(path) pgunlink(path) #define mkstemp(tempfilename) pgmkstemp(tempfilename) #endif /* defined(WIN32) || defined(__CYGWIN__) */ ----8<--------8<--------8<--------8<--------8<---- > Removed the code which is used for deleting the .lock file in case of backend crash or during server startup. Good. > Added a new function RemoveAutoConfTmpFiles used for deleting the temp files left out any during previous > postmaster session in the server startup. Good. > In SendDir(), used sizeof() -1 Good. Since you have these defined: + #define PG_AUTOCONF_DIR "config_dir" + #define PG_AUTOCONF_FILENAME "postgresql.auto.conf" + #define PG_AUTOCONF_SAMPLE_TMPFILE "postgresql.auto.conf.XXXXXX" + #define PG_AUTOCONF_TMP_FILE "postgresql.auto.conf." then the hardcoded values can also be changed everywhere. But to kill them in initdb.c, these #define's must be in pg_config_manual.h instead of guc.h. Most notably, postgresql.conf.sample contains: #include_dir = 'auto.conf.d' and initdb replaces it with include_dir = 'config_dir'. I prefer the auto.conf.d directory name. After killing all hardcoded "config_dir", changing one #define is all it takes to change it. There are two blocks of code that Tom commented as being "over-engineered": + /* Frame auto conf name and auto conf sample temp file name */ + snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR, PG_AUTOCONF_FILENAME); + StrNCpy(AutoConfFileName, ConfigFileName, sizeof(AutoConfFileName)); + get_parent_directory(AutoConfFileName); + join_path_components(AutoConfFileName, AutoConfFileName, Filename); + canonicalize_path(AutoConfFileName); + + snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR, PG_AUTOCONF_SAMPLE_TMPFILE); + StrNCpy(AutoConfTmpFileName, ConfigFileName, sizeof(AutoConfTmpFileName)); + get_parent_directory(AutoConfTmpFileName); + join_path_components(AutoConfTmpFileName, AutoConfTmpFileName, Filename); + canonicalize_path(AutoConfTmpFileName); You can simply do snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s", data_directory, PG_AUTOCONF_DIR, PG_AUTOCONF_FILENAME); snprintf(AutoConfTmpFileName, sizeof(AutoConfTmpFileName),"%s/%s/%s", data_directory, PG_AUTOCONF_DIR, PG_AUTOCONF_SAMPLE_TMPFILE); Also, mkstemp() and _mktemp_s() modify the file name in place, so the XXXXXX suffix becomes something else. You can pass the array pointer to write_auto_conf_file() together with the returned fd so the ereport() calls can report the actual file name, not the template. You removed the static keyword from before AbsoluteConfigLocation(), resulting in this compiler warning: In file included from guc.c:9274:0: guc-file.l:382:1: warning: no previous prototype for 'AbsoluteConfigLocation' [-Wmissing-prototypes] In validate_conf_option() in guc.c, you had "return NULL;" and "return 0;" mixed. Since this function returns a pointer, I changed all to "return NULL;". void-returning function write_auto_conf_file() in guc.c had a "return;" at the end. It's not needed. I have fixed these in the attached patch and also changed wording of the documentation, some comments and the error messages to be more expressive or more concise. The difference between your patch and mine is also attached. Best regards, Zoltán Böszörményi -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
Attachment
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Tom Lane
Date:
Boszormenyi Zoltan <zb@cybertec.at> writes:
> 2013-01-18 11:05 keltez�ssel, Amit kapila �rta:
>> On using mktemp, linux compilation gives below warning
>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>>
>> So I planned to use mkstemp.
> Good.
On my HPUX box, the man page disapproves of both, calling them obsolete
(and this man page is old enough to vote, I believe...)
Everywhere else that we need to do something like this, we just use our
own PID to disambiguate, iesprintf(tempfilename, "/path/to/file.%d", (int) getpid());
There is no need to deviate from that pattern or introduce portability
issues, since we can reasonably assume that no non-Postgres programs are
creating files in this directory.
regards, tom lane
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
2013-01-18 21:32 keltezéssel, Tom Lane írta: > Boszormenyi Zoltan <zb@cybertec.at> writes: >> 2013-01-18 11:05 keltezéssel, Amit kapila írta: >>> On using mktemp, linux compilation gives below warning >>> warning: the use of `mktemp' is dangerous, better use `mkstemp' >>> >>> So I planned to use mkstemp. >> Good. > On my HPUX box, the man page disapproves of both, calling them obsolete > (and this man page is old enough to vote, I believe...) > > Everywhere else that we need to do something like this, we just use our > own PID to disambiguate, ie > sprintf(tempfilename, "/path/to/file.%d", (int) getpid()); > There is no need to deviate from that pattern or introduce portability > issues, since we can reasonably assume that no non-Postgres programs are > creating files in this directory. Thanks for the enlightenment, I will post a new version soon. > > regards, tom lane > > -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
>> Boszormenyi Zoltan <zb@cybertec.at> writes:
>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
>>>> On using mktemp, linux compilation gives below warning
>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>>>>
>>>> So I planned to use mkstemp.
>>> Good.
>> On my HPUX box, the man page disapproves of both, calling them obsolete
>> (and this man page is old enough to vote, I believe...)
Then we have to at least consider what this old {p|s}age says... ;-)
>>
>> Everywhere else that we need to do something like this, we just use our
>> own PID to disambiguate, ie
>> sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
>> There is no need to deviate from that pattern or introduce portability
>> issues, since we can reasonably assume that no non-Postgres programs are
>> creating files in this directory.
>
> Thanks for the enlightenment, I will post a new version soon.
Here it is.
Best regards,
Zoltán Böszörményi
--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/
Attachment
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit kapila
Date:
On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote:
2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
>> Boszormenyi Zoltan <zb@cybertec.at> writes:
>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
>>>>> On using mktemp, linux compilation gives below warning
>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>>>>
>>>>> So I planned to use mkstemp.
>>>> Good.
>>> On my HPUX box, the man page disapproves of both, calling them obsolete
>>> (and this man page is old enough to vote, I believe...)
>Then we have to at least consider what this old {p|s}age says... ;-)
>>
>>> Everywhere else that we need to do something like this, we just use our
>>> own PID to disambiguate, ie
>>> sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
>>> There is no need to deviate from that pattern or introduce portability
>>> issues, since we can reasonably assume that no non-Postgres programs are
>>> creating files in this directory.
>
>> Thanks for the enlightenment, I will post a new version soon.
> Here it is.
Thanks a ton for providing better modified version.
I shall test it on Monday once and then we can conclude on it.
The only point in modifications, I am not comfortable is that in error messages you have
mentioned (automatic configuration directory). I am not sure if we should use work "automatic"
for config_dir or just use configuration directory.
However I shall keep as it is if no one else has any objection. We can let committer decide about it.
With Regards,
Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Craig Ringer
Date:
On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote: > > However, I am not sure whether Cygwin provides the mkstemp() call or not. > Searching... Found bugzilla reports against mkstemp on Cygwin. Is Cygwin a platform that should be targeted for the server backend these days? I can understand making sure that libpq works on Cygwin, but is there any reason at all to run a Pg server backend on Cygwin rather than as native Windows binaries? -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Craig Ringer
Date:
On 01/21/2013 10:03 AM, Craig Ringer wrote: > On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote: >> However, I am not sure whether Cygwin provides the mkstemp() call or not. >> Searching... Found bugzilla reports against mkstemp on Cygwin. > Is Cygwin a platform that should be targeted for the server backend > these days? > > I can understand making sure that libpq works on Cygwin, but is there > any reason at all to run a Pg server backend on Cygwin rather than as > native Windows binaries? I'm not suggesting immediately dropping working support, since this is so trivially worked around. I'm just wondering why anybody cares about the platform. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Monday, January 21, 2013 7:36 AM Craig Ringer wrote: > On 01/21/2013 10:03 AM, Craig Ringer wrote: > > On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote: > >> However, I am not sure whether Cygwin provides the mkstemp() call or > not. > >> Searching... Found bugzilla reports against mkstemp on Cygwin. > > Is Cygwin a platform that should be targeted for the server backend > > these days? > > > > I can understand making sure that libpq works on Cygwin, but is there > > any reason at all to run a Pg server backend on Cygwin rather than as > > native Windows binaries? > > I'm not suggesting immediately dropping working support, since this is > so trivially worked around. I'm just wondering why anybody cares about > the platform. We have avoided the use of mkstemp with small native implementation so now it won't be problem for any platform. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Magnus Hagander
Date:
<p dir="ltr"><br /> On Jan 21, 2013 3:06 AM, "Craig Ringer" <<a href="mailto:craig@2ndquadrant.com">craig@2ndquadrant.com</a>>wrote:<br /> ><br /> > On 01/21/2013 10:03 AM, CraigRinger wrote:<br /> > > On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote:<br /> > >> However, I am notsure whether Cygwin provides the mkstemp() call or not.<br /> > >> Searching... Found bugzilla reports againstmkstemp on Cygwin.<br /> > > Is Cygwin a platform that should be targeted for the server backend<br /> >> these days?<br /> > ><br /> > > I can understand making sure that libpq works on Cygwin, but is there<br/> > > any reason at all to run a Pg server backend on Cygwin rather than as<br /> > > native Windowsbinaries?<br /> ><br /> > I'm not suggesting immediately dropping working support, since this is<br /> >so trivially worked around. I'm just wondering why anybody cares about<br /> > the platform.<p dir="ltr">I have suggestedsimilar before, and been voted down :) iirc Andrew uses it, no? Either way, the consensus earlier had been thatas long as it doesn't require major surgery or blocks something else, we should try to keep it working. And as you saythis sounds like something that can be handled trivially, I think now is not the time. <p dir="ltr">/Magnus
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andrew Dunstan
Date:
On 01/21/2013 02:17 AM, Magnus Hagander wrote: > > > On Jan 21, 2013 3:06 AM, "Craig Ringer" <craig@2ndquadrant.com > <mailto:craig@2ndquadrant.com>> wrote: > > > > On 01/21/2013 10:03 AM, Craig Ringer wrote: > > > On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote: > > >> However, I am not sure whether Cygwin provides the mkstemp() call > or not. > > >> Searching... Found bugzilla reports against mkstemp on Cygwin. > > > Is Cygwin a platform that should be targeted for the server backend > > > these days? > > > > > > I can understand making sure that libpq works on Cygwin, but is there > > > any reason at all to run a Pg server backend on Cygwin rather than as > > > native Windows binaries? > > > > I'm not suggesting immediately dropping working support, since this is > > so trivially worked around. I'm just wondering why anybody cares about > > the platform. > > I have suggested similar before, and been voted down :) iirc Andrew > uses it, no? Either way, the consensus earlier had been that as long > as it doesn't require major surgery or blocks something else, we > should try to keep it working. And as you say this sounds like > something that can be handled trivially, I think now is not the time. > > No, I only use the client. But then I support plenty of things I don't use. cheers andrew
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Magnus Hagander
Date:
On Mon, Jan 21, 2013 at 9:01 AM, Andrew Dunstan <andrew@dunslane.net> wrote: > > On 01/21/2013 02:17 AM, Magnus Hagander wrote: > >> >> >> On Jan 21, 2013 3:06 AM, "Craig Ringer" <craig@2ndquadrant.com >> <mailto:craig@2ndquadrant.com>> wrote: >> > >> > On 01/21/2013 10:03 AM, Craig Ringer wrote: >> > > On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote: >> > >> However, I am not sure whether Cygwin provides the mkstemp() call or >> > >> not. >> > >> Searching... Found bugzilla reports against mkstemp on Cygwin. >> > > Is Cygwin a platform that should be targeted for the server backend >> > > these days? >> > > >> > > I can understand making sure that libpq works on Cygwin, but is there >> > > any reason at all to run a Pg server backend on Cygwin rather than as >> > > native Windows binaries? >> > >> > I'm not suggesting immediately dropping working support, since this is >> > so trivially worked around. I'm just wondering why anybody cares about >> > the platform. >> >> I have suggested similar before, and been voted down :) iirc Andrew uses >> it, no? Either way, the consensus earlier had been that as long as it >> doesn't require major surgery or blocks something else, we should try to >> keep it working. And as you say this sounds like something that can be >> handled trivially, I think now is not the time. >> > > No, I only use the client. But then I support plenty of things I don't use. Oh, I somehow thought you were. And yes, we all support things we don't use - but it certainly helps if there is *someone* out there who uses it. Having a buildfarm animal (which we do) only goes so far... --Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andrew Dunstan
Date:
On 01/21/2013 07:05 AM, Magnus Hagander wrote: >> No, I only use the client. But then I support plenty of things I don't use. > Oh, I somehow thought you were. And yes, we all support things we > don't use - but it certainly helps if there is *someone* out there who > uses it. Having a buildfarm animal (which we do) only goes so far... > It is being used. I get emails from time to time from people asking about it. cheers andrew
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit kapila
Date:
On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote:
2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
>> Boszormenyi Zoltan <zb@cybertec.at> writes:
>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
>>>>> On using mktemp, linux compilation gives below warning
>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>>>>
>>
>>> Everywhere else that we need to do something like this, we just use our
>>> own PID to disambiguate, ie
>>> sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
>>> There is no need to deviate from that pattern or introduce portability
>>> issues, since we can reasonably assume that no non-Postgres programs are
>>> creating files in this directory.
>
>> Thanks for the enlightenment, I will post a new version soon.
> Here it is.
The patch sent by you works fine.
It needs small modification as below:
The "auto.conf.d" directory should follow the postgresql.conf file directory not the data_directory.
The same is validated while parsing the postgresql.conf configuration file.
Patch is changed to use the postgresql.conf file directory as below.
StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir));
get_parent_directory(ConfigFileDir);
/* Frame auto conf name and auto conf sample temp file name */
snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s",
ConfigFileDir,
PG_AUTOCONF_DIR,
PG_AUTOCONF_FILENAME);
This closes all comments raised till now for this patch.
Kindly let me know if you feel something is missing?
With Regards,
Amit Kapila.
Attachment
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Zoltán Böszörményi
Date:
2013-01-22 13:32 keltezéssel, Amit kapila írta: > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote: > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta: >> 2013-01-18 21:32 keltezéssel, Tom Lane írta: >>> Boszormenyi Zoltan <zb@cybertec.at> writes: >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta: >>>>>> On using mktemp, linux compilation gives below warning >>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp' > >>>> Everywhere else that we need to do something like this, we just use our >>>> own PID to disambiguate, ie >>>> sprintf(tempfilename, "/path/to/file.%d", (int) getpid()); >>>> There is no need to deviate from that pattern or introduce portability >>>> issues, since we can reasonably assume that no non-Postgres programs are >>>> creating files in this directory. >>> Thanks for the enlightenment, I will post a new version soon. >> Here it is. > The patch sent by you works fine. > It needs small modification as below: > > The "auto.conf.d" directory should follow the postgresql.conf file directory not the data_directory. > The same is validated while parsing the postgresql.conf configuration file. > > Patch is changed to use the postgresql.conf file directory as below. > > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir)); > get_parent_directory(ConfigFileDir); > /* Frame auto conf name and auto conf sample temp file name */ > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s", > ConfigFileDir, > PG_AUTOCONF_DIR, > PG_AUTOCONF_FILENAME); Maybe it's just me but I prefer to have identical settings across all replicated servers. But I agree that there can be use cases with different setups. All in all, this change makes it necessary to run the same SET PERSISTENT statements on all slave servers, too, to make them identical again if the configuration is separated from the data directory (like on Debian or Ubuntu using the default packages). This needs to be documented explicitly. > > This closes all comments raised till now for this patch. > Kindly let me know if you feel something is missing? I can't think of anything else. Best regards, Zoltán Böszörményi -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote: > 2013-01-22 13:32 keltezéssel, Amit kapila írta: > > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote: > > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta: > >> 2013-01-18 21:32 keltezéssel, Tom Lane írta: > >>> Boszormenyi Zoltan <zb@cybertec.at> writes: > >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta: > >>>>>> On using mktemp, linux compilation gives below warning > >>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp' > > > >>>> Everywhere else that we need to do something like this, we just > use our > >>>> own PID to disambiguate, ie > >>>> sprintf(tempfilename, "/path/to/file.%d", (int) getpid()); > >>>> There is no need to deviate from that pattern or introduce > portability > >>>> issues, since we can reasonably assume that no non-Postgres > programs are > >>>> creating files in this directory. > >>> Thanks for the enlightenment, I will post a new version soon. > >> Here it is. > > The patch sent by you works fine. > > It needs small modification as below: > > > > The "auto.conf.d" directory should follow the postgresql.conf file > directory not the data_directory. > > The same is validated while parsing the postgresql.conf configuration > file. > > > > Patch is changed to use the postgresql.conf file directory as below. > > > > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir)); > > get_parent_directory(ConfigFileDir); > > /* Frame auto conf name and auto conf sample temp file name */ > > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s", > > ConfigFileDir, > > PG_AUTOCONF_DIR, > > PG_AUTOCONF_FILENAME); > > Maybe it's just me but I prefer to have identical > settings across all replicated servers. But I agree > that there can be use cases with different setups. > > All in all, this change makes it necessary to run the > same SET PERSISTENT statements on all slave servers, > too, to make them identical again if the configuration > is separated from the data directory (like on Debian > or Ubuntu using the default packages). This needs to be > documented explicitly. SET PERSISTENT command needs to run on all slave servers even if the path we take w.r.t data directory unless user takes backup after command. > > > > This closes all comments raised till now for this patch. > > Kindly let me know if you feel something is missing? > > I can't think of anything else. In that case can you mark it as "Ready For Committer" With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Fujii Masao
Date:
On Tue, Jan 22, 2013 at 11:07 PM, Amit Kapila <amit.kapila@huawei.com> wrote: > On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote: >> 2013-01-22 13:32 keltezéssel, Amit kapila írta: >> > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote: >> > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta: >> >> 2013-01-18 21:32 keltezéssel, Tom Lane írta: >> >>> Boszormenyi Zoltan <zb@cybertec.at> writes: >> >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta: >> >>>>>> On using mktemp, linux compilation gives below warning >> >>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp' >> > >> >>>> Everywhere else that we need to do something like this, we just >> use our >> >>>> own PID to disambiguate, ie >> >>>> sprintf(tempfilename, "/path/to/file.%d", (int) getpid()); >> >>>> There is no need to deviate from that pattern or introduce >> portability >> >>>> issues, since we can reasonably assume that no non-Postgres >> programs are >> >>>> creating files in this directory. >> >>> Thanks for the enlightenment, I will post a new version soon. >> >> Here it is. >> > The patch sent by you works fine. >> > It needs small modification as below: >> > >> > The "auto.conf.d" directory should follow the postgresql.conf file >> directory not the data_directory. >> > The same is validated while parsing the postgresql.conf configuration >> file. >> > >> > Patch is changed to use the postgresql.conf file directory as below. >> > >> > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir)); >> > get_parent_directory(ConfigFileDir); >> > /* Frame auto conf name and auto conf sample temp file name */ >> > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s", >> > ConfigFileDir, >> > PG_AUTOCONF_DIR, >> > PG_AUTOCONF_FILENAME); >> >> Maybe it's just me but I prefer to have identical >> settings across all replicated servers. But I agree >> that there can be use cases with different setups. >> >> All in all, this change makes it necessary to run the >> same SET PERSISTENT statements on all slave servers, >> too, to make them identical again if the configuration >> is separated from the data directory (like on Debian >> or Ubuntu using the default packages). This needs to be >> documented explicitly. > > SET PERSISTENT command needs to run on all slave servers even if the path we > take w.r.t data directory > unless user takes backup after command. Is it safe to write something in the directory other than data directory via SQL? postgres user usually has the write permission for the configuration directory like /etc/postgresql? >> > This closes all comments raised till now for this patch. >> > Kindly let me know if you feel something is missing? >> >> I can't think of anything else. For removing + a configuration entry from <filename>postgresql.auto.conf</filename> file, + use <command>RESET PERSISTENT</command>. The values will be effective + after reload of server configuration (SIGHUP) or server startup. The description of RESET PERSISTENT is in the document, but it seems not to be implemented. Regards, -- Fujii Masao
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Fujii Masao
Date:
On Tue, Jan 22, 2013 at 11:54 PM, Fujii Masao <masao.fujii@gmail.com> wrote: > On Tue, Jan 22, 2013 at 11:07 PM, Amit Kapila <amit.kapila@huawei.com> wrote: >> On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote: >>> 2013-01-22 13:32 keltezéssel, Amit kapila írta: >>> > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote: >>> > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta: >>> >> 2013-01-18 21:32 keltezéssel, Tom Lane írta: >>> >>> Boszormenyi Zoltan <zb@cybertec.at> writes: >>> >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta: >>> >>>>>> On using mktemp, linux compilation gives below warning >>> >>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp' >>> > >>> >>>> Everywhere else that we need to do something like this, we just >>> use our >>> >>>> own PID to disambiguate, ie >>> >>>> sprintf(tempfilename, "/path/to/file.%d", (int) getpid()); >>> >>>> There is no need to deviate from that pattern or introduce >>> portability >>> >>>> issues, since we can reasonably assume that no non-Postgres >>> programs are >>> >>>> creating files in this directory. >>> >>> Thanks for the enlightenment, I will post a new version soon. >>> >> Here it is. >>> > The patch sent by you works fine. >>> > It needs small modification as below: >>> > >>> > The "auto.conf.d" directory should follow the postgresql.conf file >>> directory not the data_directory. >>> > The same is validated while parsing the postgresql.conf configuration >>> file. >>> > >>> > Patch is changed to use the postgresql.conf file directory as below. >>> > >>> > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir)); >>> > get_parent_directory(ConfigFileDir); >>> > /* Frame auto conf name and auto conf sample temp file name */ >>> > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s", >>> > ConfigFileDir, >>> > PG_AUTOCONF_DIR, >>> > PG_AUTOCONF_FILENAME); >>> >>> Maybe it's just me but I prefer to have identical >>> settings across all replicated servers. But I agree >>> that there can be use cases with different setups. >>> >>> All in all, this change makes it necessary to run the >>> same SET PERSISTENT statements on all slave servers, >>> too, to make them identical again if the configuration >>> is separated from the data directory (like on Debian >>> or Ubuntu using the default packages). This needs to be >>> documented explicitly. >> >> SET PERSISTENT command needs to run on all slave servers even if the path we >> take w.r.t data directory >> unless user takes backup after command. > > Is it safe to write something in the directory other than data directory > via SQL? > > postgres user usually has the write permission for the configuration > directory like /etc/postgresql? > >>> > This closes all comments raised till now for this patch. >>> > Kindly let me know if you feel something is missing? >>> >>> I can't think of anything else. > > For removing > + a configuration entry from <filename>postgresql.auto.conf</filename> file, > + use <command>RESET PERSISTENT</command>. The values will be effective > + after reload of server configuration (SIGHUP) or server startup. > > The description of RESET PERSISTENT is in the document, but it > seems not to be implemented. I read only document part in the patch and did simple test. Here are the comments: storage.sgml needs to be updated. Doesn't auto.conf.d need to be explained in config.sgml? When I created my own configuration file in auto.conf.d and started the server, I got the following LOG message. This means that users should not create any their own configuration file there? Why shouldn't they? I think that it's more useful to allow users to put also their own configuration files in auto.conf.d. Then users can include any configuration file without adding new include derective into postgresql.conf because auto.conf.d has already been included. LOG: unexpected file found in automatic configuration directory: "/data/auto.conf.d/hoge" When I removed postgresql.auto.conf and restarted the server, I got the following warning message. This is not correct because I didn't remove "auto.conf.d" from postgresql.conf. What I removed is only postgresql.auto.conf. WARNING: Default "auto.conf.d" is not present in postgresql.conf. Configuration parameters changed with SET PERSISTENT command will not be effective. Regards, -- Fujii Masao
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Tuesday, January 22, 2013 8:25 PM Fujii Masao wrote: > On Tue, Jan 22, 2013 at 11:07 PM, Amit Kapila <amit.kapila@huawei.com> > wrote: > > On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote: > >> 2013-01-22 13:32 keltezéssel, Amit kapila írta: > >> > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote: > >> > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta: > >> >> 2013-01-18 21:32 keltezéssel, Tom Lane írta: > >> >>> Boszormenyi Zoltan <zb@cybertec.at> writes: > >> >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta: > >> >>>>>> On using mktemp, linux compilation gives below warning > >> >>>>>> warning: the use of `mktemp' is dangerous, better use > `mkstemp' > >> > > >> >>>> Everywhere else that we need to do something like this, we just > >> use our > >> >>>> own PID to disambiguate, ie > >> >>>> sprintf(tempfilename, "/path/to/file.%d", (int) getpid()); > >> >>>> There is no need to deviate from that pattern or introduce > >> portability > >> >>>> issues, since we can reasonably assume that no non-Postgres > >> programs are > >> >>>> creating files in this directory. > >> >>> Thanks for the enlightenment, I will post a new version soon. > >> >> Here it is. > >> > The patch sent by you works fine. > >> > It needs small modification as below: > >> > > >> > The "auto.conf.d" directory should follow the postgresql.conf file > >> directory not the data_directory. > >> > The same is validated while parsing the postgresql.conf > configuration > >> file. > >> > > >> > Patch is changed to use the postgresql.conf file directory as > below. > >> > > >> > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir)); > >> > get_parent_directory(ConfigFileDir); > >> > /* Frame auto conf name and auto conf sample temp file name */ > >> > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s", > >> > ConfigFileDir, > >> > PG_AUTOCONF_DIR, > >> > PG_AUTOCONF_FILENAME); > >> > >> Maybe it's just me but I prefer to have identical > >> settings across all replicated servers. But I agree > >> that there can be use cases with different setups. > >> > >> All in all, this change makes it necessary to run the > >> same SET PERSISTENT statements on all slave servers, > >> too, to make them identical again if the configuration > >> is separated from the data directory (like on Debian > >> or Ubuntu using the default packages). This needs to be > >> documented explicitly. > > > > SET PERSISTENT command needs to run on all slave servers even if the > path we > > take w.r.t data directory > > unless user takes backup after command. > > Is it safe to write something in the directory other than data > directory > via SQL? > > postgres user usually has the write permission for the configuration > directory like /etc/postgresql? As postgresql.conf will also be in same path and if user can change that, then what's the problem with postgresql.auto.conf Do you see any security risk? > >> > This closes all comments raised till now for this patch. > >> > Kindly let me know if you feel something is missing? > >> > >> I can't think of anything else. > > For removing > + a configuration entry from > <filename>postgresql.auto.conf</filename> file, > + use <command>RESET PERSISTENT</command>. The values will be > effective > + after reload of server configuration (SIGHUP) or server startup. > > The description of RESET PERSISTENT is in the document, but it > seems not to be implemented. This command support has been removed from patch due to parser issues, so it should be removed from documentation as well. I shall fix this along with other issues raised by you. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Tuesday, January 22, 2013 10:14 PM Fujii Masao wrote: > On Tue, Jan 22, 2013 at 11:54 PM, Fujii Masao <masao.fujii@gmail.com> > wrote: > > On Tue, Jan 22, 2013 at 11:07 PM, Amit Kapila > <amit.kapila@huawei.com> wrote: > >> On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote: > >>> 2013-01-22 13:32 keltezéssel, Amit kapila írta: > >>> > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote: > >>> > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta: > >>> >> 2013-01-18 21:32 keltezéssel, Tom Lane írta: > >>> >>> Boszormenyi Zoltan <zb@cybertec.at> writes: > >>> >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta: > >>> >>>>>> On using mktemp, linux compilation gives below warning > >>> >>>>>> warning: the use of `mktemp' is dangerous, better use > `mkstemp' > >>> > > >>> >>>> Everywhere else that we need to do something like this, we > just > >>> use our > >>> >>>> own PID to disambiguate, ie > >>> >>>> sprintf(tempfilename, "/path/to/file.%d", (int) > getpid()); > >>> >>>> There is no need to deviate from that pattern or introduce > >>> portability > >>> >>>> issues, since we can reasonably assume that no non-Postgres > >>> programs are > >>> >>>> creating files in this directory. > >>> >>> Thanks for the enlightenment, I will post a new version soon. > >>> >> Here it is. > >>> > The patch sent by you works fine. > >>> > It needs small modification as below: > >>> > > >>> > The "auto.conf.d" directory should follow the postgresql.conf > file > >>> directory not the data_directory. > >>> > The same is validated while parsing the postgresql.conf > configuration > >>> file. > >>> > > >>> > Patch is changed to use the postgresql.conf file directory as > below. > >>> > > >>> > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir)); > >>> > get_parent_directory(ConfigFileDir); > >>> > /* Frame auto conf name and auto conf sample temp file name */ > >>> > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s", > >>> > ConfigFileDir, > >>> > PG_AUTOCONF_DIR, > >>> > PG_AUTOCONF_FILENAME); > >>> > >>> Maybe it's just me but I prefer to have identical > >>> settings across all replicated servers. But I agree > >>> that there can be use cases with different setups. > >>> > >>> All in all, this change makes it necessary to run the > >>> same SET PERSISTENT statements on all slave servers, > >>> too, to make them identical again if the configuration > >>> is separated from the data directory (like on Debian > >>> or Ubuntu using the default packages). This needs to be > >>> documented explicitly. > >> > >> SET PERSISTENT command needs to run on all slave servers even if the > path we > >> take w.r.t data directory > >> unless user takes backup after command. > > > > Is it safe to write something in the directory other than data > directory > > via SQL? > > > > postgres user usually has the write permission for the configuration > > directory like /etc/postgresql? > > > >>> > This closes all comments raised till now for this patch. > >>> > Kindly let me know if you feel something is missing? > >>> > >>> I can't think of anything else. > > > > For removing > > + a configuration entry from > <filename>postgresql.auto.conf</filename> file, > > + use <command>RESET PERSISTENT</command>. The values will be > effective > > + after reload of server configuration (SIGHUP) or server startup. > > > > The description of RESET PERSISTENT is in the document, but it > > seems not to be implemented. > > I read only document part in the patch and did simple test. > Here are the comments: > > storage.sgml needs to be updated. > > Doesn't auto.conf.d need to be explained in config.sgml? I shall update both these files. > When I created my own configuration file in auto.conf.d and > started the server, I got the following LOG message. This > means that users should not create any their own configuration > file there? Why shouldn't they? It can be allowed, but for now it has been kept such that automatically generated conf files will be present in this directory, thats what the name 'auto.conf.d' suggests. > I think that it's more useful to > allow users to put also their own configuration files in auto.conf.d. > Then users can include any configuration file without adding > new include derective into postgresql.conf because auto.conf.d > has already been included. > > LOG: unexpected file found in automatic configuration directory: > "/data/auto.conf.d/hoge" Personally I don't have objection in getting other user specific conf files in this directory. But I think then we should name this directory also differently as it was initially (conf_dir) in the Patch. I would like to see opinion of others also in this matter, so that later I don't need to change it back to what currently it is. > When I removed postgresql.auto.conf and restarted the server, > I got the following warning message. This is not correct because > I didn't remove "auto.conf.d" from postgresql.conf. What I removed > is only postgresql.auto.conf. > > WARNING: Default "auto.conf.d" is not present in postgresql.conf. > Configuration parameters changed with SET PERSISTENT command will not > be effective. How about changing it to below message: WARNING: File 'postgresql.auto.conf' is not accessible, either file 'postgresql.auto.conf' or folder '%s' doesn't exist or default "auto.conf.d" is not present in postgresql.conf. Configuration parameters changed with SET PERSISTENT command will not be effective. The reason I have kept single error message for all issues related to 'postgresql.auto.conf' is to keep code little simpler, else I need to distinguish for each particular case. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Fujii Masao
Date:
On Wed, Jan 23, 2013 at 12:24 PM, Amit Kapila <amit.kapila@huawei.com> wrote: > On Tuesday, January 22, 2013 8:25 PM Fujii Masao wrote: >> On Tue, Jan 22, 2013 at 11:07 PM, Amit Kapila <amit.kapila@huawei.com> >> wrote: >> > On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote: >> >> 2013-01-22 13:32 keltezéssel, Amit kapila írta: >> >> > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote: >> >> > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta: >> >> >> 2013-01-18 21:32 keltezéssel, Tom Lane írta: >> >> >>> Boszormenyi Zoltan <zb@cybertec.at> writes: >> >> >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta: >> >> >>>>>> On using mktemp, linux compilation gives below warning >> >> >>>>>> warning: the use of `mktemp' is dangerous, better use >> `mkstemp' >> >> > >> >> >>>> Everywhere else that we need to do something like this, we just >> >> use our >> >> >>>> own PID to disambiguate, ie >> >> >>>> sprintf(tempfilename, "/path/to/file.%d", (int) getpid()); >> >> >>>> There is no need to deviate from that pattern or introduce >> >> portability >> >> >>>> issues, since we can reasonably assume that no non-Postgres >> >> programs are >> >> >>>> creating files in this directory. >> >> >>> Thanks for the enlightenment, I will post a new version soon. >> >> >> Here it is. >> >> > The patch sent by you works fine. >> >> > It needs small modification as below: >> >> > >> >> > The "auto.conf.d" directory should follow the postgresql.conf file >> >> directory not the data_directory. >> >> > The same is validated while parsing the postgresql.conf >> configuration >> >> file. >> >> > >> >> > Patch is changed to use the postgresql.conf file directory as >> below. >> >> > >> >> > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir)); >> >> > get_parent_directory(ConfigFileDir); >> >> > /* Frame auto conf name and auto conf sample temp file name */ >> >> > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s", >> >> > ConfigFileDir, >> >> > PG_AUTOCONF_DIR, >> >> > PG_AUTOCONF_FILENAME); >> >> >> >> Maybe it's just me but I prefer to have identical >> >> settings across all replicated servers. But I agree >> >> that there can be use cases with different setups. >> >> >> >> All in all, this change makes it necessary to run the >> >> same SET PERSISTENT statements on all slave servers, >> >> too, to make them identical again if the configuration >> >> is separated from the data directory (like on Debian >> >> or Ubuntu using the default packages). This needs to be >> >> documented explicitly. >> > >> > SET PERSISTENT command needs to run on all slave servers even if the >> path we >> > take w.r.t data directory >> > unless user takes backup after command. >> >> Is it safe to write something in the directory other than data >> directory >> via SQL? >> >> postgres user usually has the write permission for the configuration >> directory like /etc/postgresql? > > As postgresql.conf will also be in same path and if user can change that, > then what's the problem with postgresql.auto.conf If the configuration directory like /etc is owned by root and only root has a write permission for it, the user running PostgreSQL server would not be able to update postgresql.auto.conf there. OTOH, even in that case, a user can switch to root and update the configuration file there. I'm not sure whether the configuration directory is usually writable by the user running PostgreSQL server in Debian or Ubuntu, though. > Do you see any security risk? I have no idea. I just wondered that because some functions like pg_file_write() in adminpack are restricted to write something in the directory other than $PGDATA. >> >> > This closes all comments raised till now for this patch. >> >> > Kindly let me know if you feel something is missing? >> >> >> >> I can't think of anything else. >> >> For removing >> + a configuration entry from >> <filename>postgresql.auto.conf</filename> file, >> + use <command>RESET PERSISTENT</command>. The values will be >> effective >> + after reload of server configuration (SIGHUP) or server startup. >> >> The description of RESET PERSISTENT is in the document, but it >> seems not to be implemented. > > This command support has been removed from patch due to parser issues, so it > should be removed from documentation as well. I shall fix this along with > other issues raised by you. Okay. Regards, -- Fujii Masao
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Fujii Masao
Date:
On Wed, Jan 23, 2013 at 6:18 PM, Amit Kapila <amit.kapila@huawei.com> wrote: > On Tuesday, January 22, 2013 10:14 PM Fujii Masao wrote: >> When I removed postgresql.auto.conf and restarted the server, >> I got the following warning message. This is not correct because >> I didn't remove "auto.conf.d" from postgresql.conf. What I removed >> is only postgresql.auto.conf. >> >> WARNING: Default "auto.conf.d" is not present in postgresql.conf. >> Configuration parameters changed with SET PERSISTENT command will not >> be effective. > > How about changing it to below message: > > WARNING: File 'postgresql.auto.conf' is not accessible, either file > 'postgresql.auto.conf' or folder '%s' doesn't exist or default "auto.conf.d" > is not present in postgresql.conf. > Configuration parameters changed with SET PERSISTENT command will not be > effective. Or we should suppress such a warning message in the case where postgresql.auto.conf doesn't exist? SET PERSISTENT creates that file automatically if it doesn't exist. So we can expect that configuration parameters changed with SET PERSISTENT WILL be effective. This warning message implies that the line "include_dir 'auto.conf.d'" must not be removed from postgresql.conf? If so, we should warn that in both document and postgresql.conf.sample? Or we should hard-code so that something like auto.conf.d is always included even when that include_dir directive doesn't exist? Regards, -- Fujii Masao
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Alvaro Herrera
Date:
Fujii Masao escribió: > On Wed, Jan 23, 2013 at 12:24 PM, Amit Kapila <amit.kapila@huawei.com> wrote: > >> Is it safe to write something in the directory other than data > >> directory > >> via SQL? > >> > >> postgres user usually has the write permission for the configuration > >> directory like /etc/postgresql? > > > > As postgresql.conf will also be in same path and if user can change that, > > then what's the problem with postgresql.auto.conf > > If the configuration directory like /etc is owned by root and only root has > a write permission for it, the user running PostgreSQL server would not > be able to update postgresql.auto.conf there. OTOH, even in that case, > a user can switch to root and update the configuration file there. I'm not > sure whether the configuration directory is usually writable by the user > running PostgreSQL server in Debian or Ubuntu, though. Yes it is -- the /etc/postgresql/<version>/<cluster> directory (where postgresql.conf resides) is owned by user postgres. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-01-22 12:32:07 +0000, Amit kapila wrote: > This closes all comments raised till now for this patch. > Kindly let me know if you feel something is missing? I am coming late to this patch, so bear with me if I repeat somethign said elsewhere. Review comments of cursory pass through the patch: * most comments are hard to understand. I know the problem of that being hard for a non-native speaker by heart, but I thinkanother pass over them would be good thing. * The gram.y changes arround set_rest_(more|common) seem pretty confused to me. E.g. its not possible anymore to set thetimezone for a function. And why is it possible to persistently set the search path, but not client encoding? Why is FROMCURRENT in set_rest_more? * set_config_file should elog(ERROR), not return on an unhandled setstmt->kind * why are you creating AutoConfFileName if its not stat'able? It seems better to simply skip parsing the old file in thatcase * Writing the temporary file to .$pid seems like a bad idea, better use one file for that, SET PERSISTENT is protected byan exclusive lock anyway. * the write sequence should be: * fsync(tempfile) * fsync(directory) * rename(tempfile, configfile) * fsync(configfile) *fsync(directory) * write_auto_conf_file should probably escape quoted values? * coding style should be adhered to more closesly, there are many if (pointer) which should be if (pointer != NULL), single-lineblocks enclosed in curlies which shouldn't, etc. * replace_auto_config_file and surrounding functions need more comments in the header * the check that prevents persistent SETs in a transaction should rather be in utility.c and use PreventTransactionChainlike most of the others that need to do that (c.f. T_CreatedbStmt). I think this patch is a good bit away of being ready for committer... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Wednesday, January 23, 2013 9:51 PM Fujii Masao wrote: > On Wed, Jan 23, 2013 at 6:18 PM, Amit Kapila <amit.kapila@huawei.com> > wrote: > > On Tuesday, January 22, 2013 10:14 PM Fujii Masao wrote: > >> When I removed postgresql.auto.conf and restarted the server, > >> I got the following warning message. This is not correct because > >> I didn't remove "auto.conf.d" from postgresql.conf. What I removed > >> is only postgresql.auto.conf. > >> > >> WARNING: Default "auto.conf.d" is not present in postgresql.conf. > >> Configuration parameters changed with SET PERSISTENT command will > not > >> be effective. > > > > How about changing it to below message: > > > > WARNING: File 'postgresql.auto.conf' is not accessible, either file > > 'postgresql.auto.conf' or folder '%s' doesn't exist or default > "auto.conf.d" > > is not present in postgresql.conf. > > Configuration parameters changed with SET PERSISTENT command will not > be > > effective. > > Or we should suppress such a warning message in the case where > postgresql.auto.conf doesn't exist? SET PERSISTENT creates that > file automatically if it doesn't exist. So we can expect that > configuration > parameters changed with SET PERSISTENT WILL be effective. This Warning (message) can come during startup or reload, so if at that time postgresql.auto.conf doesn't exist the parameters set by SET PERSISTENT previous to startup or reload will not be effective. So IMO, we can think to change part of this message as below: "Configuration parameters changed before start of server with SET PERSISTENT command will not be effective." Any other idea for change in message? > This warning message implies that the line "include_dir 'auto.conf.d'" > must not be removed from postgresql.conf? If so, we should warn that > in both document and postgresql.conf.sample? I shall fix this as below: Change current WARNING in postgresql.conf.sample as below: # This includes the default configuration directory included to support # SET PERSISTENT statement. # # WARNING: User should not remove below include_dir or directory auto.conf.d, as both are required to make SET PERSISTENT command work. Any configuration parameter valuesspecified after this line # will override the values set by SET PERSISTENT statement. Also Update this in runtime.sgml under heading "Starting the Database Server" > Or we should hard-code > so that something like auto.conf.d is always included even when that > include_dir directive doesn't exist? I think this can create problem for user who intentionally wants to remove this directory inclusion. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Wednesday, January 23, 2013 11:51 PM Alvaro Herrera wrote: > Fujii Masao escribió: > > On Wed, Jan 23, 2013 at 12:24 PM, Amit Kapila > <amit.kapila@huawei.com> wrote: > > > >> Is it safe to write something in the directory other than data > > >> directory > > >> via SQL? > > >> > > >> postgres user usually has the write permission for the > configuration > > >> directory like /etc/postgresql? > > > > > > As postgresql.conf will also be in same path and if user can change > that, > > > then what's the problem with postgresql.auto.conf > > > > If the configuration directory like /etc is owned by root and only > root has > > a write permission for it, the user running PostgreSQL server would > not > > be able to update postgresql.auto.conf there. OTOH, even in that > case, > > a user can switch to root and update the configuration file there. > I'm not > > sure whether the configuration directory is usually writable by the > user > > running PostgreSQL server in Debian or Ubuntu, though. > > Yes it is -- the /etc/postgresql/<version>/<cluster> directory (where > postgresql.conf resides) is owned by user postgres. So in that case we can consider postgresql.auto.conf to be in path w.r.t postgresql.conf instead of data directory. With Regards, Amit Kapila.
On Thursday, January 24, 2013 1:56 AM Andres Freund wrote: > On 2013-01-22 12:32:07 +0000, Amit kapila wrote: > > This closes all comments raised till now for this patch. > > Kindly let me know if you feel something is missing? > > I am coming late to this patch, so bear with me if I repeat somethign > said elsewhere. Thanks for looking at patch. > Review comments of cursory pass through the patch: > * most comments are hard to understand. I know the problem of that > being hard for a non-native speaker by heart, but I think another > pass > over them would be good thing. I shall give another pass to improve the comments in code. > * The gram.y changes arround set_rest_(more|common) seem pretty > confused > to me. >E.g. its not possible anymore to set the timezone for a function. What do you exactly mean by this part of comment. > And why is it possible to persistently set the search path, > but not client encoding? Why is FROM CURRENT in set_rest_more? I think the main reason was some of the commands can work only in transaction and as SET Persistent cannot work inside transaction block, so I had kept some seggregation. I shall reply you the exact reason in separate mail. > * set_config_file should elog(ERROR), not return on an unhandled > setstmt->kind Shall fix it. > * why are you creating AutoConfFileName if its not stat'able? It seems > better to simply skip parsing the old file in that case Sure, it's better to handle as you are suggesting. > * Writing the temporary file to .$pid seems like a bad idea, better use > one file for that, SET PERSISTENT is protected by an exclusive lock > anyway. I think we can use one temporary file, infact that was one of the ways I have asked in one of the previous mails. However Tom and Zoltan felt this is better way to do it. I can think of one reasonwhere it will be better to have .$pid, and that is if some one has opened the file manually, then all other sessions can fail (in WINDOWS). Infact this is one of test Zoltan had performed. > * the write sequence should be: > * fsync(tempfile) > * fsync(directory) > * rename(tempfile, configfile) > * fsync(configfile) > * fsync(directory) Why do we need fsync(directory) and fsync(configfile)? > * write_auto_conf_file should probably escape quoted values? Can you please elaborate more, I am not able to understandyour point? > * coding style should be adhered to more closesly, there are many > if (pointer) which should be if (pointer != NULL), Are you pointing in function validate_conf_option(const char *name, char *value) for below usage: + if (value) > single-line blocks > enclosed in curlies which shouldn't, etc. Shall fix. > * replace_auto_config_file and surrounding functions need more comments > in the header Sure, shall add more comments in function header of following functions: validate_conf_option() RemoveAutoConfTmpFiles() write_auto_conf_file() replace_auto_config_value() > * the check that prevents persistent SETs in a transaction should > rather > be in utility.c and use PreventTransactionChain like most of the > others that need to do that (c.f. T_CreatedbStmt). We can move the check in utility.c, but if we use PreventTransactionChain, then it will be disallowed in functions as well. But the original idea was to not support in transaction blocks only. So I feel use of current function IsTransactionBlock() should be okay. With Regards, Amit Kapila.
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
> > * The gram.y changes arround set_rest_(more|common) seem pretty
> > confused
> > to me.
>
> >E.g. its not possible anymore to set the timezone for a function.
>
> What do you exactly mean by this part of comment.
The set_rest_more production is used in FunctionSetResetClause and youve
removed capabilities from it.
> > And why is it possible to persistently set the search path,
> > but not client encoding? Why is FROM CURRENT in set_rest_more?
>
> I think the main reason was some of the commands can work only in
> transaction and as SET Persistent cannot work inside transaction block, so I had kept
> some seggregation.
Yes, I can see reasons for doing this, I just think the split isn't
correct as youve done it.
> > * Writing the temporary file to .$pid seems like a bad idea, better use
> > one file for that, SET PERSISTENT is protected by an exclusive lock
> > anyway.
>
> I think we can use one temporary file, infact that was one of the ways I
> have asked in one of the previous mails.
> However Tom and Zoltan felt this is better way to do it.
The have? I didn't read it like that. The file can only ever written by
a running postmaster and we already have code that ensures that. There's
absolutely no need for the tempfile to have a nondeterministic
name. That makes cleanup way easier as well.
> I can think of one reason where it will be better to have .$pid, and that
> is if some one has opened the
> file manually, then all other sessions can fail (in WINDOWS). Infact this
> is one of test Zoltan had performed.
Why on earth should somebody have the tempfile open? There's an
exclusive lock around writing the file in your code and if anybody
interferes like that with postgres' temporary data you have far bigger
problems than SET PERSISTENT erroring out.
> > * the write sequence should be:
> > * fsync(tempfile)
> > * fsync(directory)
> > * rename(tempfile, configfile)
> > * fsync(configfile)
> > * fsync(directory)
>
> Why do we need fsync(directory) and fsync(configfile)?
So they don't vanish /get corrupted after a crash? The above is the
canonically safe way to safely write a file without an invalid file ever
being visible.
> > * write_auto_conf_file should probably escape quoted values?
> Can you please elaborate more, I am not able to understand your
> point?
You do something like (don't have the code right here)
if (quoted) appendStringInfo("= \"%s\"", value).
what happens if value contains a "?
> > * coding style should be adhered to more closesly, there are many
> > if (pointer) which should be if (pointer != NULL),
>
> Are you pointing in function validate_conf_option(const char *name, char *value)
> for below usage:
> + if (value)
For example, yes.
> > * the check that prevents persistent SETs in a transaction should
> > rather
> > be in utility.c and use PreventTransactionChain like most of the
> > others that need to do that (c.f. T_CreatedbStmt).
>
> We can move the check in utility.c, but if we use PreventTransactionChain,
> then it will be disallowed in functions as well. But the original idea was to not support in
> transaction blocks only.
> So I feel use of current function IsTransactionBlock() should be okay.
I don't think its even remotely safe to allow doing this from a
function. Doing it there explicitly allows doing it in a transaction.
Should we find out its safe, fine, relaxing restrictions lateron is no
problem, imposing ones after introducing a feature is.
Greetings,
Andres Freund
--Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services
On Thursday, January 24, 2013 5:25 PM Andres Freund wrote:
> On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
> > > * The gram.y changes arround set_rest_(more|common) seem pretty
> > > confused
> > > to me.
> >
> > >E.g. its not possible anymore to set the timezone for a function.
> >
> > What do you exactly mean by this part of comment.
>
> The set_rest_more production is used in FunctionSetResetClause and
> youve
> removed capabilities from it.
set_rest_more has set_reset_common, so there should be no problem.
set_rest_more: /* Generic SET syntaxes: */ set_rest_common |
var_nameFROM CURRENT_P { VariableSetStmt *n =
makeNode(VariableSetStmt); n->kind = VAR_SET_CURRENT;
n->name = $1; $$ = n; }
> > > And why is it possible to persistently set the search path,
> > > but not client encoding? Why is FROM CURRENT in set_rest_more?
> >
> > I think the main reason was some of the commands can work only in
> > transaction and as SET Persistent cannot work inside transaction
> block, so I had kept
> > some seggregation.
>
> Yes, I can see reasons for doing this, I just think the split isn't
> correct as youve done it.
>
> > > * Writing the temporary file to .$pid seems like a bad idea, better
> use
> > > one file for that, SET PERSISTENT is protected by an exclusive
> lock
> > > anyway.
> >
> > I think we can use one temporary file, infact that was one of the
> ways I
> > have asked in one of the previous mails.
> > However Tom and Zoltan felt this is better way to do it.
> The have? I didn't read it like that. The file can only ever written by
> a running postmaster and we already have code that ensures that.
> There's
> absolutely no need for the tempfile to have a nondeterministic
> name. That makes cleanup way easier as well.
Sure, I understand that cleaning will be easier. So IIUC you are suggesting,
we should just keep
name as postgresql.auto.conf.tmp
In general, please read the below mail where it has been suggested to use
.$pid
http://www.postgresql.org/message-id/28379.1358541135@sss.pgh.pa.us
> > I can think of one reason where it will be better to have .$pid,
> and that
> > is if some one has opened the
> > file manually, then all other sessions can fail (in WINDOWS).
> Infact this
> > is one of test Zoltan had performed.
>
> Why on earth should somebody have the tempfile open? There's an
> exclusive lock around writing the file in your code and if anybody
> interferes like that with postgres' temporary data you have far bigger
> problems than SET PERSISTENT erroring out.
I am also not sure, but may be some people do for test purpose.
> > > * the write sequence should be:
> > > * fsync(tempfile)
> > > * fsync(directory)
> > > * rename(tempfile, configfile)
> > > * fsync(configfile)
> > > * fsync(directory)
> >
> > Why do we need fsync(directory) and fsync(configfile)?
>
> So they don't vanish /get corrupted after a crash? The above is the
> canonically safe way to safely write a file without an invalid file
> ever
> being visible.
Do you think there will be problem if we just use as below:
* fsync(tempfile)
* rename(tempfile, configfile)
I have seen such kind of use elsewhere also in code (writeTimeLineHistory())
> > > * write_auto_conf_file should probably escape quoted values?
> > Can you please elaborate more, I am not able to understand your
> > point?
>
> You do something like (don't have the code right here)
> if (quoted)
> appendStringInfo("= \"%s\"", value).
> what happens if value contains a "?
I think this case is missed. I shall take care to handle the case when value
contains such value.
> > > * coding style should be adhered to more closesly, there are many
> > > if (pointer) which should be if (pointer != NULL),
> >
> > Are you pointing in function validate_conf_option(const char *name,
> char *value)
> > for below usage:
> > + if (value)
>
> For example, yes.
Okay, I shall fix these, but I have seen such code at other places
(set_config_option), that's why I have kept it like how it is currently.
>
> > > * the check that prevents persistent SETs in a transaction should
> > > rather
> > > be in utility.c and use PreventTransactionChain like most of the
> > > others that need to do that (c.f. T_CreatedbStmt).
> >
> > We can move the check in utility.c, but if we use
> PreventTransactionChain,
> > then it will be disallowed in functions as well. But the original
> idea was to not support in
> > transaction blocks only.
> > So I feel use of current function IsTransactionBlock() should be
> okay.
>
> I don't think its even remotely safe to allow doing this from a
> function. Doing it there explicitly allows doing it in a transaction.
As SET command is allowed inside a function, so I don't think without any
reason we should disallow SET PERSISTENT in function.
The reason why it's not allowed in transaction is that we have to delete the
temporary file in transaction end or at rollback which could have made the
logic much more complex.
> Should we find out its safe, fine, relaxing restrictions lateron is no
> problem, imposing ones after introducing a feature is.
I have discussed most of these restrictions in this thread. But there can be
few which I could have missed.
With Regards,
Amit Kapila.
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-01-24 18:37:29 +0530, Amit Kapila wrote:
> On Thursday, January 24, 2013 5:25 PM Andres Freund wrote:
> > On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
> > > > * The gram.y changes arround set_rest_(more|common) seem pretty
> > > > confused
> > > > to me.
> > >
> > > >E.g. its not possible anymore to set the timezone for a function.
> > >
> > > What do you exactly mean by this part of comment.
> >
> > The set_rest_more production is used in FunctionSetResetClause and
> > youve
> > removed capabilities from it.
>
> set_rest_more has set_reset_common, so there should be no problem.
>
> set_rest_more: /* Generic SET syntaxes: */
> set_rest_common
> | var_name FROM CURRENT_P
> {
> VariableSetStmt *n = makeNode(VariableSetStmt);
> n->kind = VAR_SET_CURRENT;
> n->name = $1;
> $$ = n;
> }
True. I still think that the split youve made isn't right as-is.
> > > > * Writing the temporary file to .$pid seems like a bad idea, better use
> > > > one file for that, SET PERSISTENT is protected by an exclusive lock
> > > > anyway.
> > >
> > > I think we can use one temporary file, infact that was one of the ways I
> > > have asked in one of the previous mails.
> > > However Tom and Zoltan felt this is better way to do it.
> > The have? I didn't read it like that. The file can only ever written by
> > a running postmaster and we already have code that ensures that.
> > There's
> > absolutely no need for the tempfile to have a nondeterministic
> > name. That makes cleanup way easier as well.
>
> Sure, I understand that cleaning will be easier. So IIUC you are suggesting,
> we should just keep
> name as postgresql.auto.conf.tmp
>
> In general, please read the below mail where it has been suggested to use
> .$pid
> http://www.postgresql.org/message-id/28379.1358541135@sss.pgh.pa.us
That was in the context of your use of mkstemp() as far as I read
it. We use constantly named temp files in other locations as well.
> > Why on earth should somebody have the tempfile open? There's an
> > exclusive lock around writing the file in your code and if anybody
> > interferes like that with postgres' temporary data you have far bigger
> > problems than SET PERSISTENT erroring out.
>
> I am also not sure, but may be some people do for test purpose.
That seems like an completely pointless reasoning.
> > > > * the write sequence should be:
> > > > * fsync(tempfile)
> > > > * fsync(directory)
> > > > * rename(tempfile, configfile)
> > > > * fsync(configfile)
> > > > * fsync(directory)
> > >
> > > Why do we need fsync(directory) and fsync(configfile)?
> >
> > So they don't vanish /get corrupted after a crash? The above is the
> > canonically safe way to safely write a file without an invalid file
> > ever
> > being visible.
>
> Do you think there will be problem if we just use as below:
> * fsync(tempfile)
> * rename(tempfile, configfile)
>
> I have seen such kind of use elsewhere also in code (writeTimeLineHistory())
Yea, there are some places where the code isn't entirely safe. No reason
to introduce more of those.
> > > > * the check that prevents persistent SETs in a transaction should
> > > > rather
> > > > be in utility.c and use PreventTransactionChain like most of the
> > > > others that need to do that (c.f. T_CreatedbStmt).
> > >
> > > We can move the check in utility.c, but if we use
> > PreventTransactionChain,
> > > then it will be disallowed in functions as well. But the original
> > idea was to not support in
> > > transaction blocks only.
> > > So I feel use of current function IsTransactionBlock() should be
> > okay.
> >
> > I don't think its even remotely safe to allow doing this from a
> > function. Doing it there explicitly allows doing it in a transaction.
>
> As SET command is allowed inside a function, so I don't think without any
> reason we should disallow SET PERSISTENT in function.
> The reason why it's not allowed in transaction is that we have to delete the
> temporary file in transaction end or at rollback which could have made the
> logic much more complex.
Yea, and allowing use in functions doesn't help you with that at all:
Consider the following plpgsql function:
$$
BEGIN SET PERSISTENT foo = 'bar'; RAISE ERROR 'blub';
END;
$$
Now you have a SET PERSISTENT that succeeded although the transaction
failed.
Greetings,
Andres Freund
-- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Alvaro Herrera
Date:
Amit Kapila escribió: > On Wednesday, January 23, 2013 11:51 PM Alvaro Herrera wrote: > > Yes it is -- the /etc/postgresql/<version>/<cluster> directory (where > > postgresql.conf resides) is owned by user postgres. > > So in that case we can consider postgresql.auto.conf to be in path w.r.t > postgresql.conf instead of data directory. That seems sensible to me. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Thursday, January 24, 2013 6:51 PM Andres Freund wrote:
> On 2013-01-24 18:37:29 +0530, Amit Kapila wrote:
> > On Thursday, January 24, 2013 5:25 PM Andres Freund wrote:
> > > On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
> > > > > * The gram.y changes arround set_rest_(more|common) seem pretty
> > > > > confused
> > > > > to me.
> > > >
> > > > >E.g. its not possible anymore to set the timezone for a
> function.
> > > >
> > > > What do you exactly mean by this part of comment.
> > >
> > > The set_rest_more production is used in FunctionSetResetClause and
> > > youve
> > > removed capabilities from it.
> >
> > set_rest_more has set_reset_common, so there should be no problem.
> >
> > set_rest_more: /* Generic SET syntaxes: */
> > set_rest_common
> > | var_name FROM CURRENT_P
> > {
> > VariableSetStmt *n =
> makeNode(VariableSetStmt);
> > n->kind = VAR_SET_CURRENT;
> > n->name = $1;
> > $$ = n;
> > }
>
> True. I still think that the split youve made isn't right as-is.
I am working on to provide the exact reasoning of split and if some variable
is not appropriate,
I shall do the appropriate movement.
> > > > > * Writing the temporary file to .$pid seems like a bad idea,
> better use
> > > > > one file for that, SET PERSISTENT is protected by an
> exclusive lock
> > > > > anyway.
> > > >
> > > > I think we can use one temporary file, infact that was one of
> the ways I
> > > > have asked in one of the previous mails.
> > > > However Tom and Zoltan felt this is better way to do it.
>
> > > The have? I didn't read it like that. The file can only ever
> written by
> > > a running postmaster and we already have code that ensures that.
> > > There's
> > > absolutely no need for the tempfile to have a nondeterministic
> > > name. That makes cleanup way easier as well.
> >
> > Sure, I understand that cleaning will be easier. So IIUC you are
> suggesting,
> > we should just keep
> > name as postgresql.auto.conf.tmp
> >
> > In general, please read the below mail where it has been suggested to
> use
> > .$pid
> > http://www.postgresql.org/message-id/28379.1358541135@sss.pgh.pa.us
>
> That was in the context of your use of mkstemp() as far as I read
> it. We use constantly named temp files in other locations as well.
Okay, as I told you that I have also proposed initially, so as now you have
pointed it specifically,
I shall do it that way unless somebody will have another strong point of not
doing it this way.
> > > Why on earth should somebody have the tempfile open? There's an
> > > exclusive lock around writing the file in your code and if anybody
> > > interferes like that with postgres' temporary data you have far
> bigger
> > > problems than SET PERSISTENT erroring out.
> >
> > I am also not sure, but may be some people do for test purpose.
>
> That seems like an completely pointless reasoning.
I have no specific reason, so I shall do it the way you have suggested.
> > > > > * the write sequence should be:
> > > > > * fsync(tempfile)
> > > > > * fsync(directory)
> > > > > * rename(tempfile, configfile)
> > > > > * fsync(configfile)
> > > > > * fsync(directory)
> > > >
> > > > Why do we need fsync(directory) and fsync(configfile)?
> > >
> > > So they don't vanish /get corrupted after a crash? The above is the
> > > canonically safe way to safely write a file without an invalid file
> > > ever
> > > being visible.
> >
> > Do you think there will be problem if we just use as below:
> > * fsync(tempfile)
> > * rename(tempfile, configfile)
> >
> > I have seen such kind of use elsewhere also in code
> (writeTimeLineHistory())
>
> Yea, there are some places where the code isn't entirely safe. No
> reason
> to introduce more of those.
Okay, I will check this and let you know if I have any doubts.
> > > > > * the check that prevents persistent SETs in a transaction
> should
> > > > > rather
> > > > > be in utility.c and use PreventTransactionChain like most of
> the
> > > > > others that need to do that (c.f. T_CreatedbStmt).
> > > >
> > > > We can move the check in utility.c, but if we use
> > > PreventTransactionChain,
> > > > then it will be disallowed in functions as well. But the
> original
> > > idea was to not support in
> > > > transaction blocks only.
> > > > So I feel use of current function IsTransactionBlock() should
> be
> > > okay.
> > >
> > > I don't think its even remotely safe to allow doing this from a
> > > function. Doing it there explicitly allows doing it in a
> transaction.
> >
> > As SET command is allowed inside a function, so I don't think without
> any
> > reason we should disallow SET PERSISTENT in function.
> > The reason why it's not allowed in transaction is that we have to
> delete the
> > temporary file in transaction end or at rollback which could have
> made the
> > logic much more complex.
>
> Yea, and allowing use in functions doesn't help you with that at all:
>
> Consider the following plpgsql function:
>
> $$
> BEGIN
> SET PERSISTENT foo = 'bar';
> RAISE ERROR 'blub';
> END;
> $$
>
> Now you have a SET PERSISTENT that succeeded although the transaction
> failed.
I think the core point of this command behavior is that it is not related to
transaction boundary.
So now the question is in such case if we don't allow this command inside
transaction block, why to allow inside
Function?
When user starts transaction and runs this command and then few other
commands and now if he rollback, this command cannot be rolledback.
OTOH in function there is no explicit way to issue rollback, only an error
can cause rollback which is same as when running outside block.
With Regards,
Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Tom Lane
Date:
Andres Freund <andres@2ndquadrant.com> writes:
> On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
>>> * Writing the temporary file to .$pid seems like a bad idea, better use
>>> one file for that, SET PERSISTENT is protected by an exclusive lock
>>> anyway.
>> I think we can use one temporary file, infact that was one of the ways I
>> have asked in one of the previous mails.
>> However Tom and Zoltan felt this is better way to do it.
> The have? I didn't read it like that. The file can only ever written by
> a running postmaster and we already have code that ensures that. There's
> absolutely no need for the tempfile to have a nondeterministic
> name. That makes cleanup way easier as well.
Say again? Surely the temp file is being written by whichever backend
is executing SET PERSISTENT, and there could be more than one.
Or, if it isn't and you really do mean that this patch is getting the
postmaster process involved in executing SET PERSISTENT, that's going to
be sufficient grounds for rejection right there. The only way that we
have to keep the postmaster reliable is to not have it doing more than
it absolutely must.
While I have not been paying very much attention to this patch, I have
to say that every time I've looked into the thread I've gotten an
overwhelming impression of overdesign and overcomplication. This should
not be a very large or subtle patch. Write a setting into a temp file,
rename it into place, done.
regards, tom lane
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-01-24 11:22:52 -0500, Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On 2013-01-24 16:45:42 +0530, Amit Kapila wrote: > >>> * Writing the temporary file to .$pid seems like a bad idea, better use > >>> one file for that, SET PERSISTENT is protected by an exclusive lock > >>> anyway. > > >> I think we can use one temporary file, infact that was one of the ways I > >> have asked in one of the previous mails. > >> However Tom and Zoltan felt this is better way to do it. > > > The have? I didn't read it like that. The file can only ever written by > > a running postmaster and we already have code that ensures that. There's > > absolutely no need for the tempfile to have a nondeterministic > > name. That makes cleanup way easier as well. > > Say again? Surely the temp file is being written by whichever backend > is executing SET PERSISTENT, and there could be more than one. Sure, but the patch acquires SetPersistentLock exlusively beforehand which seems fine to me. Any opinion whether its acceptable to allow SET PERSISTENT in functions? It seems absurd to me to allow it, but Amit seems to be of another opinion. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Tom Lane
Date:
Andres Freund <andres@2ndquadrant.com> writes:
> On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
>> Say again? Surely the temp file is being written by whichever backend
>> is executing SET PERSISTENT, and there could be more than one.
> Sure, but the patch acquires SetPersistentLock exlusively beforehand
> which seems fine to me.
Why should we have such a lock? Seems like that will probably introduce
as many problems as it fixes. Deadlock risk, blockages, etc. It is not
necessary for atomicity, since rename() would be atomic already.
> Any opinion whether its acceptable to allow SET PERSISTENT in functions?
> It seems absurd to me to allow it, but Amit seems to be of another
> opinion.
Well, it's really a definitional question I think: do you expect that
subsequent failure of the transaction should cause such a SET to roll
back?
I think it would be entirely self-consistent to define SET PERSISTENT as
a nontransactional operation. Then the implementation would just be to
write the file immediately when the command is executed, and there's no
particular reason why it can't be allowed inside a transaction block.
If you want it to be transactional, then the point of disallowing it in
transaction blocks (or functions) would be to not have a very large
window between writing the file and committing. But it's still possible
that the transaction would fail somewhere in there, leading to the
inconsistent outcome that the transaction reports failing but we applied
the SET anyway. I do agree that it would be nonsensical to allow SET
PERSISTENT in functions but not transaction blocks.
Another approach is to remember the requested setting until somewhere in
the pre-commit sequence, and then try to do the file write at that time.
I'm not terribly thrilled with that approach, though, because (a) it
only narrows the window for an inconsistent outcome, it doesn't remove
it entirely; (b) there are already too many things that want to be the
"last thing done before commit"; and (c) it adds complexity and overhead
that I'd just as soon this patch not add. But if you want S.P. to be
transactional and allowed inside functions, I think this would be the
only acceptable implementation.
regards, tom lane
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-01-24 12:02:59 -0500, Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On 2013-01-24 11:22:52 -0500, Tom Lane wrote: > >> Say again? Surely the temp file is being written by whichever backend > >> is executing SET PERSISTENT, and there could be more than one. > > > Sure, but the patch acquires SetPersistentLock exlusively beforehand > > which seems fine to me. > > Why should we have such a lock? Seems like that will probably introduce > as many problems as it fixes. Deadlock risk, blockages, etc. It is not > necessary for atomicity, since rename() would be atomic already. Well, the patch acquires it as-is, so I made the judgement based on that. But I think that lock isn't neccesarily a bad idea. While the replacement of the values clearly is atomic due to the rename I think there's another confusing behaviour lurking: Backend A: starts Backend B: starts Backend A: reads the config Backend B: reads the config Backend A: does SET PERSISTENT foobar =..; Backend B: does SET PERSISTENT foobar =..; Now B overwrites the config change A has made as they are all stored in the same file. So the only safe way to do this seems to be: LWLockAcquire(SetPersistentLock, LW_EXCLUSIVE); ProcessConfigFile(); validate_option(); write_rename(); LWLockRelease(); We can decide not to care about the above case but the window isn't that small if no reload is implied and so this seems to be overly grotty. > > Any opinion whether its acceptable to allow SET PERSISTENT in functions? > > It seems absurd to me to allow it, but Amit seems to be of another > > opinion. > > Well, it's really a definitional question I think: do you expect that > subsequent failure of the transaction should cause such a SET to roll > back? > > I think it would be entirely self-consistent to define SET PERSISTENT as > a nontransactional operation. Then the implementation would just be to > write the file immediately when the command is executed, and there's no > particular reason why it can't be allowed inside a transaction block. Thats how its implemented atm except for not allowing it in transactions. I think the reason I am weary of allowing it inside transaction is that I think the config file needs to be reloaded before writing the new one and it seems dangerous to me to reload the config in all the possible situations a function can be called. Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes:
> Backend A: does SET PERSISTENT foobar =..;
> Backend B: does SET PERSISTENT foobar =..;
> Now B overwrites the config change A has made as they are all stored in
> the same file.
Say what? I thought the plan was one setting per file, so that we don't
get involved in having to parse-and-edit the file contents. What was
all that argument about a new directory, if we're only using one file?
If we are using just one file, then I agree a lock would be needed to
synchronize updates. But that seems to add a lot of complication
elsewhere.
regards, tom lane
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-01-24 12:30:02 -0500, Tom Lane wrote: > Andres Freund <andres@anarazel.de> writes: > > Backend A: does SET PERSISTENT foobar =..; > > Backend B: does SET PERSISTENT foobar =..; > > > Now B overwrites the config change A has made as they are all stored in > > the same file. > > Say what? I thought the plan was one setting per file, so that we don't > get involved in having to parse-and-edit the file contents. What was > all that argument about a new directory, if we're only using one file? I initially thought that as well (and voted for it) but after reskimming the thread and reading the patch that doesn't seem to be the case unless its implemented in a way I don't understand: +#define PG_AUTOCONF_DIR "auto.conf.d" +#define PG_AUTOCONF_FILENAME "postgresql.auto.conf" + /* Frame auto conf name and auto conf sample temp file name */ + snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s", + ConfigFileDir, + PG_AUTOCONF_DIR, + PG_AUTOCONF_FILENAME); + snprintf(AutoConfTmpFileName, sizeof(AutoConfTmpFileName),"%s/%s/%s.%d", + ConfigFileDir, + PG_AUTOCONF_DIR, + PG_AUTOCONF_FILENAME, + (int) getpid()); + I don't understand what the conf.d is all about either if only one file is going to be used. > If we are using just one file, then I agree a lock would be needed to > synchronize updates. But that seems to add a lot of complication > elsewhere. More people seem to have voted for the single file approach but I still haven't understood why... Greetings, Andres Freund --Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Thursday, January 24, 2013 10:33 PM Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On 2013-01-24 11:22:52 -0500, Tom Lane wrote: > >> Say again? Surely the temp file is being written by whichever > backend > >> is executing SET PERSISTENT, and there could be more than one. > > > Sure, but the patch acquires SetPersistentLock exlusively beforehand > > which seems fine to me. > > Why should we have such a lock? Seems like that will probably > introduce > as many problems as it fixes. Deadlock risk, blockages, etc. It is > not > necessary for atomicity, since rename() would be atomic already. > > > Any opinion whether its acceptable to allow SET PERSISTENT in > functions? > > It seems absurd to me to allow it, but Amit seems to be of another > > opinion. > > Well, it's really a definitional question I think: do you expect that > subsequent failure of the transaction should cause such a SET to roll > back? > > I think it would be entirely self-consistent to define SET PERSISTENT > as > a nontransactional operation. Then the implementation would just be to > write the file immediately when the command is executed, and there's no > particular reason why it can't be allowed inside a transaction block. > > If you want it to be transactional, then the point of disallowing it in > transaction blocks (or functions) would be to not have a very large > window between writing the file and committing. But it's still > possible > that the transaction would fail somewhere in there, leading to the > inconsistent outcome that the transaction reports failing but we > applied > the SET anyway. I do agree that it would be nonsensical to allow SET > PERSISTENT in functions but not transaction blocks. As we have agreed on before that this command has to be non-transactional, so lets stick on that and about allowing it in functions and not in transaction blocks, I had some reasoning behind it as below, but as it is not making much sense we can disallow it for both transaction blocks as well as functions: Reasoning for disallowing in transaction blocks and allowing in functions: ---------------------------------------------------------------------------- -- " I think the core point of this command behavior is that it is not related to transaction boundary. So now the question is in such case if we don't allow this command inside transaction block, why to allow inside Function? When user starts transaction and runs this command and then few other commands and now if he rollback, this command cannot be rolledback. OTOH in function there is no explicit way to issue rollback, only an error can cause rollback which is same as when running outside block." > Say what? I thought the plan was one setting per file, so that we don't get involved > in having to parse-and-edit the file contents. > What was all that argument about a new directory, if we're only using one file? A new directory (named as auto.conf.d, initially in the patch named as config_dir) is for having automatically generated configuration files. Also it was suggested previously that it is better to have this file in directory instead of standalone file. > If we are using just one file, then I agree a lock would be needed to synchronize > updates. But that seems to add a lot of complication elsewhere. There were 2 ways to protect the file write, a. one is by having lock SetPersistentLock b. another is by opening file in X mode and if open fails wait and retry for sometime and then give error. As suggested in this thread I have implemented as Approach-1. What is the complication it can add, can we resolve that? With Regards, Amit Kapila.
On Thursday, January 24, 2013 6:51 PM Andres Freund wrote:
> On 2013-01-24 18:37:29 +0530, Amit Kapila wrote:
> > On Thursday, January 24, 2013 5:25 PM Andres Freund wrote:
> > > On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
> > > > > * The gram.y changes arround set_rest_(more|common) seem pretty
> > > > > confused
> > > > > to me.
> > > >
> > > > >E.g. its not possible anymore to set the timezone for a
> function.
> > > >
> > > > What do you exactly mean by this part of comment.
> > >
> > > The set_rest_more production is used in FunctionSetResetClause and
> > > youve
> > > removed capabilities from it.
> >
> > set_rest_more has set_reset_common, so there should be no problem.
> >
> > set_rest_more: /* Generic SET syntaxes: */
> > set_rest_common
> > | var_name FROM CURRENT_P
> > {
> > VariableSetStmt *n =
> makeNode(VariableSetStmt);
> > n->kind = VAR_SET_CURRENT;
> > n->name = $1;
> > $$ = n;
> > }
>
> True. I still think that the split youve made isn't right as-is.
We can move FROM CURRENT and client_encoding to common part.
Other syntax (ROLE, SESSION AUTHORIZATION, TRANSACTION SNAPSHOT) doesn't
exist in postgresql.conf and requires a transaction to set and during reload
we don't have transaction, so we should not allow them.
With Regards,
Amit Kapila.
Thanks to both (Andres and Fujii Masao) of you for review. Please find the patch attached with this mail which contains fixes for the comments raised by you. Kindly confirm if the fixes are okay or if I have missed any point raised by you. On Thursday, January 24, 2013 1:56 AM Andres Freund wrote: > On 2013-01-22 12:32:07 +0000, Amit kapila wrote: > > This closes all comments raised till now for this patch. > > Kindly let me know if you feel something is missing? > > I am coming late to this patch, so bear with me if I repeat somethign > said elsewhere. > > Review comments of cursory pass through the patch: > * most comments are hard to understand. I know the problem of that > being hard for a non-native speaker by heart, but I think another > pass > over them would be good thing. I have changed the comments in most parts of code. > * The gram.y changes arround set_rest_(more|common) seem pretty > confused > to me. E.g. its not possible anymore to set the timezone for a > function. And why is it possible to persistently set the search path, > but not client encoding? Why is FROM CURRENT in set_rest_more? client encoding and FROM CURRENT syntax will be allowed in SET PERSISTENT. Others cannot be moved as they need transactionto set them and they are even not present in postgresql.conf file > * set_config_file should elog(ERROR), not return on an unhandled > setstmt->kind Fixed. > * why are you creating AutoConfFileName if its not stat'able? It seems > better to simply skip parsing the old file in that case Fixed. > * Writing the temporary file to .$pid seems like a bad idea, better use > one file for that, SET PERSISTENT is protected by an exclusive lock > anyway. now only one file (postgresql.auto.conf.temp) will be created > * the write sequence should be: > * fsync(tempfile) > * fsync(directory) > * rename(tempfile, configfile) > * fsync(configfile) > * fsync(directory) to do fsync of directory we need to open the directory and then do the fsync. rename is an atomic operation, so it will always point to either new or old file. I am just not sure whether to modify the code to handle this scenario. So for now I have added the comment atthat place. > * write_auto_conf_file should probably escape quoted values? Fixed. Changed write_auto_conf_file() to handle it. > * coding style should be adhered to more closesly, there are many > if (pointer) which should be if (pointer != NULL), single-line blocks > enclosed in curlies which shouldn't, etc. Fixed. > * replace_auto_config_file and surrounding functions need more comments > in the header Added more comments. > * the check that prevents persistent SETs in a transaction should > rather > be in utility.c and use PreventTransactionChain like most of the > others that need to do that (c.f. T_CreatedbStmt). Fixed. Now this command will not work for transaction blocks or Functions. Fujii Masao Comments --------------------- > When I removed postgresql.auto.conf and restarted the server, I got > the following warning message. This is not correct because I didn't > remove "auto.conf.d" from postgresql.conf. What I removed is only > postgresql.auto.conf. > > WARNING: Default "auto.conf.d" is not present in postgresql.conf. > Configuration parameters changed with SET PERSISTENT command will not > be effective. > Warning message is changed as per discussion in another mail. > Is it safe to write something in the directory other than data > directory via SQL? As discussed, it is okay to write in config directory, so I have not changed anything for this. > The description of RESET PERSISTENT is in the document, but it seems > not to be implemented. RESET PERSISTENT description is removed from patch. > storage.sgml needs to be updated. > Doesn't auto.conf.d need to be explained in config.sgml? Updated both storage.sgml and config.sgml > This warning message implies that the line "include_dir 'auto.conf.d'" > must not be removed from postgresql.conf? If so, we should warn that in both document and postgresql.conf.sample? I have modified the WARNING in postgresql.conf.sample, I have thought of modifying runtime.sgml but not sure if that is the right place, can you please suggest which document would be better to have this warning? With Regards, Amit Kapila.
On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote: > More people seem to have voted for the single file approach but I still > haven't understood why... Me neither. Having an include directory seems good, but I can't think why we'd want to clutter it up with a bajillion automatically generated files. One .auto file that gets overwritten at need seems way nicer. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Kevin Grittner
Date:
Robert Haas <robertmhaas@gmail.com> wrote: > Andres Freund <andres@2ndquadrant.com> wrote: >> More people seem to have voted for the single file approach but I still >> haven't understood why... > > Me neither. Having an include directory seems good, but I can't think > why we'd want to clutter it up with a bajillion automatically > generated files. One .auto file that gets overwritten at need seems > way nicer. The single-value per file approach punts the concurrency issues and ordering issues to the OS, which might significantly simplify the patch. I'm not sure that the niceness of the all-in-one-file approach justifies the extra code. If you ever want to view them all in one file as of some moment in time, there is always: cat * | less -Kevin
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-01-26 07:46:50 -0500, Robert Haas wrote: > On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote: > > More people seem to have voted for the single file approach but I still > > haven't understood why... > > Me neither. Having an include directory seems good, but I can't think > why we'd want to clutter it up with a bajillion automatically > generated files. One .auto file that gets overwritten at need seems > way nicer. I unfortunately think you misunderstood what I meant. I don't really see that many arguments for putting all the variables in a single file. That is I think the one-value-per-file is preferrable. The primary reason is that the multiple values per file is just considerably harder to implement without POLA. In order to not loose values set in this or other backends you basically need to do something like: 1) exlusive lock 2) reload config file (to update in memory structures) 3) check new variable 4) write config file (needs to be done atomically) 5) optionally reload config file 6) reload lock Where as you can get acceptable behaviour in the single-value-per-file approach by: 1) check new config variable 2) write new single-variable file 3) optionally reload config file Not requiring a global lock while reading the configuration seems to make stuff noticeably simpler. The other advantages for a single value approach I know of are: * way easier to integrate into other systems, no need to parse the whole file for them * easier to get yourself out of problems if you screwed up and set a bad value (shared_buffers=16TB instead of GB ;)) Whereas the only real argument I can see for the all-in-one-file approach I can see is that it will perform better. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote:
>> More people seem to have voted for the single file approach but I still
>> haven't understood why...
> Me neither. Having an include directory seems good, but I can't think
> why we'd want to clutter it up with a bajillion automatically
> generated files. One .auto file that gets overwritten at need seems
> way nicer.
IMO an include directory containing just one file is silly. If we're
going with the single-file approach, let's lose the directory altogether
and just store the file at $PGDATA/postgresql.conf.auto.
regards, tom lane
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Craig Ringer
Date:
On 01/27/2013 01:01 AM, Tom Lane wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote: >>> More people seem to have voted for the single file approach but I still >>> haven't understood why... >> Me neither. Having an include directory seems good, but I can't think >> why we'd want to clutter it up with a bajillion automatically >> generated files. One .auto file that gets overwritten at need seems >> way nicer. > IMO an include directory containing just one file is silly. If we're > going with the single-file approach, let's lose the directory altogether > and just store the file at $PGDATA/postgresql.conf.auto. Wasn't part of the reason for having the config dir to make package managers' lives easier and make it easier to script updates to postgresql.conf? For the use of things like pg_wrapper? I think the config dir has value even if a single .auto file is used, so that packages can drop their own config snippets into it. For example, if I installed a packaged extension that had its own postgresql.conf changes or new GUCs, I'd want it to be able to drop that into the configdir, not have to script changes to my postgresql.conf. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Sat, Jan 26, 2013 at 6:47 PM, Craig Ringer <craig@2ndquadrant.com> wrote: > On 01/27/2013 01:01 AM, Tom Lane wrote: >> Robert Haas <robertmhaas@gmail.com> writes: >>> On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote: >>>> More people seem to have voted for the single file approach but I still >>>> haven't understood why... >>> Me neither. Having an include directory seems good, but I can't think >>> why we'd want to clutter it up with a bajillion automatically >>> generated files. One .auto file that gets overwritten at need seems >>> way nicer. >> IMO an include directory containing just one file is silly. If we're >> going with the single-file approach, let's lose the directory altogether >> and just store the file at $PGDATA/postgresql.conf.auto. > Wasn't part of the reason for having the config dir to make package > managers' lives easier and make it easier to script updates to > postgresql.conf? For the use of things like pg_wrapper? > > I think the config dir has value even if a single .auto file is used, so > that packages can drop their own config snippets into it. For example, > if I installed a packaged extension that had its own postgresql.conf > changes or new GUCs, I'd want it to be able to drop that into the > configdir, not have to script changes to my postgresql.conf. That was my understanding. But I just work here. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Sunday, January 27, 2013 10:07 AM Robert Haas wrote: On Sat, Jan 26, 2013 at 6:47 PM, Craig Ringer <craig@2ndquadrant.com> wrote: > On 01/27/2013 01:01 AM, Tom Lane wrote: >> Robert Haas <robertmhaas@gmail.com> writes: >>> On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote: >>>>> More people seem to have voted for the single file approach but I still >>>>> haven't understood why... >>>> Me neither. Having an include directory seems good, but I can't think >>>> why we'd want to clutter it up with a bajillion automatically >>>> generated files. One .auto file that gets overwritten at need seems >>>> way nicer. >>> IMO an include directory containing just one file is silly. If we're >>> going with the single-file approach, let's lose the directory altogether >>> and just store the file at $PGDATA/postgresql.conf.auto. >> Wasn't part of the reason for having the config dir to make package >> managers' lives easier and make it easier to script updates to >> postgresql.conf? For the use of things like pg_wrapper? > >> I think the config dir has value even if a single .auto file is used, so >> that packages can drop their own config snippets into it. For example, >> if I installed a packaged extension that had its own postgresql.conf >> changes or new GUCs, I'd want it to be able to drop that into the >> configdir, not have to script changes to my postgresql.conf. > That was my understanding. But I just work here. Current implementation of patch is to have directory and single file. However if user puts other files, it gives LOG: unexpected file found in automatic configuration directory . As now Craig has explained the usecase for having other files and previously Fujii Masao has also reported similar usecase, I think it is better to remove the LOG message from code. Currently the directory is named as auto.conf.d, so if we allow users to place other files, is this name okay or shall we change it to something like config_dir? With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
2013-01-24 18:02 keltezéssel, Tom Lane írta: > Andres Freund <andres@2ndquadrant.com> writes: >> On 2013-01-24 11:22:52 -0500, Tom Lane wrote: >>> Say again? Surely the temp file is being written by whichever backend >>> is executing SET PERSISTENT, and there could be more than one. >> Sure, but the patch acquires SetPersistentLock exlusively beforehand >> which seems fine to me. > Why should we have such a lock? Seems like that will probably introduce > as many problems as it fixes. Deadlock risk, blockages, etc. It is not > necessary for atomicity, since rename() would be atomic already. There is a problem when running SET PERSISTENT for different GUCs in parallel. All happen to read the same original file, and only one setting ends up in the result if you rely only on the rename() being atomic. The LWLock provides the serialization for that problem. > >> Any opinion whether its acceptable to allow SET PERSISTENT in functions? >> It seems absurd to me to allow it, but Amit seems to be of another >> opinion. > Well, it's really a definitional question I think: do you expect that > subsequent failure of the transaction should cause such a SET to roll > back? > > I think it would be entirely self-consistent to define SET PERSISTENT as > a nontransactional operation. Then the implementation would just be to > write the file immediately when the command is executed, and there's no > particular reason why it can't be allowed inside a transaction block. > > If you want it to be transactional, then the point of disallowing it in > transaction blocks (or functions) would be to not have a very large > window between writing the file and committing. But it's still possible > that the transaction would fail somewhere in there, leading to the > inconsistent outcome that the transaction reports failing but we applied > the SET anyway. I do agree that it would be nonsensical to allow SET > PERSISTENT in functions but not transaction blocks. > > Another approach is to remember the requested setting until somewhere in > the pre-commit sequence, and then try to do the file write at that time. > I'm not terribly thrilled with that approach, though, because (a) it > only narrows the window for an inconsistent outcome, it doesn't remove > it entirely; (b) there are already too many things that want to be the > "last thing done before commit"; and (c) it adds complexity and overhead > that I'd just as soon this patch not add. But if you want S.P. to be > transactional and allowed inside functions, I think this would be the > only acceptable implementation. > > regards, tom lane > > -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote: > 2013-01-24 18:02 keltezéssel, Tom Lane írta: > >Andres Freund <andres@2ndquadrant.com> writes: > >>On 2013-01-24 11:22:52 -0500, Tom Lane wrote: > >>>Say again? Surely the temp file is being written by whichever backend > >>>is executing SET PERSISTENT, and there could be more than one. > >>Sure, but the patch acquires SetPersistentLock exlusively beforehand > >>which seems fine to me. > >Why should we have such a lock? Seems like that will probably introduce > >as many problems as it fixes. Deadlock risk, blockages, etc. It is not > >necessary for atomicity, since rename() would be atomic already. > > There is a problem when running SET PERSISTENT for different GUCs > in parallel. All happen to read the same original file, and only one > setting ends up in the result if you rely only on the rename() being atomic. > The LWLock provides the serialization for that problem. Tom was voting for one-setting-per-file, in that case the problem doesn't exist. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
2013-01-26 13:46 keltezéssel, Robert Haas írta: > On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote: >> More people seem to have voted for the single file approach but I still >> haven't understood why... > Me neither. Having an include directory seems good, but I can't think > why we'd want to clutter it up with a bajillion automatically > generated files. One .auto file that gets overwritten at need seems > way nicer. > Sarcasm at work. Nice. :-) -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
2013-02-11 15:25 keltezéssel, Andres Freund írta: > On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote: >> 2013-01-24 18:02 keltezéssel, Tom Lane írta: >>> Andres Freund <andres@2ndquadrant.com> writes: >>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote: >>>>> Say again? Surely the temp file is being written by whichever backend >>>>> is executing SET PERSISTENT, and there could be more than one. >>>> Sure, but the patch acquires SetPersistentLock exlusively beforehand >>>> which seems fine to me. >>> Why should we have such a lock? Seems like that will probably introduce >>> as many problems as it fixes. Deadlock risk, blockages, etc. It is not >>> necessary for atomicity, since rename() would be atomic already. >> There is a problem when running SET PERSISTENT for different GUCs >> in parallel. All happen to read the same original file, and only one >> setting ends up in the result if you rely only on the rename() being atomic. >> The LWLock provides the serialization for that problem. > Tom was voting for one-setting-per-file, in that case the problem > doesn't exist. I voted for the one-file approach and was arguing from the POV of the current implementation. -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Josh Berkus
Date:
On 02/11/2013 06:33 AM, Boszormenyi Zoltan wrote: > 2013-02-11 15:25 keltezéssel, Andres Freund írta: >> On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote: >>> 2013-01-24 18:02 keltezéssel, Tom Lane írta: >>>> Andres Freund <andres@2ndquadrant.com> writes: >>>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote: >>>>>> Say again? Surely the temp file is being written by whichever >>>>>> backend >>>>>> is executing SET PERSISTENT, and there could be more than one. >>>>> Sure, but the patch acquires SetPersistentLock exlusively beforehand >>>>> which seems fine to me. >>>> Why should we have such a lock? Seems like that will probably >>>> introduce >>>> as many problems as it fixes. Deadlock risk, blockages, etc. It is >>>> not >>>> necessary for atomicity, since rename() would be atomic already. >>> There is a problem when running SET PERSISTENT for different GUCs >>> in parallel. All happen to read the same original file, and only one >>> setting ends up in the result if you rely only on the rename() being >>> atomic. >>> The LWLock provides the serialization for that problem. >> Tom was voting for one-setting-per-file, in that case the problem >> doesn't exist. > > I voted for the one-file approach and was arguing from the POV > of the current implementation. I thought we discussed this ad naseum, and decided to go with the one-single-file approach for the first round, since we already had an implementation for that. I still think that's the right approach to take with this patch; if it doesn't work out, we can go do one-file-per-setting in 9.4. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-02-11 11:17:16 -0800, Josh Berkus wrote: > On 02/11/2013 06:33 AM, Boszormenyi Zoltan wrote: > > 2013-02-11 15:25 keltezéssel, Andres Freund írta: > >> On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote: > >>> 2013-01-24 18:02 keltezéssel, Tom Lane írta: > >>>> Andres Freund <andres@2ndquadrant.com> writes: > >>>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote: > >>>>>> Say again? Surely the temp file is being written by whichever > >>>>>> backend > >>>>>> is executing SET PERSISTENT, and there could be more than one. > >>>>> Sure, but the patch acquires SetPersistentLock exlusively beforehand > >>>>> which seems fine to me. > >>>> Why should we have such a lock? Seems like that will probably > >>>> introduce > >>>> as many problems as it fixes. Deadlock risk, blockages, etc. It is > >>>> not > >>>> necessary for atomicity, since rename() would be atomic already. > >>> There is a problem when running SET PERSISTENT for different GUCs > >>> in parallel. All happen to read the same original file, and only one > >>> setting ends up in the result if you rely only on the rename() being > >>> atomic. > >>> The LWLock provides the serialization for that problem. > >> Tom was voting for one-setting-per-file, in that case the problem > >> doesn't exist. > > > > I voted for the one-file approach and was arguing from the POV > > of the current implementation. > > I thought we discussed this ad naseum, and decided to go with the > one-single-file approach for the first round, since we already had an > implementation for that. I still think that's the right approach to take > with this patch; if it doesn't work out, we can go do > one-file-per-setting in 9.4. Well, several people (at least Tom, I, and I think Zoltan as well) think that the one-file approach is considerably more complex. Check http://www.postgresql.org/message-id/20130126162728.GA5482@awork2.anarazel.de and related messages for some of the problems. Most of which are unhandled in the current patch, i.e. currently you *will* loose changes made in concurrent sessions. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Craig Ringer
Date:
On 02/12/2013 03:24 AM, Andres Freund wrote: > Well, several people (at least Tom, I, and I think Zoltan as well) > think that the one-file approach is considerably more complex. Check > http://www.postgresql.org/message-id/20130126162728.GA5482@awork2.anarazel.de > and related messages for some of the problems. Most of which are > unhandled in the current patch, i.e. currently you *will* loose > changes made in concurrent sessions. We need to read all files in the directory anyway to make the configdir approach useful for system package managers that want to create config files for extensions, akin to how /etc/yum.repos.ds/ , /etc/apt/sources.list.d/ , /etc/apache2/conf.d/ and so on work. OTOH, it'll be easier for a sysadmin to understand what's going on if Pg's auto-written config is a single file, so any system-installed settings aren't lost in a giant clutter of Pg-created files. It seems that SQL-level configuration and the config dir feature are actually more separate than it might've appeared. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote: > On 2013-02-11 11:17:16 -0800, Josh Berkus wrote: > > On 02/11/2013 06:33 AM, Boszormenyi Zoltan wrote: > > > 2013-02-11 15:25 keltezéssel, Andres Freund írta: > > >> On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote: > > >>> 2013-01-24 18:02 keltezéssel, Tom Lane írta: > > >>>> Andres Freund <andres@2ndquadrant.com> writes: > > >>>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote: > > >>>>>> Say again? Surely the temp file is being written by whichever > > >>>>>> backend > > >>>>>> is executing SET PERSISTENT, and there could be more than one. > > >>>>> Sure, but the patch acquires SetPersistentLock exlusively > beforehand > > >>>>> which seems fine to me. > > >>>> Why should we have such a lock? Seems like that will probably > > >>>> introduce > > >>>> as many problems as it fixes. Deadlock risk, blockages, etc. > It is > > >>>> not > > >>>> necessary for atomicity, since rename() would be atomic already. > > >>> There is a problem when running SET PERSISTENT for different GUCs > > >>> in parallel. All happen to read the same original file, and only > one > > >>> setting ends up in the result if you rely only on the rename() > being > > >>> atomic. > > >>> The LWLock provides the serialization for that problem. > > >> Tom was voting for one-setting-per-file, in that case the problem > > >> doesn't exist. > > > > > > I voted for the one-file approach and was arguing from the POV > > > of the current implementation. > > > > I thought we discussed this ad naseum, and decided to go with the > > one-single-file approach for the first round, since we already had an > > implementation for that. I still think that's the right approach to > take > > with this patch; if it doesn't work out, we can go do > > one-file-per-setting in 9.4. > > Well, several people (at least Tom, I, and I think Zoltan as well) > think > that the one-file approach is considerably more complex. Zoltan has reviewed this patch very thoroughly, I have never seen a comment from him that the current Patch (one-file-all-settings) approach is not as good as having one-file-per-setting approach. Zoltan, please correct me, If I am wrong. What I could understand from mails is Tom has initially suggested to have one-file-per-setting but for the current patch (one-file-all-settings) he was telling that if we wanted to go for single file approach, then there is no need for separate directory, but for that CRAIG has given a scenario where separate directory is useful. Also Robert has suggested from the beginning that (one-file-all-settings) is better approach. It took months of discussion to reach the consensus of this level, if we again want to change approach, then I think it will be tough to touch this feature again. I think it would be better if we could try to see if there are any problems in existing patch which cannot be handled because of it's current design, then it will make more sense to conclude on changing approach. > Check > http://www.postgresql.org/message- > id/20130126162728.GA5482@awork2.anarazel.de > and related messages for some of the problems. Most of which are > unhandled in the current patch, i.e. currently you *will* loose changes > made in concurrent sessions. How will somebody loose changes in concurrent sessions? Each session currently waits with LWLock if some other session is operating on file. Also after having lock they don't acquire any other lock, so there should be no chances of any other problem. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
2013-02-12 04:54 keltezéssel, Amit Kapila írta: > On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote: >> On 2013-02-11 11:17:16 -0800, Josh Berkus wrote: >>> On 02/11/2013 06:33 AM, Boszormenyi Zoltan wrote: >>>> 2013-02-11 15:25 keltezéssel, Andres Freund írta: >>>>> On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote: >>>>>> 2013-01-24 18:02 keltezéssel, Tom Lane írta: >>>>>>> Andres Freund <andres@2ndquadrant.com> writes: >>>>>>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote: >>>>>>>>> Say again? Surely the temp file is being written by whichever >>>>>>>>> backend >>>>>>>>> is executing SET PERSISTENT, and there could be more than one. >>>>>>>> Sure, but the patch acquires SetPersistentLock exlusively >> beforehand >>>>>>>> which seems fine to me. >>>>>>> Why should we have such a lock? Seems like that will probably >>>>>>> introduce >>>>>>> as many problems as it fixes. Deadlock risk, blockages, etc. >> It is >>>>>>> not >>>>>>> necessary for atomicity, since rename() would be atomic already. >>>>>> There is a problem when running SET PERSISTENT for different GUCs >>>>>> in parallel. All happen to read the same original file, and only >> one >>>>>> setting ends up in the result if you rely only on the rename() >> being >>>>>> atomic. >>>>>> The LWLock provides the serialization for that problem. >>>>> Tom was voting for one-setting-per-file, in that case the problem >>>>> doesn't exist. >>>> I voted for the one-file approach and was arguing from the POV >>>> of the current implementation. >>> I thought we discussed this ad naseum, and decided to go with the >>> one-single-file approach for the first round, since we already had an >>> implementation for that. I still think that's the right approach to >> take >>> with this patch; if it doesn't work out, we can go do >>> one-file-per-setting in 9.4. >> Well, several people (at least Tom, I, and I think Zoltan as well) >> think >> that the one-file approach is considerably more complex. > Zoltan has reviewed this patch very thoroughly, I have never seen a comment > from him that the current > Patch (one-file-all-settings) approach is not as good as having > one-file-per-setting approach. > Zoltan, please correct me, If I am wrong. I cannot recall arguing for the one-file-per-GUC way. But I haven't re-read my mails in this thread, either. > What I could understand from mails is Tom has initially suggested to have > one-file-per-setting but for the current patch > (one-file-all-settings) he was telling that if we wanted to go for single > file approach, then there > is no need for separate directory, but for that CRAIG has given a scenario > where separate directory is useful. > > Also Robert has suggested from the beginning that (one-file-all-settings) is > better approach. > > > It took months of discussion to reach the consensus of this level, if we > again want to change approach, > then I think it will be tough to touch this feature again. > > I think it would be better if we could try to see if there are any problems > in existing patch which cannot be handled because of it's current design, > then it will make more sense to conclude on changing approach. > >> Check >> http://www.postgresql.org/message- >> id/20130126162728.GA5482@awork2.anarazel.de >> and related messages for some of the problems. Most of which are >> unhandled in the current patch, i.e. currently you *will* loose changes >> made in concurrent sessions. This mail lists this order for the single file approach: > 1) exlusive lock > 2) reload config file (to update in memory structures) > 3) check new variable > 4) write config file (needs to be done atomically) > 5) optionally reload config file > 6) reload lock The patch does it this way: 1. check new variable. No problem with that, validation for proper GUC name, type of the value, etc. can be done outsidethe lock. 2. grab lock 3. parse the config file 4. write the new config file 5. release lock Reloading the config file is intentionally not done, it's even documented. You can do SELECT pg_reload_conf() after SET PERSISTENT if you need it. > How will somebody loose changes in concurrent sessions? > Each session currently waits with LWLock if some other session is operating > on file. Also after having lock they don't acquire any other lock, so there > should be no chances of any other problem. Specifically, LWLockAcquire() is called first, then ParseConfigFp() in a PG_TRY() block, so reading the original postgresql.auto.conf is serialized. No chance to lose changes done in parallel. Best regards, Zoltán Böszörményi -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote: > 2013-02-12 04:54 keltezéssel, Amit Kapila írta: > > On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote: > >> On 2013-02-11 11:17:16 -0800, Josh Berkus wrote: > >>> On 02/11/2013 06:33 AM, Boszormenyi Zoltan wrote: > >>>> 2013-02-11 15:25 keltezéssel, Andres Freund írta: > >>>>> On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote: > >>>>>> 2013-01-24 18:02 keltezéssel, Tom Lane írta: > >>>>>>> Andres Freund <andres@2ndquadrant.com> writes: > >>>>>>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote: > >>>>>>>>> Say again? Surely the temp file is being written by > whichever > >>>>>>>>> backend > >>>>>>>>> is executing SET PERSISTENT, and there could be more than > one. > >>>>>>>> Sure, but the patch acquires SetPersistentLock exlusively > >> beforehand > >>>>>>>> which seems fine to me. > >>>>>>> Why should we have such a lock? Seems like that will probably > >>>>>>> introduce > >>>>>>> as many problems as it fixes. Deadlock risk, blockages, etc. > >> It is > >>>>>>> not > >>>>>>> necessary for atomicity, since rename() would be atomic > already. > >>>>>> There is a problem when running SET PERSISTENT for different > GUCs > >>>>>> in parallel. All happen to read the same original file, and only > >> one > >>>>>> setting ends up in the result if you rely only on the rename() > >> being > >>>>>> atomic. > >>>>>> The LWLock provides the serialization for that problem. > >>>>> Tom was voting for one-setting-per-file, in that case the problem > >>>>> doesn't exist. > >>>> I voted for the one-file approach and was arguing from the POV > >>>> of the current implementation. > >>> I thought we discussed this ad naseum, and decided to go with the > >>> one-single-file approach for the first round, since we already had > an > >>> implementation for that. I still think that's the right approach to > >> take > >>> with this patch; if it doesn't work out, we can go do > >>> one-file-per-setting in 9.4. > >> Well, several people (at least Tom, I, and I think Zoltan as well) > >> think > >> that the one-file approach is considerably more complex. > > Zoltan has reviewed this patch very thoroughly, I have never seen a > comment > > from him that the current > > Patch (one-file-all-settings) approach is not as good as having > > one-file-per-setting approach. > > Zoltan, please correct me, If I am wrong. > > I cannot recall arguing for the one-file-per-GUC way. > But I haven't re-read my mails in this thread, either. Thanks for confirmation. I also don't think if ever have argument over this approach. > > What I could understand from mails is Tom has initially suggested to > have > > one-file-per-setting but for the current patch > > (one-file-all-settings) he was telling that if we wanted to go for > single > > file approach, then there > > is no need for separate directory, but for that CRAIG has given a > scenario > > where separate directory is useful. > > > > Also Robert has suggested from the beginning that (one-file-all- > settings) is > > better approach. > > > > > > It took months of discussion to reach the consensus of this level, if > we > > again want to change approach, > > then I think it will be tough to touch this feature again. > > > > I think it would be better if we could try to see if there are any > problems > > in existing patch which cannot be handled because of it's current > design, > > then it will make more sense to conclude on changing approach. > > > >> Check > >> http://www.postgresql.org/message- > >> id/20130126162728.GA5482@awork2.anarazel.de > >> and related messages for some of the problems. Most of which are > >> unhandled in the current patch, i.e. currently you *will* loose > changes > >> made in concurrent sessions. > > This mail lists this order for the single file approach: > > > 1) exlusive lock > > 2) reload config file (to update in memory structures) > > 3) check new variable > > 4) write config file (needs to be done atomically) > > 5) optionally reload config file > > 6) reload lock > > The patch does it this way: > 1. check new variable. No problem with that, validation for proper GUC > name, > type of the value, etc. can be done outside the lock. > 2. grab lock > 3. parse the config file > 4. write the new config file > 5. release lock Before step-3, we open the auto conf file and in step-3 parse will update in-memory structures. So I think Andres's Step-2 is same as the missing step and Step-3 mentioned by you. > Reloading the config file is intentionally not done, it's even > documented. > You can do SELECT pg_reload_conf() after SET PERSISTENT if you need it. > > > How will somebody loose changes in concurrent sessions? > > Each session currently waits with LWLock if some other session is > operating > > on file. Also after having lock they don't acquire any other lock, so > there > > should be no chances of any other problem. > > Specifically, LWLockAcquire() is called first, then ParseConfigFp() > in a PG_TRY() block, so reading the original postgresql.auto.conf > is serialized. No chance to lose changes done in parallel. True. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-02-12 14:57:51 +0530, Amit Kapila wrote: > On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote: > > 2013-02-12 04:54 keltezéssel, Amit Kapila írta: > > > On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote: > > > Zoltan has reviewed this patch very thoroughly Sorry, the patch as referenced in the commitfest app is *far* from being thoroughly reviewed. Its not even remotely ready for commit. I absolutely don't want to say *anything* against Zoltans reviews or such. I have looked over the patch before as well, but I found several severe issues in a 5min look. > > > I have never seen a comment > > > from him that the current > > > Patch (one-file-all-settings) approach is not as good as having > > > one-file-per-setting approach. > > > Zoltan, please correct me, If I am wrong. > > > > I cannot recall arguing for the one-file-per-GUC way. > > But I haven't re-read my mails in this thread, either. Zoltan, mis(read|remembered) something, sorry. > Thanks for confirmation. I also don't think if ever have argument over this > approach. No idea what youre trying to say here though? > > >> Check > > >> http://www.postgresql.org/message- > > >> id/20130126162728.GA5482@awork2.anarazel.de > > >> and related messages for some of the problems. Most of which are > > >> unhandled in the current patch, i.e. currently you *will* loose > > changes > > >> made in concurrent sessions. > > > > This mail lists this order for the single file approach: > > > > > 1) exlusive lock > > > 2) reload config file (to update in memory structures) > > > 3) check new variable > > > 4) write config file (needs to be done atomically) > > > 5) optionally reload config file > > > 6) reload lock > > > > The patch does it this way: > > 1. check new variable. No problem with that, validation for proper GUC > > name, > > type of the value, etc. can be done outside the lock. > > 2. grab lock > > 3. parse the config file > > 4. write the new config file > > 5. release lock 1) You need to grab the lock before the value is checked since some variables are interdependent (e.g. log_statement_stats,wal_level, archive_mode) and thus the check needs to be made after preventing concurrent changes. 2) You need to apply the current config file for exactly the same reason before checking the new value. Also validate_conf_option/find_optiondoesn't work appropriately without an up2date config file. E.g. CURRENT does absurd thingswithout but I think there are other cases as well. I am not saying its impossible to do the one-file approach correctly, I just think its harder while not being more useful. > > Reloading the config file is intentionally not done, it's even > > documented. You can do SELECT pg_reload_conf() after SET PERSISTENT > > if you need it. Not being done and it being documented as not doing so doesn't make it correct :P I think a SET having no immediate results is confusing. Especially if I am right and we do need to apply previous config changes before doing the next SET. But I don't have *that* strong feelings about it. > > Specifically, LWLockAcquire() is called first, then ParseConfigFp() > > in a PG_TRY() block, so reading the original postgresql.auto.conf > > is serialized. No chance to lose changes done in parallel. Not a fundamental issue, but I just noticed LWLockRelease isn't called in the PG_CATCH branch in set_config_file. There's also an ereport() which needs to be moved into the PG_TRY to avoid exiting with a held lock. I think you also forgot to adjust copyfuncs.c et al for your VariableSetStmt change (addition of is_persistent). You should also check for GUC_DISALLOW_IN_FILE in validate_conf_option, its not the same as PGC_INTERNAL. What do you mean by "framing" a variable? Surrounding it by ""? It might be worth adding a check that setting and especially resetting a user-defined variable works correctly. I.e. persistently setting foo.bar = 'blub' and then resetting it again. You cannot use the name passed in via the VariableSetStmt in set_config_file, you should use the one in 'record' as returned by find_option (via validate_conf_option) as it handles the correct name. Otherwise you will get into problems with stuff like TimeZone and similar. I think validate_conf_option duplicates far too much code. You need to unify parts of set_config_option with validate_conf_option. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote: > On 2013-02-12 14:57:51 +0530, Amit Kapila wrote: > > On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote: > > > 2013-02-12 04:54 keltezéssel, Amit Kapila írta: > > > > On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote: > > > > Zoltan has reviewed this patch very thoroughly > > Sorry, the patch as referenced in the commitfest app is *far* from > being > thoroughly reviewed. Its not even remotely ready for commit. I > absolutely don't want to say *anything* against Zoltans reviews or > such. I have looked over the patch before as well, but I found several > severe issues in a 5min look. > > > > I have never seen a comment > > > > from him that the current > > > > Patch (one-file-all-settings) approach is not as good as having > > > > one-file-per-setting approach. > > > > Zoltan, please correct me, If I am wrong. > > > > > > I cannot recall arguing for the one-file-per-GUC way. > > > But I haven't re-read my mails in this thread, either. > > Zoltan, mis(read|remembered) something, sorry. > > > Thanks for confirmation. I also don't think if ever have argument > over this > > approach. > > No idea what youre trying to say here though? Forget it, this is not important. Just wanted to say, there is no confusion about approach between 2 of us. > > > >> Check > > > >> http://www.postgresql.org/message- > > > >> id/20130126162728.GA5482@awork2.anarazel.de > > > >> and related messages for some of the problems. Most of which are > > > >> unhandled in the current patch, i.e. currently you *will* loose > > > changes > > > >> made in concurrent sessions. > > > > > > This mail lists this order for the single file approach: > > > > > > > 1) exlusive lock > > > > 2) reload config file (to update in memory structures) > > > > 3) check new variable > > > > 4) write config file (needs to be done atomically) > > > > 5) optionally reload config file > > > > 6) reload lock > > > > > > The patch does it this way: > > > 1. check new variable. No problem with that, validation for proper > GUC > > > name, > > > type of the value, etc. can be done outside the lock. > > > 2. grab lock > > > 3. parse the config file > > > 4. write the new config file > > > 5. release lock > > 1) You need to grab the lock before the value is checked since some > variables are interdependent (e.g. log_statement_stats, wal_level, > archive_mode) and thus the check needs to be made after preventing > concurrent changes. This can happen if we do any SIGHUP after the command, otherwise it will have old value only. > 2) You need to apply the current config file for exactly the same > reason before checking the new value. Also > validate_conf_option/find_option doesn't work appropriately without > an up2date config file. E.g. CURRENT does absurd things without but > I > think there are other cases as well. At this moment, I am not able to think through this, could you explain by small example. I am thinking that shall we remove check hook function and do other validation only, as this will be done at time of reload, similar to what will get done when user manually edits the postgresql.conf file. > I am not saying its impossible to do the one-file approach correctly, I > just think its harder while not being more useful. > > > > Reloading the config file is intentionally not done, it's even > > > documented. You can do SELECT pg_reload_conf() after SET > PERSISTENT > > > if you need it. > > Not being done and it being documented as not doing so doesn't make it > correct :P > I think a SET having no immediate results is confusing. Especially if I > am right and we do need to apply previous config changes before doing > the next SET. But I don't have *that* strong feelings about it. I don't think any such expectation should be there, as with this feature (SET PERSISTENT), we will allow user to change the settings in file with command instead of manually editing the file. > > > Specifically, LWLockAcquire() is called first, then ParseConfigFp() > > > in a PG_TRY() block, so reading the original postgresql.auto.conf > > > is serialized. No chance to lose changes done in parallel. > > Not a fundamental issue, but I just noticed LWLockRelease isn't called > in the PG_CATCH branch in set_config_file. There's also an ereport() > which needs to be moved into the PG_TRY to avoid exiting with a held > lock. I think rollback/abort transaction due to error will handle release of locks. > I think you also forgot to adjust copyfuncs.c et al for your > VariableSetStmt change (addition of is_persistent). It is there in _copyVariableSetStmt() function. > You should also check for GUC_DISALLOW_IN_FILE in validate_conf_option, > its not the same as PGC_INTERNAL. I shall update the patch to address this issue. > What do you mean by "framing" a variable? Surrounding it by ""? Sorry, I am not able to find "framing" in quotes. > It might be worth adding a check that setting and especially resetting > a > user-defined variable works correctly. I.e. persistently setting > foo.bar > = 'blub' and then resetting it again. I shall update the patch to address this issue. > You cannot use the name passed in via the VariableSetStmt in > set_config_file, you should use the one in 'record' as returned by > find_option (via validate_conf_option) as it handles the correct > name. Otherwise you will get into problems with stuff like TimeZone and > similar. I shall check. > I think validate_conf_option duplicates far too much code. You need to > unify parts of set_config_option with validate_conf_option. I had thought of doing this initially, but found it might be little tricky as duplicate part is not one single chunk. I shall check, if it can be done in a reasonable way. Thank you for feedback. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-02-12 20:19:43 +0530, Amit Kapila wrote: > On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote: > > On 2013-02-12 14:57:51 +0530, Amit Kapila wrote: > > > On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote: > > > > This mail lists this order for the single file approach: > > > > > > > > > 1) exlusive lock > > > > > 2) reload config file (to update in memory structures) > > > > > 3) check new variable > > > > > 4) write config file (needs to be done atomically) > > > > > 5) optionally reload config file > > > > > 6) reload lock > > > > > > > > The patch does it this way: > > > > 1. check new variable. No problem with that, validation for proper > > GUC > > > > name, > > > > type of the value, etc. can be done outside the lock. > > > > 2. grab lock > > > > 3. parse the config file > > > > 4. write the new config file > > > > 5. release lock > > > > 1) You need to grab the lock before the value is checked since some > > variables are interdependent (e.g. log_statement_stats, wal_level, > > archive_mode) and thus the check needs to be made after preventing > > concurrent changes. > > This can happen if we do any SIGHUP after the command, otherwise it will > have old value only. Yes, and thats a problem imo. SET PERSISTENT log_statement_stats = true; restart; SET PERSISTENT log_statement_stats = false; SET PERSISTENT log_parser_stats = true; ERROR... thats why I think the config file needs to be processed. > > 2) You need to apply the current config file for exactly the same > > reason before checking the new value. Also > > validate_conf_option/find_option doesn't work appropriately without > > an up2date config file. E.g. CURRENT does absurd things without but > > I > > think there are other cases as well. > > At this moment, I am not able to think through this, could you explain by > small example. The most trivial one I can think of is: Transaction A: SET PERSISTENT blub = 'bar'; Transaction B: SET PERSISTENT blub FROM CURRENT; > I am thinking that shall we remove check hook function and do other > validation only, as this will be done at time > of reload, similar to what will get done when user manually edits the > postgresql.conf file. Why? The user isn't editing the file by hand for a reason. > > I am not saying its impossible to do the one-file approach correctly, I > > just think its harder while not being more useful. > > > > > > Reloading the config file is intentionally not done, it's even > > > > documented. You can do SELECT pg_reload_conf() after SET > > PERSISTENT > > > > if you need it. > > > > Not being done and it being documented as not doing so doesn't make it > > correct :P > > I think a SET having no immediate results is confusing. Especially if I > > am right and we do need to apply previous config changes before doing > > the next SET. But I don't have *that* strong feelings about it. > > I don't think any such expectation should be there, as with this feature > (SET PERSISTENT), > we will allow user to change the settings in file with command instead of > manually editing the file. I don't see why that follows. The users *do* want something different, otherwise they would hand-edit the file. > > > > Specifically, LWLockAcquire() is called first, then ParseConfigFp() > > > > in a PG_TRY() block, so reading the original postgresql.auto.conf > > > > is serialized. No chance to lose changes done in parallel. > > > > Not a fundamental issue, but I just noticed LWLockRelease isn't called > > in the PG_CATCH branch in set_config_file. There's also an ereport() > > which needs to be moved into the PG_TRY to avoid exiting with a held > > lock. > > I think rollback/abort transaction due to error will handle release of > locks. Yes, in a proper transaction abort this is done but for a utility command it might be possible to get there without a StartTransaction being done. I don't immediately see how, but I wouldn't want to rely on it, especially as doing it is simple. > > I think you also forgot to adjust copyfuncs.c et al for your > > VariableSetStmt change (addition of is_persistent). > > It is there in _copyVariableSetStmt() function. Oh, sorry, skipped over it somehow. > > What do you mean by "framing" a variable? Surrounding it by ""? > > Sorry, I am not able to find "framing" in quotes. The quotes were just there to quote the word ;). I was referring to the following comment: + /* + * The "auto.conf.d" directory should follow the postgresql.conf file + * directory not the data_directory. The same is validated while parsing + * the postgresql.conf configuration file. So while framing the + * postgresql.auto.conf and it's temp file we need to follow the + * postgresql.conf file directory. + */ > > I think validate_conf_option duplicates far too much code. You need to > > unify parts of set_config_option with validate_conf_option. > > I had thought of doing this initially, but found it might be little tricky > as duplicate part is not one single chunk. > I shall check, if it can be done in a reasonable way. I think calling validate_conf_option() from set_config_option and removing the now redundant validation from there should do the trick. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Tom Lane
Date:
Andres Freund <andres@2ndquadrant.com> writes:
> On 2013-02-12 20:19:43 +0530, Amit Kapila wrote:
>> On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote:
>>> 1) You need to grab the lock before the value is checked since some
>>> variables are interdependent (e.g. log_statement_stats, wal_level,
>>> archive_mode) and thus the check needs to be made after preventing
>>> concurrent changes.
>> This can happen if we do any SIGHUP after the command, otherwise it will
>> have old value only.
> Yes, and thats a problem imo.
That sounds to me like an entirely unreasonable requirement to put on
the patch. There is no way to positively guarantee that a value with
interdependencies will load successfully into other sessions, so why
try to enforce it at all? I note also that trying to make the value
active in the current session doesn't necessarily result in a meaningful
configuration --- what if there's an active session-level SET, for
instance? You can't just override that.
(I've said this before, but this discussion smells of overdesign every
time I look into the thread. I can't help thinking this is a big patch
with a small patch struggling to get out.)
regards, tom lane
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-02-12 10:45:28 -0500, Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On 2013-02-12 20:19:43 +0530, Amit Kapila wrote: > >> On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote: > >>> 1) You need to grab the lock before the value is checked since some > >>> variables are interdependent (e.g. log_statement_stats, wal_level, > >>> archive_mode) and thus the check needs to be made after preventing > >>> concurrent changes. > > >> This can happen if we do any SIGHUP after the command, otherwise it will > >> have old value only. > > > Yes, and thats a problem imo. > > That sounds to me like an entirely unreasonable requirement to put on > the patch. There is no way to positively guarantee that a value with > interdependencies will load successfully into other sessions, so why > try to enforce it at all? I note also that trying to make the value > active in the current session doesn't necessarily result in a meaningful > configuration --- what if there's an active session-level SET, for > instance? You can't just override that. Fair point. The reason I think its reasonable to put more effort into validation is that I am pretty sure that users will expect that its hard to get into a state where the server doesn't restart anymore. And getting 80% of that is better than 50%. And it seems required to reload the configuration anyway to make FROM CURRENT work properly. Obviously we can also just document that it throws away previously set values. There's also the point that I find it suprising behaviour that SET PERSISTENT doesn't change the active value unless a manual pg_reload_conf() is issued. If we decide that those are not important enough warts to care, yes, there is no reason for any complication here. > (I've said this before, but this discussion smells of overdesign every > time I look into the thread. I can't help thinking this is a big patch > with a small patch struggling to get out.) To be honest, I don't really think thats fair to the patch. I don't see much that can be made smaller as long as the "one file for all persistent settings" dogma is upheld which more people seem to vote for (exluding me). There's the duplication around validate_conf_option and some comment cleanup, but otherwise there really isn't that much code that looks superflous for a simple version? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Tuesday, February 12, 2013 9:02 PM Andres Freund wrote: > On 2013-02-12 20:19:43 +0530, Amit Kapila wrote: > > On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote: > > > On 2013-02-12 14:57:51 +0530, Amit Kapila wrote: > > > > On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote: > > > > > This mail lists this order for the single file approach: > > > > > > > > > > > 1) exlusive lock > > > > > > 2) reload config file (to update in memory structures) > > > > > > 3) check new variable > > > > > > 4) write config file (needs to be done atomically) > > > > > > 5) optionally reload config file > > > > > > 6) reload lock > > > > > > > > > > The patch does it this way: > > > > > 1. check new variable. No problem with that, validation for > proper > > > GUC > > > > > name, > > > > > type of the value, etc. can be done outside the lock. > > > > > 2. grab lock > > > > > 3. parse the config file > > > > > 4. write the new config file > > > > > 5. release lock > > > > > > 1) You need to grab the lock before the value is checked since some > > > variables are interdependent (e.g. log_statement_stats, > wal_level, > > > archive_mode) and thus the check needs to be made after > preventing > > > concurrent changes. > > > > This can happen if we do any SIGHUP after the command, otherwise it > will > > have old value only. > > Yes, and thats a problem imo. > > SET PERSISTENT log_statement_stats = true; > restart; > SET PERSISTENT log_statement_stats = false; > SET PERSISTENT log_parser_stats = true; > ERROR... > > thats why I think the config file needs to be processed. > > > > 2) You need to apply the current config file for exactly the same > > > reason before checking the new value. Also > > > validate_conf_option/find_option doesn't work appropriately > without > > > an up2date config file. E.g. CURRENT does absurd things without > but > > > I > > > think there are other cases as well. > > > > At this moment, I am not able to think through this, could you > explain by > > small example. > > The most trivial one I can think of is: > > Transaction A: SET PERSISTENT blub = 'bar'; > Transaction B: SET PERSISTENT blub FROM CURRENT; > > > I am thinking that shall we remove check hook function and do other > > validation only, as this will be done at time > > of reload, similar to what will get done when user manually edits the > > postgresql.conf file. > > Why? The user isn't editing the file by hand for a reason. > > > > I am not saying its impossible to do the one-file approach > correctly, I > > > just think its harder while not being more useful. > > > > > > > > Reloading the config file is intentionally not done, it's even > > > > > documented. You can do SELECT pg_reload_conf() after SET > > > PERSISTENT > > > > > if you need it. > > > > > > Not being done and it being documented as not doing so doesn't make > it > > > correct :P > > > I think a SET having no immediate results is confusing. Especially > if I > > > am right and we do need to apply previous config changes before > doing > > > the next SET. But I don't have *that* strong feelings about it. > > > > I don't think any such expectation should be there, as with this > feature > > (SET PERSISTENT), > > we will allow user to change the settings in file with command > instead of > > manually editing the file. > > I don't see why that follows. The users *do* want something different, > otherwise they would hand-edit the file. > > > > > > Specifically, LWLockAcquire() is called first, then > ParseConfigFp() > > > > > in a PG_TRY() block, so reading the original > postgresql.auto.conf > > > > > is serialized. No chance to lose changes done in parallel. > > > > > > Not a fundamental issue, but I just noticed LWLockRelease isn't > called > > > in the PG_CATCH branch in set_config_file. There's also an > ereport() > > > which needs to be moved into the PG_TRY to avoid exiting with a > held > > > lock. > > > > I think rollback/abort transaction due to error will handle release > of > > locks. > > Yes, in a proper transaction abort this is done but for a utility > command it might be possible to get there without a StartTransaction > being done. I don't immediately see how, but I wouldn't want to rely on > it, especially as doing it is simple. True, doing it is simple but without reason can raise questions again like why is it released when it will be done as part of abort. I will think more on it to see if there can be any way it can reach without StartTransaction. > > > I think you also forgot to adjust copyfuncs.c et al for your > > > VariableSetStmt change (addition of is_persistent). > > > > It is there in _copyVariableSetStmt() function. > > Oh, sorry, skipped over it somehow. > > > > What do you mean by "framing" a variable? Surrounding it by ""? > > > > Sorry, I am not able to find "framing" in quotes. > > The quotes were just there to quote the word ;). I was referring to the > following comment: > > + /* > + * The "auto.conf.d" directory should follow the postgresql.conf > file > + * directory not the data_directory. The same is validated while > parsing > + * the postgresql.conf configuration file. So while framing the > + * postgresql.auto.conf and it's temp file we need to follow the > + * postgresql.conf file directory. > + */ Shall we change it as one of below: a. "framing the path for postgresql.auto.conf ... " b. "framing the name for postgresql.auto.conf ... " c. "generating the path for postgresql.auto.conf ..." d. any other suggestion? > > > I think validate_conf_option duplicates far too much code. You need > to > > > unify parts of set_config_option with validate_conf_option. > > > > I had thought of doing this initially, but found it might be little > tricky > > as duplicate part is not one single chunk. > > I shall check, if it can be done in a reasonable way. > > I think calling validate_conf_option() from set_config_option and > removing the now redundant validation from there should do the trick. Okay. I have intentionally not replied some parts of message as I want to wait if there can be conclusion for the points you have mentioned in another mail. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Josh Berkus
Date:
> To be honest, I don't really think thats fair to the patch. I don't see > much that can be made smaller as long as the "one file for all > persistent settings" dogma is upheld which more people seem to vote for > (exluding me). I'm not a fan of "one file to rule them all" either. Hovever, I'm strongly in favor of anything that can get into 9.3. So if we can make a 9.3 patch out of this by doing "file-per-setting", then I'm all for it. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Tuesday, February 12, 2013 9:02 PM Andres Freund wrote:
> On 2013-02-12 20:19:43 +0530, Amit Kapila wrote:
> > On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote:
> > > On 2013-02-12 14:57:51 +0530, Amit Kapila wrote:
> > > > On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote:
> > > > > This mail lists this order for the single file approach:
> > > > >
> > > > > > 1) exlusive lock
> > > > > > 2) reload config file (to update in memory structures)
> > > > > > 3) check new variable
> > > > > > 4) write config file (needs to be done atomically)
> > > > > > 5) optionally reload config file
> > > > > > 6) reload lock
> > > > >
> > > > > The patch does it this way:
> > > > > 1. check new variable. No problem with that, validation for
> proper
> > > GUC
> > > > > name,
> > > > > type of the value, etc. can be done outside the lock.
> > > > > 2. grab lock
> > > > > 3. parse the config file
> > > > > 4. write the new config file
> > > > > 5. release lock
> > >
> > > 1) You need to grab the lock before the value is checked since some
> > > variables are interdependent (e.g. log_statement_stats,
> wal_level,
> > > archive_mode) and thus the check needs to be made after
> preventing
> > > concurrent changes.
> >
> > This can happen if we do any SIGHUP after the command, otherwise it
> will
> > have old value only.
>
> Yes, and thats a problem imo.
>
> SET PERSISTENT log_statement_stats = true;
> restart;
> SET PERSISTENT log_statement_stats = false;
> SET PERSISTENT log_parser_stats = true;
> ERROR...
>
> thats why I think the config file needs to be processed.
>
> > > 2) You need to apply the current config file for exactly the same
> > > reason before checking the new value. Also
> > > validate_conf_option/find_option doesn't work appropriately
> without
> > > an up2date config file. E.g. CURRENT does absurd things without
> but
> > > I
> > > think there are other cases as well.
> >
> > At this moment, I am not able to think through this, could you
> explain by
> > small example.
>
> The most trivial one I can think of is:
>
> Transaction A: SET PERSISTENT blub = 'bar';
> Transaction B: SET PERSISTENT blub FROM CURRENT;
>
> > I am thinking that shall we remove check hook function and do other
> > validation only, as this will be done at time
> > of reload, similar to what will get done when user manually edits the
> > postgresql.conf file.
>
> Why? The user isn't editing the file by hand for a reason.
>
> > > I am not saying its impossible to do the one-file approach
> correctly, I
> > > just think its harder while not being more useful.
> > >
> > > > > Reloading the config file is intentionally not done, it's even
> > > > > documented. You can do SELECT pg_reload_conf() after SET
> > > PERSISTENT
> > > > > if you need it.
> > >
> > > Not being done and it being documented as not doing so doesn't make
> it
> > > correct :P
> > > I think a SET having no immediate results is confusing. Especially
> if I
> > > am right and we do need to apply previous config changes before
> doing
> > > the next SET. But I don't have *that* strong feelings about it.
> >
> > I don't think any such expectation should be there, as with this
> feature
> > (SET PERSISTENT),
> > we will allow user to change the settings in file with command
> instead of
> > manually editing the file.
>
> I don't see why that follows. The users *do* want something different,
> otherwise they would hand-edit the file.
>
> > > > > Specifically, LWLockAcquire() is called first, then
> ParseConfigFp()
> > > > > in a PG_TRY() block, so reading the original
> postgresql.auto.conf
> > > > > is serialized. No chance to lose changes done in parallel.
> > >
> > > Not a fundamental issue, but I just noticed LWLockRelease isn't
> called
> > > in the PG_CATCH branch in set_config_file. There's also an
> ereport()
> > > which needs to be moved into the PG_TRY to avoid exiting with a
> held
> > > lock.
> >
> > I think rollback/abort transaction due to error will handle release
> of
> > locks.
>
> Yes, in a proper transaction abort this is done but for a utility
> command it might be possible to get there without a StartTransaction
> being done. I don't immediately see how, but I wouldn't want to rely on
> it, especially as doing it is simple.
We cannot directly call LWLockRelease in catch block as it will lead to
Assert failure.
The reason is errfinish will set InterruptHoldOffCount to 0, and now in
LWLockRelease,
when it will call RESUME_INTERRUPTS it will check InterruptHoldOffCount
which leads to Assertion failure.
To handle it, we need to call HOLD_INTERRUPTS before it similar to
LWLockReleaseAll().
So I am not sure if we are not sure of scenario, we should add such calls in
Catch block.
> > > I think you also forgot to adjust copyfuncs.c et al for your
> > > VariableSetStmt change (addition of is_persistent).
> >
> > It is there in _copyVariableSetStmt() function.
>
> Oh, sorry, skipped over it somehow.
>
> > > What do you mean by "framing" a variable? Surrounding it by ""?
> >
> > Sorry, I am not able to find "framing" in quotes.
>
> The quotes were just there to quote the word ;). I was referring to the
> following comment:
>
> + /*
> + * The "auto.conf.d" directory should follow the postgresql.conf
> file
> + * directory not the data_directory. The same is validated while
> parsing
> + * the postgresql.conf configuration file. So while framing the
> + * postgresql.auto.conf and it's temp file we need to follow the
> + * postgresql.conf file directory.
> + */
>
> > > I think validate_conf_option duplicates far too much code. You need
> to
> > > unify parts of set_config_option with validate_conf_option.
> >
> > I had thought of doing this initially, but found it might be little
> tricky
> > as duplicate part is not one single chunk.
> > I shall check, if it can be done in a reasonable way.
>
> I think calling validate_conf_option() from set_config_option and
> removing the now redundant validation from there should do the trick.
I have tried it, it will not be straight forward, please see some problems
which we need to handle:
1. few checks like GUC_DISALLOW_IN_FILE will be different for persistent
function - we can address this by passing ispersistent flag
2. Cannot call validate_config_option directly, because the below part of
code is tightly coupled. a. if (value) .. else if() .. else
Now if the value part is already checked in validate_config_option,
adjusting this loop will be tricky. b. some of the variable like newextra are required outside function, so
extra parameter needs to be passed. c. memory free issues, as newextra will be required in function
set_config_option, however set_config_file doesn't need it, so it would be better to free that memory inside
validate_config_option
3. Extra parameter elevel would be required in validate_conf_option.
4. Function set_config_option is used from many places, so impact can be
higher if we do too many adjustments.
Considering all above, I think it might not be advisable to remove common
code of validate_conf_option
> It might be worth adding a check that setting and especially resetting a
user-defined variable works correctly. I.e.
> persistently setting foo.bar = 'blub' and then resetting it again.
What exactly you want in this handling?
With Regards,
Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote:
> On 2013-02-12 14:57:51 +0530, Amit Kapila wrote:
> > On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote:
> > > 2013-02-12 04:54 keltezéssel, Amit Kapila írta:
> > > > On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote:
> > > > Zoltan has reviewed this patch very thoroughly
>
> > >
> > > This mail lists this order for the single file approach:
> > >
> > > > 1) exlusive lock
> > > > 2) reload config file (to update in memory structures)
> > > > 3) check new variable
> > > > 4) write config file (needs to be done atomically)
> > > > 5) optionally reload config file
> > > > 6) reload lock
> > >
> > > The patch does it this way:
> > > 1. check new variable. No problem with that, validation for proper
> GUC
> > > name,
> > > type of the value, etc. can be done outside the lock.
> > > 2. grab lock
> > > 3. parse the config file
> > > 4. write the new config file
> > > 5. release lock
>
> 1) You need to grab the lock before the value is checked since some
> variables are interdependent (e.g. log_statement_stats, wal_level,
> archive_mode) and thus the check needs to be made after preventing
> concurrent changes.
> SET PERSISTENT log_statement_stats = true; restart; SET PERSISTENT
log_statement_stats = false; SET PERSISTENT log_parser_stats = true; ERROR
As we are not loading configuration at end of each command, so this will be
a valid case.
> 2) You need to apply the current config file for exactly the same
> reason before checking the new value. Also
> validate_conf_option/find_option doesn't work appropriately without
> an up2date config file. E.g. CURRENT does absurd things without but
> I
> think there are other cases as well.
> Transaction A: SET PERSISTENT blub = 'bar'; Transaction B: SET PERSISTENT
blub FROM CURRENT;
This behavior is not only for CURRENT, but any other setting will have same
behavior.
It just retains the last set value for any parameter before restart.
> I am not saying its impossible to do the one-file approach correctly, I
> just think its harder while not being more useful.
IMHO, let's try to see if there are any fundamental problems with this
Approach, then it
will make much more sense to change the Approach.
> > > Reloading the config file is intentionally not done, it's even
> > > documented. You can do SELECT pg_reload_conf() after SET
> PERSISTENT
> > > if you need it.
>
> Not being done and it being documented as not doing so doesn't make it
> correct :P
> I think a SET having no immediate results is confusing. Especially if I
> am right and we do need to apply previous config changes before doing
> the next SET. But I don't have *that* strong feelings about it.
>
> > > Specifically, LWLockAcquire() is called first, then ParseConfigFp()
> > > in a PG_TRY() block, so reading the original postgresql.auto.conf
> > > is serialized. No chance to lose changes done in parallel.
>
> Not a fundamental issue, but I just noticed LWLockRelease isn't called
> in the PG_CATCH branch in set_config_file. There's also an ereport()
> which needs to be moved into the PG_TRY to avoid exiting with a held
> lock.
We cannot directly call LWLockRelease in catch block as it will lead to
Assert failure.
The reason is errfinish will set InterruptHoldOffCount to 0, and now in
LWLockRelease, when it will call RESUME_INTERRUPTS it will check
InterruptHoldOffCount which leads to Assertion failure.
To handle it, we need to call HOLD_INTERRUPTS before it similar to
LWLockReleaseAll().
So I am not sure if we should add such calls in Catch block.
If you are very sure that such scenario exist where it can reach without
transaction then
we can handle it in above way.
> I think you also forgot to adjust copyfuncs.c et al for your
> VariableSetStmt change (addition of is_persistent).
> You should also check for GUC_DISALLOW_IN_FILE in validate_conf_option,
> its not the same as PGC_INTERNAL.
Fixed.
> What do you mean by "framing" a variable? Surrounding it by ""?
Changed the line to "faraming the name for".
Here framing means constructing the variable with different inputs.
> It might be worth adding a check that setting and especially resetting
> a
> user-defined variable works correctly. I.e. persistently setting
> foo.bar
> = 'blub' and then resetting it again.
Can you please elaborate a little bit more on this as I am not able to
clearly
Understand.
> You cannot use the name passed in via the VariableSetStmt in
> set_config_file, you should use the one in 'record' as returned by
> find_option (via validate_conf_option) as it handles the correct
> name. Otherwise you will get into problems with stuff like TimeZone and
> similar.
The name passed in via the VariableSetStmt always is lowercase, which will
be same
same as per current postgresql.conf.
Doing it is very simple, but I am not able to understand what is the exact
problem you are
seeing in the current way.
> I think validate_conf_option duplicates far too much code. You need to
> unify parts of set_config_option with validate_conf_option.
I have changed few things in validate_conf_option:
a. comments on top of function
b. memory was not getting freed for one variable.
I have tried to unify, it doesn't seem to be straight forward, please see
some problems which we need to handle:
1. few checks like GUC_DISALLOW_IN_FILE will be different for persistent
function - we can address this by passing ispersistent flag 2. Cannot call
validate_config_option directly, because the below part of code is tightly coupled. a. if (value) .. else
if() .. else
Now if the value part is already checked in validate_config_option,
adjusting this loop will be tricky. b. some of the variable like newextra are required outside function, so
extra parameter needs to be passed. c. memory free issues, as newextra will be required in function
set_config_option, however set_config_file doesn't need it, so it would be better to free that memory inside
validate_config_option
3. Extra parameter elevel would be required in validate_conf_option.
4. Function set_config_option is used from many places, so impact can be
higher if we do too many adjustments.
Considering all above, I think it might not be advisable to remove common
code of validate_conf_option.
With Regards,
Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Greg Smith
Date:
This submission didn't have any listed reviewers in the CF app. I added Zoltan and Andres since both of you went through the usual review steps and have given lots of feedback. There are two main sets of issues that keep popping up with this feature: -The implementation seems too long, at around 2189 lines of diff. I have a few ideas for how things might be trimmed below. I do appreciate that a good part of the length is a long set of regression tests, relative to what a feature like this normally gets. -It might be possible to get a simpler implementation with a file per setting. I can make a pass over cleaning up the wording in your comments and documentation. There are enough coding issues that I think that should wait until another rev of the patch first. = Directory vs. single file = The main reason I've advocated a configuration file directory is to try and suggest a standard for tool generated config files to co-exist in. This particular feature doesn't *need* that. But in the long term I was hoping to have more tools that can write out .conf files without having to read and understand every existing file that's in there. It doesn't make that big of a difference whether this particular one lives in $PGDATA/postgresql.auto.conf or $PGDATA/postgresql.auto.conf. The main reason for the directory is to make the second such tool not further clutter the main $PGDATA directory. I would like to see the name of the directory be conf.d instead of auto.conf.d though. What's "auto" about it? Using that word just adds a source of confusion. The same problem exists with the file name itself. I was hoping for conf.d/persistent.conf here, and no use of the word "auto" in the code itself. What I don't see a lot of value in is the config file per setting idea. I was hoping SET PERSISTENT put all of its changesinto once place. It should be obvious where they came from, and that people can't edit that file manually without breaking the feature. To me something like persistent.conf gives that impression, while postgresql.auto.conf sounds like the output from some auto-tuning tool. = Length reduction = I'm not sure why you are opening the old auto config file with ParseConfigFp. Can't you just navigate the existing GUCs in memory and directly write the new one out? If someone is going to manually edit this file and use SET PERSISTENT, they're going to end up in trouble regardless. I don't think it's really worth the extra complexity needed to try and handle that case. It sounds like you've thought a good bit about how to eliminate all the code in the validate_conf_option function. I think that idea may take some more thinking to try and be creative about it. Have you looked at whether there's any way to refactor this to be smaller by modifying guc.c to break out parts of its implementation needed here? Is it possible to modify your changes to src/backend/parser/gram.y to produce a smaller diff? There looks to be a good amount of code (~25 lines) that aren't really changed that I can see. I suspect they might have been hit with an editor whitespace change. = General code issues = There is some trivial bit rot on your v10 here: 1 out of 8 hunks FAILED -- saving rejects to file src/backend/parser/gram.y.rej 1 out of 3 hunks FAILED -- saving rejects to file src/bin/initdb/initdb.c.rej Your change to gram.y was broken by PROGRAM being added. I'm not sure why the initdb.c one doesn't apply; whitespace issue maybe? As written right now, even with fixing those two, I'm getting this compiler error: gram.y:12694.27-36: warning: type clash on default action: <keyword> != <> gram.y:1306.31-40: symbol PERSISTENT is used, but is not defined as a token and has no rules This combination in src/test/regress/sql/persistent.sql a few times makes me nervous: + select pg_reload_conf(); + select pg_sleep(1); Why the pause? There are some regression tests with sleeps in them, such as the stats collector and the timezone tests. I would expect pg_reload_conf would not return until its configuration is stable, so this is a bit of a surprise. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Josh Berkus
Date:
> = Directory vs. single file = > > The main reason I've advocated a configuration file directory is to try > and suggest a standard for tool generated config files to co-exist in. > This particular feature doesn't *need* that. But in the long term I was > hoping to have more tools that can write out .conf files without having > to read and understand every existing file that's in there. It doesn't > make that big of a difference whether this particular one lives in > $PGDATA/postgresql.auto.conf or $PGDATA/postgresql.auto.conf. The main > reason for the directory is to make the second such tool not further > clutter the main $PGDATA directory. > > I would like to see the name of the directory be conf.d instead of > auto.conf.d though. What's "auto" about it? Using that word just adds > a source of confusion. The same problem exists with the file name > itself. I was hoping for conf.d/persistent.conf here, and no use of the > word "auto" in the code itself. my $0.03: I agree with Greg that using the directory is a good idea, and that his naming is an improvement. > What I don't see a lot of value in is the config file per setting idea. > I was hoping SET PERSISTENT put all of its changes into once place. It > should be obvious where they came from, and that people can't edit that > file manually without breaking the feature. To me something like > persistent.conf gives that impression, while postgresql.auto.conf sounds > like the output from some auto-tuning tool. I think most users would prefer a single file. The reason file-per-setting is being discussed is that it avoids a lot of locking issues and code complexity. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Tom Lane
Date:
Josh Berkus <josh@agliodbs.com> writes:
>> I would like to see the name of the directory be conf.d instead of
>> auto.conf.d though. What's "auto" about it? Using that word just adds
>> a source of confusion. The same problem exists with the file name
>> itself. I was hoping for conf.d/persistent.conf here, and no use of the
>> word "auto" in the code itself.
> my $0.03: I agree with Greg that using the directory is a good idea, and
> that his naming is an improvement.
Neither of those names is consistent with any other PGDATA subdirectory
name we use. It should just be config, or perhaps pg_config, though the
latter risks confusion with the tool of the same name.
FWIW, I do think that having "auto" or some such in the file name(s)
would be a good idea, to help warn people off editing them manually.
regards, tom lane
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Greg Smith
Date:
On 3/3/13 8:34 PM, Tom Lane wrote: > Neither of those names is consistent with any other PGDATA subdirectory > name we use. It should just be config, or perhaps pg_config, though the > latter risks confusion with the tool of the same name. I'd be just as happy with config/ as the directory name. I have a bias toward wanting this to look like Apache that I've been scolded for before, I forgot to avoid that this time. > FWIW, I do think that having "auto" or some such in the file name(s) > would be a good idea, to help warn people off editing them manually. I can see that for the file name itself, as long as it's not in the directory name. I don't like giving the idea that everything in there is automatically generated. config/persistent-auto.conf or config/persist-auto.conf maybe? The way files are sorted by name means that using config/auto-persist[ent].conf would make the file more likely to be processed before other things in that directory. I could live with that order too. I think it is important to name the file such that it suggests being connected to the SET PERSISTENT feature though. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Craig Ringer
Date:
On 03/04/2013 09:34 AM, Tom Lane wrote: > Josh Berkus <josh@agliodbs.com> writes: >>> I would like to see the name of the directory be conf.d instead of >>> auto.conf.d though. What's "auto" about it? Using that word just adds >>> a source of confusion. The same problem exists with the file name >>> itself. I was hoping for conf.d/persistent.conf here, and no use of the >>> word "auto" in the code itself. >> my $0.03: I agree with Greg that using the directory is a good idea, and >> that his naming is an improvement. > Neither of those names is consistent with any other PGDATA subdirectory > name we use. It should just be config, or perhaps pg_config, though the > latter risks confusion with the tool of the same name. > > FWIW, I do think that having "auto" or some such in the file name(s) > would be a good idea, to help warn people off editing them manually. Yep; auto in the dir name seems unwise given that we want to allow tools to drop config snippets in there, but auto in the file name would IMO help people realise it shouldn't be edited directly. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Monday, March 04, 2013 7:26 AM Greg Smith wrote: > On 3/3/13 8:34 PM, Tom Lane wrote: > > Neither of those names is consistent with any other PGDATA > subdirectory > > name we use. It should just be config, or perhaps pg_config, though > the > > latter risks confusion with the tool of the same name. > > I'd be just as happy with config/ as the directory name. I have a bias > toward wanting this to look like Apache that I've been scolded for > before, I forgot to avoid that this time. > > > FWIW, I do think that having "auto" or some such in the file name(s) > > would be a good idea, to help warn people off editing them manually. > > I can see that for the file name itself, as long as it's not in the > directory name. I don't like giving the idea that everything in there > is automatically generated. config/persistent-auto.conf or > config/persist-auto.conf maybe? > > The way files are sorted by name means that using > config/auto-persist[ent].conf would make the file more likely to be > processed before other things in that directory. I could live with > that > order too. I think it is important to name the file such that it > suggests being connected to the SET PERSISTENT feature though. If we want to link it with SET PERSISTENT feature, we can choose name as you suggest or config/auto.persistent.conf? If we see the current files postgresql.conf, recovery.conf etc, it seems most are related to the functionality it provides. For example recovery.conf suggests that it contains settings for doing recovery. The current name postgresql.auto.conf has been chosen to make it's connection with user editable postgresql.conf. I thought it will give impression to user that this contains settings similar to postgresql.conf but this file is auto generated, so it can't be editable by user. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Monday, March 04, 2013 9:41 AM Craig Ringer wrote: > On 03/04/2013 09:07 AM, Greg Smith wrote: > > I'm not sure why you are opening the old auto config file with > > ParseConfigFp. Can't you just navigate the existing GUCs in memory > > and directly write the new one out? If someone is going to manually > > edit this file and use SET PERSISTENT, they're going to end up in > > trouble regardless. I don't think it's really worth the extra > > complexity needed to try and handle that case. > Additionally, if you want to avoid silently overwriting user changes, > you could store a timestamp for when we last updated the persistent > config and compare it to the on-disk timestamp before writing. If they > don't match a warning would be issued and the config would be > overwritten anyway. There's a race, of course, but since the worst case > is that we fail to issue a warning it's a pretty harmless one. > As for the per-file vs single-file issue and concerns about locking > complexity: Can't we just use a global lock in shm to enforce that > exactly one backend at a time may be modifying the global > configuration? Yes, this is currently used. > I don't see this ever becoming a realistic concern for concurrency and > performance, and the shm cost would be tiny. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Monday, March 04, 2013 6:38 AM Greg Smith wrote: > This submission didn't have any listed reviewers in the CF app. I > added > Zoltan and Andres since both of you went through the usual review steps > and have given lots of feedback. Thank you for review. > There are two main sets of issues that keep popping up with this > feature: > > -The implementation seems too long, at around 2189 lines of diff. I > have a few ideas for how things might be trimmed below. I do > appreciate > that a good part of the length is a long set of regression tests, > relative to what a feature like this normally gets. > > -It might be possible to get a simpler implementation with a file per > setting. > > I can make a pass over cleaning up the wording in your comments and > documentation. There are enough coding issues that I think that should > wait until another rev of the patch first. I have tried to completely handle all the comments, but need 1 more day. > = Directory vs. single file = > > The main reason I've advocated a configuration file directory is to try > and suggest a standard for tool generated config files to co-exist in. > This particular feature doesn't *need* that. But in the long term I > was > hoping to have more tools that can write out .conf files without having > to read and understand every existing file that's in there. It doesn't > make that big of a difference whether this particular one lives in > $PGDATA/postgresql.auto.conf or $PGDATA/postgresql.auto.conf. The main > reason for the directory is to make the second such tool not further > clutter the main $PGDATA directory. > > I would like to see the name of the directory be conf.d instead of > auto.conf.d though. What's "auto" about it? Using that word just adds > a source of confusion. The same problem exists with the file name > itself. I was hoping for conf.d/persistent.conf here, and no use of > the > word "auto" in the code itself. We can name the directory as config. For file, I am awaiting your reponse to select from below options a. persistent.auto.conf b. persist.auto.conf c. auto.persist.conf d. postgresql.auto.conf > What I don't see a lot of value in is the config file per setting idea. > I was hoping SET PERSISTENT put all of its changes into once place. > It should be obvious where they came from, and that people can't edit > that file manually without breaking the feature. To me something like > persistent.conf gives that impression, while postgresql.auto.conf > sounds > like the output from some auto-tuning tool. > > = Length reduction = > > I'm not sure why you are opening the old auto config file with > ParseConfigFp. Can't you just navigate the existing GUCs in memory and > directly write the new one out? If someone is going to manually edit > this file and use SET PERSISTENT, they're going to end up in trouble > regardless. I don't think it's really worth the extra complexity > needed > to try and handle that case. This is for logic to see if variable already exist just replace it's value at the same location in .auto file, else append at end. IIUC what you are suggesting is set first in memory then while writing just check the memory, whatever Is new value just write it. if we just traverse in memory how will we know where this exist in file. This might have complication what if write to memory is success, but to file it fails. > It sounds like you've thought a good bit about how to eliminate all the > code in the validate_conf_option function. I think that idea may take > some more thinking to try and be creative about it. Have you looked at > whether there's any way to refactor this to be smaller by modifying > guc.c to break out parts of its implementation needed here? Today I tried again by having single common function for both places, but It is getting clumsy. So I can to write multiple functions for each datatype? > Is it possible to modify your changes to src/backend/parser/gram.y to > produce a smaller diff? There looks to be a good amount of code (~25 > lines) that aren't really changed that I can see. I suspect they might > have been hit with an editor whitespace change. I will do pgindent. Can you point what are extra lines that can be removed. > = General code issues = > > There is some trivial bit rot on your v10 here: > > 1 out of 8 hunks FAILED -- saving rejects to file > src/backend/parser/gram.y.rej > 1 out of 3 hunks FAILED -- saving rejects to file > src/bin/initdb/initdb.c.rej > > Your change to gram.y was broken by PROGRAM being added. I'm not sure > why the initdb.c one doesn't apply; whitespace issue maybe? As written > right now, even with fixing those two, I'm getting this compiler error: > > gram.y:12694.27-36: warning: type clash on default action: <keyword> != > <> > gram.y:1306.31-40: symbol PERSISTENT is used, but is not defined as a > token and has no rules I shall take care of these, minor code related issues due to recent changes in HEAD. > This combination in src/test/regress/sql/persistent.sql a few times > makes me nervous: > > + select pg_reload_conf(); > + select pg_sleep(1); > > Why the pause? There are some regression tests with sleeps in them, > such as the stats collector and the timezone tests. I would expect > pg_reload_conf would not return until its configuration is stable, so > this is a bit of a surprise. pg_reload_conf(), doesn't ensure that configuration has been propagated to memory. It just sends the SIGHUP signal to postmaster. If I remove, it will fail the tests. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Craig Ringer
Date:
On 03/04/2013 09:07 AM, Greg Smith wrote: > I'm not sure why you are opening the old auto config file with > ParseConfigFp. Can't you just navigate the existing GUCs in memory > and directly write the new one out? If someone is going to manually > edit this file and use SET PERSISTENT, they're going to end up in > trouble regardless. I don't think it's really worth the extra > complexity needed to try and handle that case. Additionally, if you want to avoid silently overwriting user changes, you could store a timestamp for when we last updated the persistent config and compare it to the on-disk timestamp before writing. If they don't match a warning would be issued and the config would be overwritten anyway. There's a race, of course, but since the worst case is that we fail to issue a warning it's a pretty harmless one. As for the per-file vs single-file issue and concerns about locking complexity: Can't we just use a global lock in shm to enforce that exactly one backend at a time may be modifying the global configuration? I don't see this ever becoming a realistic concern for concurrency and performance, and the shm cost would be tiny. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Monday, March 04, 2013 8:46 PM Amit Kapila wrote: > On Monday, March 04, 2013 6:38 AM Greg Smith wrote: > > This submission didn't have any listed reviewers in the CF app. I > > added > > Zoltan and Andres since both of you went through the usual review > steps > > and have given lots of feedback. > > Thank you for review. Fujii Masao has also done the review and test of patch multiple times. > > There are two main sets of issues that keep popping up with this > > feature: > > > > -The implementation seems too long, at around 2189 lines of diff. I > > have a few ideas for how things might be trimmed below. I do > > appreciate > > that a good part of the length is a long set of regression tests, > > relative to what a feature like this normally gets. > > > > -It might be possible to get a simpler implementation with a file per > > setting. > > > > I can make a pass over cleaning up the wording in your comments and > > documentation. There are enough coding issues that I think that > should > > wait until another rev of the patch first. > > I have tried to completely handle all the comments, but need 1 more > day. > > > = Directory vs. single file = > > > > The main reason I've advocated a configuration file directory is to > try > > and suggest a standard for tool generated config files to co-exist > in. > > This particular feature doesn't *need* that. But in the long term I > > was > > hoping to have more tools that can write out .conf files without > having > > to read and understand every existing file that's in there. It > doesn't > > make that big of a difference whether this particular one lives in > > $PGDATA/postgresql.auto.conf or $PGDATA/postgresql.auto.conf. The > main > > reason for the directory is to make the second such tool not further > > clutter the main $PGDATA directory. > > > > I would like to see the name of the directory be conf.d instead of > > auto.conf.d though. What's "auto" about it? Using that word just > adds > > a source of confusion. The same problem exists with the file name > > itself. I was hoping for conf.d/persistent.conf here, and no use of > > the > > word "auto" in the code itself. > > We can name the directory as config. > For file, I am awaiting your reponse to select from below options > a. persistent.auto.conf > b. persist.auto.conf > c. auto.persist.conf > d. postgresql.auto.conf In v11 patch, I have changed name of directory to config. For file name, currently I have not changed, but if you feel it needs to be changed, kindly suggest any one of above or if any other better you have in mind. > > What I don't see a lot of value in is the config file per setting > idea. > > I was hoping SET PERSISTENT put all of its changes into once place. > > It should be obvious where they came from, and that people can't edit > > that file manually without breaking the feature. To me something > like > > persistent.conf gives that impression, while postgresql.auto.conf > > sounds > > like the output from some auto-tuning tool. > > > > = Length reduction = > > > > I'm not sure why you are opening the old auto config file with > > ParseConfigFp. Can't you just navigate the existing GUCs in memory > and > > directly write the new one out? If someone is going to manually edit > > this file and use SET PERSISTENT, they're going to end up in trouble > > regardless. I don't think it's really worth the extra complexity > > needed > > to try and handle that case. > > This is for logic to see if variable already exist just replace it's > value > at the same > location in .auto file, else append at end. > IIUC what you are suggesting is set first in memory then while writing > just > check the memory, whatever > Is new value just write it. > if we just traverse in memory how will we know where this exist in > file. > This might have complication what if write to memory is success, but to > file > it fails. > > > It sounds like you've thought a good bit about how to eliminate all > the > > code in the validate_conf_option function. I think that idea may > take > > some more thinking to try and be creative about it. Have you looked > at > > whether there's any way to refactor this to be smaller by modifying > > guc.c to break out parts of its implementation needed here? > > Today I tried again by having single common function for both places, > but > It is getting clumsy. So I can to write multiple functions for each > datatype? Finally, I could come up with unified function with common parts together based on your suggestions. > > Is it possible to modify your changes to src/backend/parser/gram.y to > > produce a smaller diff? There looks to be a good amount of code (~25 > > lines) that aren't really changed that I can see. I suspect they > might > > have been hit with an editor whitespace change. > > I will do pgindent. > Can you point what are extra lines that can be removed. I think it is mainly due to the changes done in gram.y. > > = General code issues = > > > > There is some trivial bit rot on your v10 here: > > > > 1 out of 8 hunks FAILED -- saving rejects to file > > src/backend/parser/gram.y.rej > > 1 out of 3 hunks FAILED -- saving rejects to file > > src/bin/initdb/initdb.c.rej > > > > Your change to gram.y was broken by PROGRAM being added. I'm not > sure > > why the initdb.c one doesn't apply; whitespace issue maybe? As > written > > right now, even with fixing those two, I'm getting this compiler > error: > > > > gram.y:12694.27-36: warning: type clash on default action: <keyword> > != > > <> > > gram.y:1306.31-40: symbol PERSISTENT is used, but is not defined as a > > token and has no rules > > I shall take care of these, minor code related issues due to recent > changes > in HEAD. Gram.y - PERSISTENT word is missed from one of the list Initdb.c - structure subdirs, doesn't contain new directory. Fixed both of the above. > > This combination in src/test/regress/sql/persistent.sql a few times > > makes me nervous: > > > > + select pg_reload_conf(); > > + select pg_sleep(1); > > > > Why the pause? There are some regression tests with sleeps in them, > > such as the stats collector and the timezone tests. I would expect > > pg_reload_conf would not return until its configuration is stable, so > > this is a bit of a surprise. > > pg_reload_conf(), doesn't ensure that configuration has been propagated > to > memory. > It just sends the SIGHUP signal to postmaster. > If I remove, it will fail the tests. I have kept this intact. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Greg Smith
Date:
On 3/5/13 9:07 AM, Amit Kapila wrote: > In v11 patch, I have changed name of directory to config. > For file name, currently I have not changed, but if you feel it needs to be > changed, kindly suggest any one of above or if any other better you have in > mind. This seems fine for now. Hashing out exactly which name is best is not going to be the thing that determines whether this can be committed or not. Your v11 applies cleanly for me too, and the code does look a lot nicer. The code diff itself is down to 1300 lines andnot nearly as disruptive, the rest is docs or the many regression tests you've added. That's still not the sort of small change Tom washoping this was possible, but it's closer to the right range. It may not really be possible to make this much smaller. I'm out of time for today, but I'll try to do some functional review work on this tomorrow if no one beats me to it. I would find it very useful to me personally if this feature were committed, and we know there's plenty of user demand for it too. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Thursday, March 07, 2013 10:54 AM Greg Smith wrote: > On 3/5/13 9:07 AM, Amit Kapila wrote: > > In v11 patch, I have changed name of directory to config. > > For file name, currently I have not changed, but if you feel it needs > to be > > changed, kindly suggest any one of above or if any other better you > have in > > mind. > > This seems fine for now. Hashing out exactly which name is best is not > going to be the thing that determines whether this can be committed or > not. > > Your v11 applies cleanly for me too, and the code does look a lot > nicer. > The code diff itself is down to 1300 lines and not nearly as > disruptive, the rest is docs or the many regression tests you've added. I also think the tests added for regression may be more than required. Currently it is organized as below: 1. 3 type of parameters to use with set persistent. a. parameters for which new connection is required (log_connections) b. parameters for which server restart is required c. parameters for which reload is required I think in third category "c", there are many parameters, I think may be 3-4 parameters should be sufficient. The initial intention to keep more parameters is to have test it for various type(Boolean, integer, float, string) 2. Set some of the parameters to default, then try reload and reconnect. 3. Again set those parameters to some value and re-verify by reload and reconnect. 4. Test for set persistent command inside functions and transaction. 5. Set all parameters value to default. I think tests for step-3 can be removed and tests for step-5 can be merged to step-2. If you think above optimization's to reduce number of tests are okay, then I will update the patch. > That's still not the sort of small change Tom was hoping this was > possible, but it's closer to the right range. It may not really be > possible to make this much smaller. > > I'm out of time for today, but I'll try to do some functional review > work on this tomorrow if no one beats me to it. I would find it very > useful to me personally if this feature were committed, and we know > there's plenty of user demand for it too. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Greg Smith
Date:
On 3/7/13 2:42 AM, Amit Kapila wrote: > I also think the tests added for regression may be more than required... > If you think above optimization's to reduce number of tests are okay, then I > will update the patch. I was not trying to get you to remove regression tests. I was just pointing out to everyone that the patch seems longer than it really is, because you put so many of them in there. This is not a problem, and I didn't bring it up to say you should remove some of them. This feature is close to being functional enough to make a lot of people happy. Most of it works the way I've wanted it to for a few years now, an end point several people have pushed toward in pieces for years now. There is a decent sized list of issues that I foundunder careful testing though. Maybe all these can be resolved quickly and this moves on to commit candidate. I'd like to see this feature hit the code base, but it feels like a good amount of work is left to reach there to me right now. I'm focusing on functional rather than code quality review here. My opinion on how the implementation is coded isn't worth very much anyway--I have a bad track record for correctly judging whether something is good quality when it starts messing with the GUC system. The patch has shrunk since original submission a bit, and it has many regression tests it passes. It may still be objectionably long. There is a good amount of cleanup in comments and documentation needed, and I will be happy to work on that part myself. I wanted to get the functional issues outlined first though. I would be shocked if this goes in without someone else wanting more code cleanup or simplification. I don't know who is going to do that other than one of the committers though. If Amit can address the functional ones, I can tweak the text/level of the log messages myself during the cleanup round I do. = Functional changes = There are a number of individually small but serious blocker items I think need to be changed in how this works. I see 7 functional changes to be made before there's no surprising behavior here. I have a lot of commentary about how the log/error messages from this might be improved in addition to that. 1) When you change a sighup or user setting, it writes the config file out. But it does not signal for a reload. Example: $ psql -c "show work_mem" -x work_mem | 1MB $ psql -c "set persistent work_mem='2MB'" SET $ psql -c "show work_mem" -x work_mem | 1MB That the above doesn't leave work_mem set to 2MB for new sessions is surprising. SET PERSISTENT can't do anything about postmaster parameters like shared_buffers. But it should be able to change work_mem or a sighup parameter like checkpoint_segments. The regression test examples call pg_reload_conf() in order to activate the changes. As a user, I would expect the config reload to be automatic after executing SET PERSISTENT on parameters that can be changed dynamically. 2) Once automatic activation is happening, a hint is really needed in cases where it can't happen. I'd like to see a warning about that. There's one like this in the code already: LOG: parameter "shared_buffers" cannot be changed without restarting the server Something like this would be appropriate for SET PERSISTENT: LOG: parameter "shared_buffers" cannot be changed without restarting the server. The changed setting has been saved to config/postgresql.auto.conf. 3) The config/postgresql.auto.conf file (which I'd like to see named 'config/persistent.auto.conf') is completely overwritten when a change is made. When the file is written out, both at initdb time and when changes are made, it should have this as its first line: # Do not edit this file manually! It is overwritten by the SET PERSISTENT command. 4) There is one bad problem case left if I try to make this not work. If I do this: -rm -rf config/ -Remove "include_dir config" -Restart the server. It gives me a warning that SET PERSISTENT won't work because at least one of the multiple things it needs are missing. -mkdir config -psql -c "set persistent checkpoint_segments='32'" That command happily writes out postgresql.auto.conf, but since I removed the include_dir it doesn't matter. It will never become active. The check for the include_dir entry needs to happen each time SET PERSISTENT runs. This is more important than noticing it at server start time. In fact, I think if this problem case includes the WARNING about "include_dir config" not being in the postgresql.conf, the check that happens at server start (shown below) might even go away. People who object to it as log file garbage will be happier, and users will be told about the feature being broken if they try to use it. 5) An error message appears every time you reload configuration, if some part of the SET PERSISTENT mechanism isn't functional. This is going to be too much for some people. And it's confusing when it happens as part an interactive psql session. I have an example way down at the very end of this message. 6) Putting spaces into memory units results in broken config files: $ psql -c "set persistent work_mem='2 MB'" SET $ cat $PGDATA/config/postgresql.auto.conf work_mem = 2 MB This is wrong and you'll get this on reload: LOG: syntax error in file "/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf" line 1, near token "MB" LOG: configuration file "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors; no changes were applied It needs to look like this: work_mem = '2 MB' A regression test for some cases with spaces in the middle should be added too. This case is really weird, due to how the code is reading the existing postgresql.auto.conf file. If I first manually fix the file, and then run a change again, it stays fixed! Look at this weird sequence: $ psql -c "set persistent work_mem='2 MB'" SET $ cat $PGDATA/config/postgresql.auto.conf work_mem = 2 MB [Here this setting is broken] $ vi $PGDATA/config/postgresql.auto.conf [Add quotes around the value, now it works] $ cat $PGDATA/config/postgresql.auto.conf work_mem = '2 MB' $ pg_ctl reload server signaled $ psql -c "set persistent work_mem='4 MB'" SET $ cat $PGDATA/config/postgresql.auto.conf work_mem = '4 MB' Below I'll rant some more about how looking at what's in the existing file, rather than the in-memory GUCs, has some other surprising properties. 7) If I run SET PERSISTENT a lot concurrently, something happens that slows down connecting to the server. Restarting the server makes it go away. I have a pgbench test case demonstrating the problem below, in the "Concurrency" section. I haven't tried to replicate it on another system yet. = Tested features that work fine = Entries added here are tracked as you'd expect: $ psql -c "set persistent checkpoint_segments='32'" $ pg_ctl reload $ psql -xc "select name,setting,sourcefile,sourceline from pg_settings where name='checkpoint_segments'" name | checkpoint_segments setting | 32 sourcefile | /Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf sourceline | 1 When the postgresql.auto.conf file is missing and you use SET PERSISTENT, it quietly creates it when writing out the new setting. = Concurrency and performance slowdown = I made two pgbench scripts that adjust a guc, one user and the other sighup, to a random value: random-join-limit.sql: \setrandom limit 1 8 set persistent join_collapse_limit=:limit; select pg_reload_conf(); random-segments.sql: \setrandom segments 3 16 set persistent checkpoint_segments=:segments; select pg_reload_conf(); I then fired off a bunch of these in parallel: pgbench -n -f random-join-limit.sql -f random-segments.sql -c 8 -T 60 This ran at 1547 TPS on my 8 core Mac, so that's not bad. No assertions and the config file was still valid at the end, which is a good sign the locking method isn't allowing utter chaos. Without the pg_reload_conf() in the test files, it also completed. With the reload happening in one file but not the other, things were also fine. However, one thing I saw is that the server got significantly slower the more I ran this test script. After a few minutes it was down to only 400 TPS. The delay shows up between when I run psql and when I get the prompt back. Here's normal behavior: $ time psql -c "select 1" real 0m0.006s And here's what I get after a single run of this pgbench hammering: $ time psql -c "select 1" real 0m0.800s 800ms? The slowdown is all for psql to start and connect, it's not in the executor: $ time psql -c "explain analyze select 1" QUERY PLAN ------------------------------------------------------------------------------------ Result (cost=0.00..0.01 rows=1 width=0)(actual time=0.001..0.001 rows=1 loops=1) Total runtime: 0.039 ms (2 rows) real 0m0.807s Stopping and restarting the server brings the performance back to normal. Maybe this is one of those assertion build issues, but it smells like there could be nasty memory leak somewhere instead. Clearing whatever weirdness is going on here is a blocking issue. = Loss of in-memory changes = In this example, I change two settings, corrupt the file, and then change one of them a second time: $ psql -c "set persistent checkpoint_segments='32'" $ psql -c "set persistent work_mem='2MB'" $ cat postgresql.auto.conf checkpoint_segments = 32 work_mem = 2MB $ rm postgresql.auto.conf $ psql -c "set persistent work_mem='2MB'" SET $ cat postgresql.auto.conf work_mem = 2MB $ psql -xc "show checkpoint_segments" checkpoint_segments | 3 That this happens isn't unreasonable, and I can live with the limitation. The change to work_mem is lost, but it didn't have to be. A user might expect that in this case the last SET PERSISTENT would have written out a file with both the settings still intact. The running GUC entries certainly knew that postgresql.auto.conf had two lines with these entries at that point. All in-memory persistent changes could be dumped out to postgresql.auto.conf. That's what I had hoped was possible in the implementation. Amit may have this right though, because I think the code he's using is simpler and reliable than what I had in mind. I'm really not sure which way is better. This one isn't a blocker, and if this gets committed it could easily be nudged around later, as an internal change without a catversion bump. = Error messages = If you remove postgresql.auto.conf and restart the server, it gives a warning that SET PERSISTENT won't work until you put it back. The error in this and several similar cases is pretty generic too: WARNING: File "postgresql.auto.conf" is not accessible, either file "postgresql.auto.conf" or folder "config" doesn't exist. or default "config" is not present in postgresql.conf. It would be nice if the error were more specific, individually identifying which of these is the actual problem. I can rewrite that long text entry to be more readable, but I think it should be a series of smaller error checks with their own individual messages instead. If you remove postgresql.auto.conf then exeute "pg_ctl reload", it gives this error 6 times, which seems excessive. Reducing how often it appears in the reload case would be nice. Deleting the whole config directory gives this: LOG: could not open configuration directory "/Users/gsmith/pgwork/data/autoconf/config": No such file or directory LOG: configuration file "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors; no changes were applied If you now try to use the feature, the error message could be better. $ psql -c "set persistent checkpoint_segments='32'" ERROR: Failed to open auto conf temp file "/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf.temp": No such file or directory It would be nice to complain about the config directory being missing, as a first check before the file is opened. Restarting the server in this situation throws the correct error in your face though: LOG: could not open configuration directory "/Users/gsmith/pgwork/data/autoconf/config": No such file or directory FATAL: configuration file "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors If you render this whole system unavailable by removing the "include_dir config", at every server start you'll see this: WARNING: File "postgresql.auto.conf" is not accessible, either file "postgresql.auto.conf" or folder "config" doesn't exist. or default "config" is not present in postgresql.conf.Configuration parameters changed before start/reload of server with SET PERSISTENT command will not be effective. LOG: database system was shut down at 2013-03-09 23:55:03 EST This is a debatable design choice. Some people might not want the file and remove it, but don't want to be nagged about it. If people want to wipe out the file or directory and work the old way, without this feature available, that's fine and they can. To me, helping users who accidentally break this is more important than reducing harmless warning messages for things people did intentionally. WARNING might not be the right level for this though. The existing checks like this I showed above use LOG for this sort of thing. The bigger problem is that this message shows up whenever you reload the config too. Watch this bizarre sequence: gsmith=# select pg_reload_conf(); pg_reload_conf ---------------- t (1 row) gsmith=# gsmith=# select 1; WARNING: File "postgresql.auto.conf" is not accessible, either file "postgresql.auto.conf" or folder "config" doesn't exist. or default "config" is not present in postgresql.conf. Configuration parameters changed before start/reload of server with SET PERSISTENT command will not be effective. ?column? ---------- 1 And as I commented above, shifting more of the checks to SET PERISTENT time could eliminate this check from running at server start and reload altogether. I would be fine with them *also* happening at server start. But I could understand that other people mightnot like that. And having this pop up on every reload, appearing to a client next to another statement altogether, that isn't acceptable though. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Craig Ringer
Date:
On 03/10/2013 11:12 PM, Greg Smith wrote: > > 5) An error message appears every time you reload configuration, if > some part of the SET PERSISTENT mechanism isn't functional. This is > going to be too much for some people. And it's confusing when it > happens as part an interactive psql session. I have an example way > down at the very end of this message. An error when `SET PERSISTENT` is actually run seems sufficient to me if the `config` dir is absent. If the config dir or auto config file exist but are not included, that probably deserves a warning during startup. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Sunday, March 10, 2013 8:43 PM Greg Smith wrote: > On 3/7/13 2:42 AM, Amit Kapila wrote: > > I also think the tests added for regression may be more than > required... > > If you think above optimization's to reduce number of tests are okay, > then I > > will update the patch. > > I was not trying to get you to remove regression tests. I was just > pointing out to everyone that the patch seems longer than it really is, > because you put so many of them in there. This is not a problem, and I > didn't bring it up to say you should remove some of them. > > This feature is close to being functional enough to make a lot of > people > happy. Most of it works the way I've wanted it to for a few years now, > an end point several people have pushed toward in pieces for years now. > There is a decent sized list of issues that I found under careful > testing though. Maybe all these can be resolved quickly and this moves > on to commit candidate. I'd like to see this feature hit the code > base, > but it feels like a good amount of work is left to reach there to me > right now. Thank you for a careful and detail testing. > If Amit can address the functional ones, I can tweak the text/level of > the log messages myself during the cleanup round I do. Sure, I would address the functional issues. > = Functional changes = > > There are a number of individually small but serious blocker items I > think need to be changed in how this works. I see 7 functional changes > to be made before there's no surprising behavior here. I have a lot of > commentary about how the log/error messages from this might be improved > in addition to that. > > 1) When you change a sighup or user setting, it writes the config file > out. But it does not signal for a reload. Example: > > $ psql -c "show work_mem" -x > work_mem | 1MB > $ psql -c "set persistent work_mem='2MB'" > SET > $ psql -c "show work_mem" -x > work_mem | 1MB > > That the above doesn't leave work_mem set to 2MB for new sessions is > surprising. SET PERSISTENT can't do anything about postmaster > parameters like shared_buffers. But it should be able to change > work_mem or a sighup parameter like checkpoint_segments. The > regression > test examples call pg_reload_conf() in order to activate the changes. > As a user, I would expect the config reload to be automatic after > executing SET PERSISTENT on parameters that can be changed dynamically. I think we cannot guarantee even after calling pg_reload_conf(), as this is an asynchronous function call. We already once had a discussion about this point and as I can understand it is concluded that it should not be handled. Refer the below mail: http://www.postgresql.org/message-id/21869.1360683928@sss.pgh.pa.us > 2) Once automatic activation is happening, a hint is really needed in > cases where it can't happen. I'd like to see a warning about that. > There's one like this in the code already: > > LOG: parameter "shared_buffers" cannot be changed without restarting > the server > > Something like this would be appropriate for SET PERSISTENT: > > LOG: parameter "shared_buffers" cannot be changed without restarting > the server. The changed setting has been saved to > config/postgresql.auto.conf. > > 3) The config/postgresql.auto.conf file (which I'd like to see named > 'config/persistent.auto.conf') is completely overwritten when a change > is made. When the file is written out, both at initdb time and when > changes are made, it should have this as its first line: > > # Do not edit this file manually! It is overwritten by the SET > PERSISTENT command. 2 things, you want as part of this comment: a. change the name of .auto file to persistent.auto.conf b. new comment in beginning of persistent.auto.conf file. > 4) There is one bad problem case left if I try to make this not work. > If I do this: > > -rm -rf config/ > -Remove "include_dir config" > -Restart the server. It gives me a warning that SET PERSISTENT won't > work because at least one of the multiple things it needs are missing. > -mkdir config > -psql -c "set persistent checkpoint_segments='32'" > > That command happily writes out postgresql.auto.conf, but since I > removed the include_dir it doesn't matter. It will never become > active. > > The check for the include_dir entry needs to happen each time SET > PERSISTENT runs. This is more important than noticing it at server > start time. In fact, I think if this problem case includes the WARNING > about "include_dir config" not being in the postgresql.conf, the check > that happens at server start (shown below) might even go away. People > who object to it as log file garbage will be happier, and users will be > told about the feature being broken if they try to use it. This can be handled, but for this we need to parse the whole postgresql.conf file and then give the warning. This will increase the time of SET PERSISTENT command. Or the other way might be to note down the information about include dir at startup and then use it, but implementation can get complicated for scenarios if somebody change the postgresql.conf and reload it. I will check whichever way is less complicated, I will handle it that way. > 5) An error message appears every time you reload configuration, if > some > part of the SET PERSISTENT mechanism isn't functional. This is going > to > be too much for some people. And it's confusing when it happens as > part > an interactive psql session. I have an example way down at the very > end > of this message. > > 6) Putting spaces into memory units results in broken config files: > > $ psql -c "set persistent work_mem='2 MB'" > SET > $ cat $PGDATA/config/postgresql.auto.conf > work_mem = 2 MB > > This is wrong and you'll get this on reload: > > LOG: syntax error in file > "/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf" line > 1, > near token "MB" > LOG: configuration file > "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors; > no > changes were applied > > It needs to look like this: > > work_mem = '2 MB' > > A regression test for some cases with spaces in the middle should be > added too. This case is really weird, due to how the code is reading > the existing postgresql.auto.conf file. If I first manually fix the > file, and then run a change again, it stays fixed! Look at this weird > sequence: > > $ psql -c "set persistent work_mem='2 MB'" > SET > $ cat $PGDATA/config/postgresql.auto.conf > work_mem = 2 MB > [Here this setting is broken] > $ vi $PGDATA/config/postgresql.auto.conf > [Add quotes around the value, now it works] > $ cat $PGDATA/config/postgresql.auto.conf > work_mem = '2 MB' > $ pg_ctl reload > server signaled > $ psql -c "set persistent work_mem='4 MB'" > SET > $ cat $PGDATA/config/postgresql.auto.conf > work_mem = '4 MB' I shall fix this behavior. > Below I'll rant some more about how looking at what's in the existing > file, rather than the in-memory GUCs, has some other surprising > properties. > > 7) If I run SET PERSISTENT a lot concurrently, something happens that > slows down connecting to the server. Restarting the server makes it go > away. I have a pgbench test case demonstrating the problem below, in > the "Concurrency" section. I haven't tried to replicate it on another > system yet. > > = Tested features that work fine = > > Entries added here are tracked as you'd expect: > > $ psql -c "set persistent checkpoint_segments='32'" > $ pg_ctl reload > $ psql -xc "select name,setting,sourcefile,sourceline from pg_settings > where name='checkpoint_segments'" > name | checkpoint_segments > setting | 32 > sourcefile | > /Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf > sourceline | 1 > > When the postgresql.auto.conf file is missing and you use SET > PERSISTENT, it quietly creates it when writing out the new setting. > > = Concurrency and performance slowdown = > > I made two pgbench scripts that adjust a guc, one user and the other > sighup, to a random value: > > random-join-limit.sql: > > \setrandom limit 1 8 > set persistent join_collapse_limit=:limit; > select pg_reload_conf(); > > random-segments.sql: > > \setrandom segments 3 16 > set persistent checkpoint_segments=:segments; > select pg_reload_conf(); > > I then fired off a bunch of these in parallel: > > pgbench -n -f random-join-limit.sql -f random-segments.sql -c 8 -T 60 > > This ran at 1547 TPS on my 8 core Mac, so that's not bad. No > assertions > and the config file was still valid at the end, which is a good sign > the > locking method isn't allowing utter chaos. Without the > pg_reload_conf() > in the test files, it also completed. With the reload happening in one > file but not the other, things were also fine. > > However, one thing I saw is that the server got significantly slower > the > more I ran this test script. After a few minutes it was down to only > 400 TPS. The delay shows up between when I run psql and when I get the > prompt back. Here's normal behavior: > > $ time psql -c "select 1" > real 0m0.006s > > And here's what I get after a single run of this pgbench hammering: > > $ time psql -c "select 1" > real 0m0.800s > > 800ms? The slowdown is all for psql to start and connect, it's not in > the executor: > > $ time psql -c "explain analyze select 1" > QUERY PLAN > ----------------------------------------------------------------------- > ------------- > Result (cost=0.00..0.01 rows=1 width=0) (actual time=0.001..0.001 > rows=1 loops=1) > Total runtime: 0.039 ms > (2 rows) > > real 0m0.807s > > Stopping and restarting the server brings the performance back to > normal. Maybe this is one of those assertion build issues, but it > smells like there could be nasty memory leak somewhere instead. > Clearing whatever weirdness is going on here is a blocking issue. Okay, I shall look into this issue. > = Loss of in-memory changes = > > In this example, I change two settings, corrupt the file, and then > change one of them a second time: > > $ psql -c "set persistent checkpoint_segments='32'" > $ psql -c "set persistent work_mem='2MB'" > $ cat postgresql.auto.conf > checkpoint_segments = 32 > work_mem = 2MB > $ rm postgresql.auto.conf > $ psql -c "set persistent work_mem='2MB'" > SET > $ cat postgresql.auto.conf > work_mem = 2MB > $ psql -xc "show checkpoint_segments" > checkpoint_segments | 3 > > That this happens isn't unreasonable, and I can live with the > limitation. The change to work_mem is lost, but it didn't have to be. > A user might expect that in this case the last SET PERSISTENT would > have > written out a file with both the settings still intact. The running > GUC > entries certainly knew that postgresql.auto.conf had two lines with > these entries at that point. All in-memory persistent changes could be > dumped out to postgresql.auto.conf. That's what I had hoped was > possible in the implementation. > > Amit may have this right though, because I think the code he's using is > simpler and reliable than what I had in mind. I'm really not sure > which > way is better. This one isn't a blocker, and if this gets committed it > could easily be nudged around later, as an internal change without a > catversion bump. In-memory values are not guaranteed after running SET PERSISTENT, even though we do pg_reload_conf(), so IMO we should leave this behavior as it is. > = Error messages = > > If you remove postgresql.auto.conf and restart the server, it gives a > warning that SET PERSISTENT won't work until you put it back. The > error > in this and several similar cases is pretty generic too: > > WARNING: File "postgresql.auto.conf" is not accessible, either file > "postgresql.auto.conf" or folder "config" doesn't exist. or default > "config" is not present in postgresql.conf. > > It would be nice if the error were more specific, individually > identifying which of these is the actual problem. I can rewrite that > long text entry to be more readable, but I think it should be a series > of smaller error checks with their own individual messages instead. You are right, we should change this by identifying actual problems. > If you remove postgresql.auto.conf then exeute "pg_ctl reload", it > gives > this error 6 times, which seems excessive. Reducing how often it > appears in the reload case would be nice. > Deleting the whole config directory gives this: > > LOG: could not open configuration directory > "/Users/gsmith/pgwork/data/autoconf/config": No such file or directory > LOG: configuration file > "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors; > no > changes were applied > > If you now try to use the feature, the error message could be better. > > $ psql -c "set persistent checkpoint_segments='32'" > ERROR: Failed to open auto conf temp file > "/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf.temp": > No such file or directory > > It would be nice to complain about the config directory being missing, > as a first check before the file is opened. Restarting the server in > this situation throws the correct error in your face though: > > LOG: could not open configuration directory > "/Users/gsmith/pgwork/data/autoconf/config": No such file or directory > FATAL: configuration file > "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors > > If you render this whole system unavailable by removing the > "include_dir > config", at every server start you'll see this: > > WARNING: File "postgresql.auto.conf" is not accessible, either file > "postgresql.auto.conf" or folder "config" doesn't exist. or default > "config" is not present in postgresql.conf. > Configuration parameters changed before start/reload of server > with SET > PERSISTENT command will not be effective. > LOG: database system was shut down at 2013-03-09 23:55:03 EST > > This is a debatable design choice. Some people might not want the file > and remove it, but don't want to be nagged about it. If people want to > wipe out the file or directory and work the old way, without this > feature available, that's fine and they can. To me, helping users who > accidentally break this is more important than reducing harmless > warning > messages for things people did intentionally. WARNING might not be the > right level for this though. The existing checks like this I showed > above use LOG for this sort of thing. > > The bigger problem is that this message shows up whenever you reload > the > config too. Watch this bizarre sequence: > > gsmith=# select pg_reload_conf(); > pg_reload_conf > ---------------- > t > (1 row) > > gsmith=# > gsmith=# select 1; > WARNING: File "postgresql.auto.conf" is not accessible, either file > "postgresql.auto.conf" or folder "config" doesn't exist. or default > "config" is not present in postgresql.conf. > Configuration parameters changed before start/reload of server with SET > PERSISTENT command will not be effective. > ?column? > ---------- > 1 > > And as I commented above, shifting more of the checks to SET PERISTENT > time could eliminate this check from running at server start and reload > altogether. I would be fine with them *also* happening at server > start. > But I could understand that other people might not like that. And > having this pop up on every reload, appearing to a client next to > another statement altogether, that isn't acceptable though. Agreed, I will add the check at time of SET PERSISTENT command. Also I will try to remove this message at time of reload. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Greg Smith
Date:
On 3/11/13 2:48 AM, Amit Kapila wrote: >> 1) When you change a sighup or user setting, it writes the config file >> out. But it does not signal for a reload. Example: > > I think we cannot guarantee even after calling pg_reload_conf(), as this is > an > asynchronous function call. > We already once had a discussion about this point and as I can understand it > is > concluded that it should not be handled. Refer the below mail: > http://www.postgresql.org/message-id/21869.1360683928@sss.pgh.pa.us I wasn't complaining that the change isn't instant. I understand that can't be done. But I think the signal to reload should be sent. If people execute SET PERSISTENT, and it doesn't actually do anything until the server is next restarted, they will be very surprised. It's OK if it doesn't do anything for a second, or until new sessions connect, because that's just how SIGHUP/session variables work. That's a documentation issue. Not reloading the config at all, I think that's going to trigger a ton of future support problems. > 2 things, you want as part of this comment: > a. change the name of .auto file to persistent.auto.conf > b. new comment in beginning of persistent.auto.conf file. Right. config/persistent.auto.conf seems like it addresses everyone's ideas on how the file should be named. If there's a warning not to edit it in the file, too, it would be making obvious to users what is happening. The file handles persistent changes, it's automatically configured, and it's a config file. Putting "postgresql" in the name seems unnecessary to me. >> The check for the include_dir entry needs to happen each time SET >> PERSISTENT runs. > This can be handled, but for this we need to parse the whole postgresql.conf > file > and then give the warning. This will increase the time of SET PERSISTENT > command. I don't think that's a problem. I can run the command 1800 times per second on my laptop right now. If it could only run once per second, that would still be fast enough. Doing this the most reliable way it can be handled is the important part; if that happens to be the slowest way, too, that is acceptable. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Fujii Masao
Date:
On Mon, Mar 11, 2013 at 4:39 PM, Greg Smith <greg@2ndquadrant.com> wrote: > On 3/11/13 2:48 AM, Amit Kapila wrote: >>> >>> 1) When you change a sighup or user setting, it writes the config file >>> out. But it does not signal for a reload. Example: >> >> >> I think we cannot guarantee even after calling pg_reload_conf(), as this >> is >> an >> asynchronous function call. >> We already once had a discussion about this point and as I can understand >> it >> is >> concluded that it should not be handled. Refer the below mail: >> http://www.postgresql.org/message-id/21869.1360683928@sss.pgh.pa.us > > > I wasn't complaining that the change isn't instant. I understand that can't > be done. But I think the signal to reload should be sent. If people > execute SET PERSISTENT, and it doesn't actually do anything until the server > is next restarted, they will be very surprised. It's OK if it doesn't do > anything for a second, or until new sessions connect, because that's just > how SIGHUP/session variables work. That's a documentation issue. Not > reloading the config at all, I think that's going to trigger a ton of future > support problems. I agree with you if SET PERSISTENT reloads only postgresql.auto.conf. But in current conf reload mechanism, other configuration files like pg_hba.conf are also reloaded when pg_read_conf() is executed. Probably I don't like this behavior. Users might get surprised that the configuration changes by their manual operation (by not SET PERSISTENT) are also activated by SET PERSISTENT. And, this change would make the patch bigger... >>> The check for the include_dir entry needs to happen each time SET >>> PERSISTENT runs. >> >> This can be handled, but for this we need to parse the whole >> postgresql.conf >> >> file >> and then give the warning. This will increase the time of SET PERSISTENT >> command. > > > I don't think that's a problem. I can run the command 1800 times per second > on my laptop right now. If it could only run once per second, that would > still be fast enough. Doing this the most reliable way it can be handled is > the important part; if that happens to be the slowest way, too, that is > acceptable. Why should the include_dir entry for SET PERSISTENT exist in postgresql.conf? Is there any use case that users change that include_dir setting? If not, what about hardcoding the include_dir entry for SET PERSISTENT in the core? If we do this, we don't need to handle this problem at all. Regards, -- Fujii Masao
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Josh Berkus
Date:
> I agree with you if SET PERSISTENT reloads only postgresql.auto.conf. > But in current conf reload mechanism, other configuration files like > pg_hba.conf are also reloaded when pg_read_conf() is executed. Probably > I don't like this behavior. Users might get surprised that the configuration > changes by their manual operation (by not SET PERSISTENT) are also > activated by SET PERSISTENT. I'm not sure I see this as a problem if it's well-documented. Basically you're saying that if a user does: 1. make some changes to pg_hba.conf 2. make some changes via SET PERSISTENT 3. pg_hba.conf changes will be automatically loaded It's a little surprising for some users, but if the behavior is well-documented, then I really don't see it as a problem. Consider the parallel of this: 1. make some changes to postgresql.conf 2. make some changes via SET PERSISTENT 3. manual pg.conf changes are automatically loaded I don't see this path as a problem ... in fact, I kind of think that as many users will expect the above as be surprised by it. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On Mon, Mar 11, 2013 at 7:39 AM, Greg Smith <greg@2ndquadrant.com> wrote: > I wasn't complaining that the change isn't instant. I understand that can't > be done. But I think the signal to reload should be sent. If people > execute SET PERSISTENT, and it doesn't actually do anything until the server > is next restarted, they will be very surprised. It's OK if it doesn't do > anything for a second, or until new sessions connect, because that's just > how SIGHUP/session variables work. That's a documentation issue. Not > reloading the config at all, I think that's going to trigger a ton of future > support problems. Think also about the case where someone wants to change multiple values together and having just some set and not others would be inconsistent. I can see you're right about surprising users but is there not another way to solve the same problem without making that impossible? -- greg
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Craig Ringer
Date:
On 03/12/2013 03:19 AM, Greg Stark wrote: > Think also about the case where someone wants to change multiple > values together and having just some set and not others would be > inconsistent. Yeah, that's a killer. The reload would need to be scheduled for COMMIT time, it can't be done by `SET PERSISTENT` directly. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Monday, March 11, 2013 11:02 PM Fujii Masao wrote: > On Mon, Mar 11, 2013 at 4:39 PM, Greg Smith <greg@2ndquadrant.com> > wrote: > > On 3/11/13 2:48 AM, Amit Kapila wrote: > >>> > >>> 1) When you change a sighup or user setting, it writes the config > file > >>> out. But it does not signal for a reload. Example: > >> > >> > >> I think we cannot guarantee even after calling pg_reload_conf(), as > this > >> is > >> an > >> asynchronous function call. > >> We already once had a discussion about this point and as I can > understand > >> it > >> is > >> concluded that it should not be handled. Refer the below mail: > >> http://www.postgresql.org/message-id/21869.1360683928@sss.pgh.pa.us > > > > > > I wasn't complaining that the change isn't instant. I understand > that can't > > be done. But I think the signal to reload should be sent. If people > > execute SET PERSISTENT, and it doesn't actually do anything until the > server > > is next restarted, they will be very surprised. It's OK if it > doesn't do > > anything for a second, or until new sessions connect, because that's > just > > how SIGHUP/session variables work. That's a documentation issue. > Not > > reloading the config at all, I think that's going to trigger a ton of > future > > support problems. > > I agree with you if SET PERSISTENT reloads only postgresql.auto.conf. > But in current conf reload mechanism, other configuration files like > pg_hba.conf are also reloaded when pg_read_conf() is executed. Probably > I don't like this behavior. Users might get surprised that the > configuration > changes by their manual operation (by not SET PERSISTENT) are also > activated by SET PERSISTENT. > > And, this change would make the patch bigger... it's more of issue in consistency as pg_reload_conf() is asynchronous. > >>> The check for the include_dir entry needs to happen each time SET > >>> PERSISTENT runs. > >> > >> This can be handled, but for this we need to parse the whole > >> postgresql.conf > >> > >> file > >> and then give the warning. This will increase the time of SET > PERSISTENT > >> command. > > > > > > I don't think that's a problem. I can run the command 1800 times per > second > > on my laptop right now. If it could only run once per second, that > would > > still be fast enough. Doing this the most reliable way it can be > handled is > > the important part; if that happens to be the slowest way, too, that > is > > acceptable. > > Why should the include_dir entry for SET PERSISTENT exist in > postgresql.conf? > Is there any use case that users change that include_dir setting? If > not, what > about hardcoding the include_dir entry for SET PERSISTENT in the core? > If we do this, we don't need to handle this problem at all. It will be for users who want to use the configuration settings as per previous version, means by manually setting in conf file. With Regards, Amit Kapila.
> From: gsstark@gmail.com [mailto:gsstark@gmail.com] On Behalf Of Greg > Stark > Sent: Tuesday, March 12, 2013 12:50 AM > To: Greg Smith > Cc: Amit Kapila; Andres Freund; Boszormenyi Zoltan; pgsql- > hackers@postgresql.org > Subject: Re: Re: Proposal for Allow postgresql.conf values to be > changed via SQL [review] > > On Mon, Mar 11, 2013 at 7:39 AM, Greg Smith <greg@2ndquadrant.com> > wrote: > > > I wasn't complaining that the change isn't instant. I understand > that can't > > be done. But I think the signal to reload should be sent. If people > > execute SET PERSISTENT, and it doesn't actually do anything until the > server > > is next restarted, they will be very surprised. It's OK if it > doesn't do > > anything for a second, or until new sessions connect, because that's > just > > how SIGHUP/session variables work. That's a documentation issue. > Not > > reloading the config at all, I think that's going to trigger a ton of > future > > support problems. > > Think also about the case where someone wants to change multiple > values together and having just some set and not others would be > inconsistent. Do you mean to say that because some variables can only be set after restart can lead to inconsistency, or is it because of asynchronous nature of pg_reload_conf()? > I can see you're right about surprising users but is there not another > way to solve the same problem without making that impossible? With Regards, Amit Kapila.
On 3/11/13 12:19 PM, Greg Stark wrote: > Think also about the case where someone wants to change multiple > values together and having just some set and not others would be > inconsistent. Isn't that an argument for syntax to make an exception though? If starting from a blank slate I would say this case should be: SET PERSISTENT (POSTPONE) maintenance_work_mem='2GB'; SET PERSISTENT work_mem='2GB'; That's jumping right over a few rounds of simpler ways to do this, and just going right to the approach we know allows adding more such options later with minimal grammar impact. From the perspective of what a useful minimal commit looks like, I would think 90% of the use cases here expect immediate reload of just the postgresql.conf--but probably not pg_hba.conf--and any other refinement could wait until a next release. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
On Tue, Mar 12, 2013 at 9:06 AM, Greg Smith <greg@2ndquadrant.com> wrote: > That's jumping right over a few rounds of simpler ways to do this, and just > going right to the approach we know allows adding more such options later > with minimal grammar impact. As Craig intimated, the minimal grammar impact would be simply BEGIN; set persistent maintenance_work_mem='2GB'; set persistent work_mem='2GB'; COMMIT; Sending the sighup at transaction end seems like a fairly safe thing to do too. It's hard to imagine it failing and if it did the worst case would be that other backends would still have the old values too. To be honest I went through the gucs last night looking for a better example than this and didn't really find any compelling examples. The best I could come up with was adjusting several query costs in tandem. Hardly the end of the world if it didn't work. The worst case would be some security sensitive option alongside a second parameter for limiting its scope, but I couldn't find any such examples in a quick search. (And that's probably a good thing) -- greg
Greg Stark <stark@mit.edu> writes:
> As Craig intimated, the minimal grammar impact would be simply
> BEGIN;
> set persistent maintenance_work_mem='2GB';
> set persistent work_mem='2GB';
> COMMIT;
> Sending the sighup at transaction end seems like a fairly safe thing
> to do too. It's hard to imagine it failing and if it did the worst
> case would be that other backends would still have the old values too.
This would only be sane if we also postponed writing the file until
commit. I don't know what is in the patch at the moment, but I remember
that we had talked of executing the file write immediately when SET
PERSISTENT is issued.
Maybe it's worth going in that direction just to cover the
multiple-updates case more conveniently.
regards, tom lane
On Tuesday, March 12, 2013 9:37 PM Tom Lane wrote: > Greg Stark <stark@mit.edu> writes: > > As Craig intimated, the minimal grammar impact would be simply > > > BEGIN; > > set persistent maintenance_work_mem='2GB'; > > set persistent work_mem='2GB'; > > COMMIT; > > > Sending the sighup at transaction end seems like a fairly safe thing > > to do too. It's hard to imagine it failing and if it did the worst > > case would be that other backends would still have the old values > too. > > This would only be sane if we also postponed writing the file until > commit. I don't know what is in the patch at the moment, but I > remember > that we had talked of executing the file write immediately when SET > PERSISTENT is issued. Currently the patch handles it by executing the file write immediately when SET PERSISTENT is issued. You have pointed at that time it's better to write file immediately as if we postpone it to commit time, it can have some other complications. The only doubt I have in doing pg_reload_conf() is about the point raised by Fujii Masao yesterday, that is it will load all other configurations apart from what had been changed by SET PERSISTENT. Are you okay with it, if we implement this and document such behavior clearly? With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Craig Ringer
Date:
On 03/12/2013 06:27 AM, Craig Ringer wrote: >> > Think also about the case where someone wants to change multiple >> > values together and having just some set and not others would be >> > inconsistent. > Yeah, that's a killer. The reload would need to be scheduled for COMMIT > time, it can't be done by `SET PERSISTENT` directly. Thinking about this some more, I'm not sure this is a good idea. Right now, SET takes effect immediately. Always, without exception. Delaying SET PERSISTENT's effects until commit would make it inconsistent with SET's normal behaviour. However, *not* delaying it would make it another quirky not-transactional not-atomic command. That's OK, but if it's not going to follow transactional semantics it should not be allowed to run within a transaction, like VACUUM . Writing the changes immediately but deferring the reload until commit seems to be the worst of those two worlds. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
2013-03-13 07:42 keltezéssel, Craig Ringer írta: > On 03/12/2013 06:27 AM, Craig Ringer wrote: >>>> Think also about the case where someone wants to change multiple >>>> values together and having just some set and not others would be >>>> inconsistent. >> Yeah, that's a killer. The reload would need to be scheduled for COMMIT >> time, it can't be done by `SET PERSISTENT` directly. > Thinking about this some more, I'm not sure this is a good idea. > > Right now, SET takes effect immediately. Always, without exception. > Delaying SET PERSISTENT's effects until commit would make it > inconsistent with SET's normal behaviour. > > However, *not* delaying it would make it another quirky > not-transactional not-atomic command. That's OK, but if it's not going > to follow transactional semantics it should not be allowed to run within > a transaction, like VACUUM . > > Writing the changes immediately but deferring the reload until commit > seems to be the worst of those two worlds. I was thinking about it a little. There is a hook that runs at the end of (sub-)transactions. It can be abused for this purpose to make SET PERSISTENT transactional. The subtransactions can also stack these settings, by forgetting settings upon ROLLBACK [ TO SAVEPOINT ] and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT. All it needs is a list and maintaining intermediate pointers when entering into a new level of SAVEPOINT. The functions to register such hooks are in src/include/access/xact.h: extern void RegisterXactCallback(XactCallback callback, void *arg); extern void UnregisterXactCallback(XactCallback callback, void *arg); extern void RegisterSubXactCallback(SubXactCallback callback, void *arg); extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg); The per-session local effect can also be immediate, it just needs the normal SET and SET PERSISTENT code paths to be unified. On the other hand, it would make a lot of sense to implement it as one setting per file or extending the code to allow modifying settings in bulk. The one setting per file should be easier and it would also allow extensions to drop their settings automatically into the automatic config directory. I don't know who mentioned this idea about extensions but it also came up a while back. Best regards, Zoltán Böszörményi -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
2013-03-13 09:09 keltezéssel, Boszormenyi Zoltan írta: > 2013-03-13 07:42 keltezéssel, Craig Ringer írta: >> On 03/12/2013 06:27 AM, Craig Ringer wrote: >>>>> Think also about the case where someone wants to change multiple >>>>> values together and having just some set and not others would be >>>>> inconsistent. >>> Yeah, that's a killer. The reload would need to be scheduled for COMMIT >>> time, it can't be done by `SET PERSISTENT` directly. >> Thinking about this some more, I'm not sure this is a good idea. >> >> Right now, SET takes effect immediately. Always, without exception. >> Delaying SET PERSISTENT's effects until commit would make it >> inconsistent with SET's normal behaviour. >> >> However, *not* delaying it would make it another quirky >> not-transactional not-atomic command. That's OK, but if it's not going >> to follow transactional semantics it should not be allowed to run within >> a transaction, like VACUUM . >> >> Writing the changes immediately but deferring the reload until commit >> seems to be the worst of those two worlds. > > I was thinking about it a little. There is a hook that runs at the end > of (sub-)transactions. It can be abused for this purpose to make > SET PERSISTENT transactional. The subtransactions can also stack > these settings, by forgetting settings upon ROLLBACK [ TO SAVEPOINT ] > and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT. By "overwriting at RELEASE SAVEPOINT" I meant only the memory copy so the value that was set last is remembered at the same level of subxact. The final settings make it to the configuration file at COMMIT time. > All it needs is a list and maintaining intermediate pointers when entering > into a new level of SAVEPOINT. The functions to register such hooks are > in src/include/access/xact.h: > > extern void RegisterXactCallback(XactCallback callback, void *arg); > extern void UnregisterXactCallback(XactCallback callback, void *arg); > extern void RegisterSubXactCallback(SubXactCallback callback, void *arg); > extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg); > > The per-session local effect can also be immediate, it just needs > the normal SET and SET PERSISTENT code paths to be unified. > > On the other hand, it would make a lot of sense to implement it > as one setting per file or extending the code to allow modifying > settings in bulk. The one setting per file should be easier and > it would also allow extensions to drop their settings automatically > into the automatic config directory. I don't know who mentioned > this idea about extensions but it also came up a while back. > > Best regards, > Zoltán Böszörményi > -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-03-12 10:46:53 +0530, Amit Kapila wrote: > Do you mean to say that because some variables can only be set after restart > can lead to > inconsistency, or is it because of asynchronous nature of pg_reload_conf()? As long as SET PERSISTENT cannot be executed inside a transaction - or only takes effect after its end - there doesn't seem to be any problem executing ProcessConfigFile() directly. The reason its not executed directly is that it is normally called during query execution and it wouldn't be nice changing stuff after half of a query has been processed. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-03-13 09:09:24 +0100, Boszormenyi Zoltan wrote: > 2013-03-13 07:42 keltezéssel, Craig Ringer írta: > >On 03/12/2013 06:27 AM, Craig Ringer wrote: > >>>>Think also about the case where someone wants to change multiple > >>>>values together and having just some set and not others would be > >>>>inconsistent. > >>Yeah, that's a killer. The reload would need to be scheduled for COMMIT > >>time, it can't be done by `SET PERSISTENT` directly. > >Thinking about this some more, I'm not sure this is a good idea. > > > >Right now, SET takes effect immediately. Always, without exception. > >Delaying SET PERSISTENT's effects until commit would make it > >inconsistent with SET's normal behaviour. > > > >However, *not* delaying it would make it another quirky > >not-transactional not-atomic command. That's OK, but if it's not going > >to follow transactional semantics it should not be allowed to run within > >a transaction, like VACUUM . > > > >Writing the changes immediately but deferring the reload until commit > >seems to be the worst of those two worlds. > > I was thinking about it a little. There is a hook that runs at the end > of (sub-)transactions. It can be abused for this purpose to make > SET PERSISTENT transactional. The subtransactions can also stack > these settings, by forgetting settings upon ROLLBACK [ TO SAVEPOINT ] > and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT. > All it needs is a list and maintaining intermediate pointers when entering > into a new level of SAVEPOINT. The functions to register such hooks are > in src/include/access/xact.h: > > extern void RegisterXactCallback(XactCallback callback, void *arg); > extern void UnregisterXactCallback(XactCallback callback, void *arg); > extern void RegisterSubXactCallback(SubXactCallback callback, void *arg); > extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg); (sub)xact commit/abort already calls AtEOXact_GUC(commit, nestlevel). So you wouldn't even need that. It seems we could add another value to enum GucStackState, like GUC_SET_PERSISTENT - and process those only if commit && nestlevel == 1. Everytime you see one with commit && nestlevel > 1 you put them into them into the stack one level up. This seems like its somewhat in line with the way SET LOCAL is implemented? > On the other hand, it would make a lot of sense to implement it > as one setting per file or extending the code to allow modifying > settings in bulk. The one setting per file should be easier and > it would also allow extensions to drop their settings automatically > into the automatic config directory. I don't know who mentioned > this idea about extensions but it also came up a while back. I still think one-setting-per-file is the right way to go, yes. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Wednesday, March 13, 2013 6:10 PM Andres Freund wrote: > On 2013-03-12 10:46:53 +0530, Amit Kapila wrote: > > Do you mean to say that because some variables can only be set after > restart > > can lead to > > inconsistency, or is it because of asynchronous nature of > pg_reload_conf()? > > As long as SET PERSISTENT cannot be executed inside a transaction - or > only takes effect after its end - there doesn't seem to be any problem > executing ProcessConfigFile() directly. Do you mean to say we call directly ProcessConfigFile() at end of SET PERSISTENT instead Of pg_reload_conf() but in that case would it load the variables for other backends? With Regards, Amit Kapila.
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-03-13 18:38:12 +0530, Amit Kapila wrote: > On Wednesday, March 13, 2013 6:10 PM Andres Freund wrote: > > On 2013-03-12 10:46:53 +0530, Amit Kapila wrote: > > > Do you mean to say that because some variables can only be set after > > restart > > > can lead to > > > inconsistency, or is it because of asynchronous nature of > > pg_reload_conf()? > > > > As long as SET PERSISTENT cannot be executed inside a transaction - or > > only takes effect after its end - there doesn't seem to be any problem > > executing ProcessConfigFile() directly. > > Do you mean to say we call directly ProcessConfigFile() at end of SET > PERSISTENT instead > Of pg_reload_conf() but in that case would it load the variables for other > backends? I'd say do both. Yes, we would evaluate config potentially twice. Who cares. Messages inside non-postmaster environments are only output at DEBUG2 anyway. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Wednesday, March 13, 2013 6:44 PM Andres Freund wrote: > On 2013-03-13 18:38:12 +0530, Amit Kapila wrote: > > On Wednesday, March 13, 2013 6:10 PM Andres Freund wrote: > > > On 2013-03-12 10:46:53 +0530, Amit Kapila wrote: > > > > Do you mean to say that because some variables can only be set > after > > > restart > > > > can lead to > > > > inconsistency, or is it because of asynchronous nature of > > > pg_reload_conf()? > > > > > > As long as SET PERSISTENT cannot be executed inside a transaction - > or > > > only takes effect after its end - there doesn't seem to be any > problem > > > executing ProcessConfigFile() directly. > > > > Do you mean to say we call directly ProcessConfigFile() at end of SET > > PERSISTENT instead > > Of pg_reload_conf() but in that case would it load the variables for > other > > backends? > > I'd say do both. Yes, we would evaluate config potentially twice. Who > cares. Messages inside non-postmaster environments are only output at > DEBUG2 > anyway. I could see your point, when you say do both that you want that in current session, the values will be immediately available which can make user happy. However if there is any error during function ProcessConfigFile(), it could be little inconvenient for user as the setting would have been done in file but memory processing has created problem. Ideally, there should not be any error in function ProcessConfigFile(), but might be some palloc or such call would be sufficient to make user unhappy. With Regards, Amit Kapila.
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Andres Freund
Date:
On 2013-03-13 18:52:48 +0530, Amit Kapila wrote: > On Wednesday, March 13, 2013 6:44 PM Andres Freund wrote: > > On 2013-03-13 18:38:12 +0530, Amit Kapila wrote: > > > On Wednesday, March 13, 2013 6:10 PM Andres Freund wrote: > > > > On 2013-03-12 10:46:53 +0530, Amit Kapila wrote: > > > > > Do you mean to say that because some variables can only be set > > after > > > > restart > > > > > can lead to > > > > > inconsistency, or is it because of asynchronous nature of > > > > pg_reload_conf()? > > > > > > > > As long as SET PERSISTENT cannot be executed inside a transaction - > > or > > > > only takes effect after its end - there doesn't seem to be any > > problem > > > > executing ProcessConfigFile() directly. > > > > > > Do you mean to say we call directly ProcessConfigFile() at end of SET > > > PERSISTENT instead > > > Of pg_reload_conf() but in that case would it load the variables for > > other > > > backends? > > > > I'd say do both. Yes, we would evaluate config potentially twice. Who > > cares. Messages inside non-postmaster environments are only output at > > DEBUG2 > > anyway. > > I could see your point, when you say do both that you want that in current > session, > the values will be immediately available which can make user happy. > However if there is any error during function ProcessConfigFile(), it could > be little inconvenient for user as the setting would have been done in file but memory > processing has created problem. But thats pretty independent from this? If anything it allows for *better* reporting of problems since you could convert the log level to WARNING if ProcessConfigFile() is executed in foreground - which at least interactive sessions normally will noramlly be displayed for the user. If you do don't do it immediately you're in the same situation after the pg_reload_config(), just that the user won't see any error messages. There is something I am more worried about which is that it ight be bad if a postmaster child adapts new values bfore postmaster does. I right now can't think of any new dangers since the reverse is already true... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
2013-03-13 13:45 keltezéssel, Andres Freund írta: > On 2013-03-13 09:09:24 +0100, Boszormenyi Zoltan wrote: >> 2013-03-13 07:42 keltezéssel, Craig Ringer írta: >>> On 03/12/2013 06:27 AM, Craig Ringer wrote: >>>>>> Think also about the case where someone wants to change multiple >>>>>> values together and having just some set and not others would be >>>>>> inconsistent. >>>> Yeah, that's a killer. The reload would need to be scheduled for COMMIT >>>> time, it can't be done by `SET PERSISTENT` directly. >>> Thinking about this some more, I'm not sure this is a good idea. >>> >>> Right now, SET takes effect immediately. Always, without exception. >>> Delaying SET PERSISTENT's effects until commit would make it >>> inconsistent with SET's normal behaviour. >>> >>> However, *not* delaying it would make it another quirky >>> not-transactional not-atomic command. That's OK, but if it's not going >>> to follow transactional semantics it should not be allowed to run within >>> a transaction, like VACUUM . >>> >>> Writing the changes immediately but deferring the reload until commit >>> seems to be the worst of those two worlds. >> I was thinking about it a little. There is a hook that runs at the end >> of (sub-)transactions. It can be abused for this purpose to make >> SET PERSISTENT transactional. The subtransactions can also stack >> these settings, by forgetting settings upon ROLLBACK [ TO SAVEPOINT ] >> and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT. >> All it needs is a list and maintaining intermediate pointers when entering >> into a new level of SAVEPOINT. The functions to register such hooks are >> in src/include/access/xact.h: >> >> extern void RegisterXactCallback(XactCallback callback, void *arg); >> extern void UnregisterXactCallback(XactCallback callback, void *arg); >> extern void RegisterSubXactCallback(SubXactCallback callback, void *arg); >> extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg); > (sub)xact commit/abort already calls AtEOXact_GUC(commit, nestlevel). So you > wouldn't even need that. Yes, thanks. > It seems we could add another value to enum > GucStackState, like GUC_SET_PERSISTENT - and process those only if commit && > nestlevel == 1. Maybe it's not needed, only enum GucAction needs a new GUC_ACTION_PERSISTENT value since that's what has any business in push_old_value(). Adding two new members to GucStack like these are enough bool has_persistent; config_var_value persistent; and SET PERSISTENT can be treated as GUC_SET. push_old_value() can merge GUC values set in the same transaction level. > Everytime you see one with commit && nestlevel > 1 you put them into them into > the stack one level up. > > This seems like its somewhat in line with the way SET LOCAL is implemented? > >> On the other hand, it would make a lot of sense to implement it >> as one setting per file or extending the code to allow modifying >> settings in bulk. The one setting per file should be easier and >> it would also allow extensions to drop their settings automatically >> into the automatic config directory. I don't know who mentioned >> this idea about extensions but it also came up a while back. > I still think one-setting-per-file is the right way to go, yes. > > Greetings, > > Andres Freund > -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
On Wednesday, March 13, 2013 8:06 PM Andres Freund wrote: > On 2013-03-13 18:52:48 +0530, Amit Kapila wrote: > > On Wednesday, March 13, 2013 6:44 PM Andres Freund wrote: > > > On 2013-03-13 18:38:12 +0530, Amit Kapila wrote: > > > > On Wednesday, March 13, 2013 6:10 PM Andres Freund wrote: > > > > > On 2013-03-12 10:46:53 +0530, Amit Kapila wrote: > > > > > > Do you mean to say that because some variables can only be > set > > > after > > > > > restart > > > > > > can lead to > > > > > > inconsistency, or is it because of asynchronous nature of > > > > > pg_reload_conf()? > > > > > > > > > > As long as SET PERSISTENT cannot be executed inside a > transaction - > > > or > > > > > only takes effect after its end - there doesn't seem to be any > > > problem > > > > > executing ProcessConfigFile() directly. > > > > > > > > Do you mean to say we call directly ProcessConfigFile() at end of > SET > > > > PERSISTENT instead > > > > Of pg_reload_conf() but in that case would it load the variables > for > > > other > > > > backends? > > > > > > I'd say do both. Yes, we would evaluate config potentially twice. > Who > > > cares. Messages inside non-postmaster environments are only output > at > > > DEBUG2 > > > anyway. > > > > I could see your point, when you say do both that you want that in > current > > session, > > the values will be immediately available which can make user happy. > > However if there is any error during function ProcessConfigFile(), it > could > > be little inconvenient for user as the setting would have been done > in file but memory > > processing has created problem. > > But thats pretty independent from this? If anything it allows for > *better* reporting of problems since you could convert the log level to > WARNING if ProcessConfigFile() is executed in foreground - which at > least > interactive sessions normally will noramlly be displayed for the user. I checked in ProcessConfigFile(), error level will be DEBUG2 for IsUnderPostmaster. So it should not be a problem. There is another observation from Greg Smith's test-7 (refer his Review mail), that if we call pg_reload_conf() after SET Persistent, it is probably causing some memory leak which leads to delay of further connections. I am checking that problem that why calling pg_reload_conf() leads to memory increase which further causes delay in connection. I think once the reason for same is clear, we can decide on how to load conf file after command execution. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Amit Kapila
Date:
On Monday, March 11, 2013 12:18 PM Amit Kapila wrote: > On Sunday, March 10, 2013 8:43 PM Greg Smith wrote: > > On 3/7/13 2:42 AM, Amit Kapila wrote: > > > I also think the tests added for regression may be more than > > required... > > > If you think above optimization's to reduce number of tests are > okay, > > then I > > > will update the patch. > > > > I was not trying to get you to remove regression tests. I was just > > pointing out to everyone that the patch seems longer than it really > is, > > 1) When you change a sighup or user setting, it writes the config > file > > out. But it does not signal for a reload. Example: > > > > $ psql -c "show work_mem" -x > > work_mem | 1MB > > $ psql -c "set persistent work_mem='2MB'" > > SET > > $ psql -c "show work_mem" -x > > work_mem | 1MB > > > > That the above doesn't leave work_mem set to 2MB for new sessions is > > surprising. SET PERSISTENT can't do anything about postmaster > > parameters like shared_buffers. But it should be able to change > > work_mem or a sighup parameter like checkpoint_segments. The > > regression > > test examples call pg_reload_conf() in order to activate the changes. > > As a user, I would expect the config reload to be automatic after > > executing SET PERSISTENT on parameters that can be changed > dynamically. > > I think we cannot guarantee even after calling pg_reload_conf(), as > this is an > asynchronous function call. > We already once had a discussion about this point and as I can > understand it is > concluded that it should not be handled. Refer the below mail: > http://www.postgresql.org/message-id/21869.1360683928@sss.pgh.pa.us > > > > 2) Once automatic activation is happening, a hint is really needed in > > cases where it can't happen. I'd like to see a warning about that. > > There's one like this in the code already: > > > > LOG: parameter "shared_buffers" cannot be changed without restarting > > the server > > > > Something like this would be appropriate for SET PERSISTENT: > > > > LOG: parameter "shared_buffers" cannot be changed without restarting > > the server. The changed setting has been saved to > > config/postgresql.auto.conf. I have not updated code to address 1 and 2, because as per my observation the reason for your test-7 is that you have called pg_reload_conf(), after each test. So I feel calling directly in code will also lead to same problem. I will hold to change code for this till that issue is resolved. > > 3) The config/postgresql.auto.conf file (which I'd like to see named > > 'config/persistent.auto.conf') is completely overwritten when a > change > > is made. When the file is written out, both at initdb time and when > > changes are made, it should have this as its first line: > > > > # Do not edit this file manually! It is overwritten by the SET > > PERSISTENT command. > > 2 things, you want as part of this comment: > a. change the name of .auto file to persistent.auto.conf > b. new comment in beginning of persistent.auto.conf file. Fixed both 2a and 2b. > > 4) There is one bad problem case left if I try to make this not work. > > If I do this: > > > > -rm -rf config/ > > -Remove "include_dir config" > > -Restart the server. It gives me a warning that SET PERSISTENT won't > > work because at least one of the multiple things it needs are > missing. > > -mkdir config > > -psql -c "set persistent checkpoint_segments='32'" > > > > That command happily writes out postgresql.auto.conf, but since I > > removed the include_dir it doesn't matter. It will never become > > active. > > > > The check for the include_dir entry needs to happen each time SET > > PERSISTENT runs. This is more important than noticing it at server > > start time. In fact, I think if this problem case includes the > WARNING > > about "include_dir config" not being in the postgresql.conf, the > check > > that happens at server start (shown below) might even go away. > People > > who object to it as log file garbage will be happier, and users will > be > > told about the feature being broken if they try to use it. > > This can be handled, but for this we need to parse the whole > postgresql.conf file > and then give the warning. This will increase the time of SET > PERSISTENT command. Fixed by parsing the file and if include_dir is missing, then give Warning to user. In case user deletes the folder, ERROR will be thrown, because in anycase it cannot perform the Command if folder is deleted. If persistent.auto.conf file is missing, then don't raise any error as SET PERSISTENT command will create the file. > > 5) An error message appears every time you reload configuration, if > > some > > part of the SET PERSISTENT mechanism isn't functional. This is going > > to > > be too much for some people. And it's confusing when it happens as > > part > > an interactive psql session. I have an example way down at the very > > end > > of this message. Now for reload there will be no error, as we handled the same in SET PERSISTENT command. at time of startup it will LOG the message. > > 6) Putting spaces into memory units results in broken config files: > > > > $ psql -c "set persistent work_mem='2 MB'" > > SET > > $ cat $PGDATA/config/postgresql.auto.conf > > work_mem = 2 MB > > > > This is wrong and you'll get this on reload: > > > > LOG: syntax error in file > > "/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf" line > > 1, > > near token "MB" > > LOG: configuration file > > "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors; > > no > > changes were applied > > > > It needs to look like this: > > > > work_mem = '2 MB' > > > > A regression test for some cases with spaces in the middle should be > > added too. This case is really weird, due to how the code is reading > > the existing postgresql.auto.conf file. If I first manually fix the > > file, and then run a change again, it stays fixed! Look at this > weird > > sequence: > > > > $ psql -c "set persistent work_mem='2 MB'" > > SET > > $ cat $PGDATA/config/postgresql.auto.conf > > work_mem = 2 MB > > [Here this setting is broken] > > $ vi $PGDATA/config/postgresql.auto.conf > > [Add quotes around the value, now it works] > > $ cat $PGDATA/config/postgresql.auto.conf > > work_mem = '2 MB' > > $ pg_ctl reload > > server signaled > > $ psql -c "set persistent work_mem='4 MB'" > > SET > > $ cat $PGDATA/config/postgresql.auto.conf > > work_mem = '4 MB' > > I shall fix this behavior. Fixed. Now all values will be in quotes, as at that point it is difficult to distinguish as per current implementation. > > > > 7) If I run SET PERSISTENT a lot concurrently, something happens that > > slows down connecting to the server. Restarting the server makes it > go > > away. I have a pgbench test case demonstrating the problem below, in > > the "Concurrency" section. I haven't tried to replicate it on > another > > system yet. > > > > = Tested features that work fine = > > > > Entries added here are tracked as you'd expect: > > > > $ psql -c "set persistent checkpoint_segments='32'" > > $ pg_ctl reload > > $ psql -xc "select name,setting,sourcefile,sourceline from > pg_settings > > where name='checkpoint_segments'" > > name | checkpoint_segments > > setting | 32 > > sourcefile | > > /Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf > > sourceline | 1 > > > > When the postgresql.auto.conf file is missing and you use SET > > PERSISTENT, it quietly creates it when writing out the new setting. > > > > = Concurrency and performance slowdown = > > > > I made two pgbench scripts that adjust a guc, one user and the other > > sighup, to a random value: > > > > random-join-limit.sql: > > > > \setrandom limit 1 8 > > set persistent join_collapse_limit=:limit; > > select pg_reload_conf(); > > > > random-segments.sql: > > > > \setrandom segments 3 16 > > set persistent checkpoint_segments=:segments; > > select pg_reload_conf(); > > > > I then fired off a bunch of these in parallel: > > > > pgbench -n -f random-join-limit.sql -f random-segments.sql -c 8 -T 60 > > > > This ran at 1547 TPS on my 8 core Mac, so that's not bad. No > > assertions > > and the config file was still valid at the end, which is a good sign > > the > > locking method isn't allowing utter chaos. Without the > > pg_reload_conf() > > in the test files, it also completed. With the reload happening in > one > > file but not the other, things were also fine. > > > > However, one thing I saw is that the server got significantly slower > > the > > more I ran this test script. After a few minutes it was down to only > > 400 TPS. The delay shows up between when I run psql and when I get > the > > prompt back. Here's normal behavior: > > > > $ time psql -c "select 1" > > real 0m0.006s > > > > And here's what I get after a single run of this pgbench hammering: > > > > $ time psql -c "select 1" > > real 0m0.800s > > > > 800ms? The slowdown is all for psql to start and connect, it's not > in > > the executor: I suspect this is due to pg_reload_conf(). I am looking into it. However could you please try once without pg_reload_conf(). > > = Error messages = > > > > If you remove postgresql.auto.conf and restart the server, it gives a > > warning that SET PERSISTENT won't work until you put it back. The > > error > > in this and several similar cases is pretty generic too: > > > > WARNING: File "postgresql.auto.conf" is not accessible, either file > > "postgresql.auto.conf" or folder "config" doesn't exist. or default > > "config" is not present in postgresql.conf. > > > > It would be nice if the error were more specific, individually > > identifying which of these is the actual problem. I can rewrite that > > long text entry to be more readable, but I think it should be a > series > > of smaller error checks with their own individual messages instead. > > You are right, we should change this by identifying actual problems. > > > If you remove postgresql.auto.conf then exeute "pg_ctl reload", it > > gives > > this error 6 times, which seems excessive. Reducing how often it > > appears in the reload case would be nice. > > > Deleting the whole config directory gives this: > > > > LOG: could not open configuration directory > > "/Users/gsmith/pgwork/data/autoconf/config": No such file or > directory > > LOG: configuration file > > "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors; > > no > > changes were applied > > > > If you now try to use the feature, the error message could be better. > > > > $ psql -c "set persistent checkpoint_segments='32'" > > ERROR: Failed to open auto conf temp file > > > "/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf.temp": > > No such file or directory > > > > It would be nice to complain about the config directory being > missing, > > as a first check before the file is opened. Restarting the server in > > this situation throws the correct error in your face though: > > > > LOG: could not open configuration directory > > "/Users/gsmith/pgwork/data/autoconf/config": No such file or > directory > > FATAL: configuration file > > "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors > > > > If you render this whole system unavailable by removing the > > "include_dir > > config", at every server start you'll see this: > > > > WARNING: File "postgresql.auto.conf" is not accessible, either file > > "postgresql.auto.conf" or folder "config" doesn't exist. or default > > "config" is not present in postgresql.conf. > > Configuration parameters changed before start/reload of server > > with SET > > PERSISTENT command will not be effective. > > LOG: database system was shut down at 2013-03-09 23:55:03 EST > > > > This is a debatable design choice. Some people might not want the > file > > and remove it, but don't want to be nagged about it. If people want > to > > wipe out the file or directory and work the old way, without this > > feature available, that's fine and they can. To me, helping users > who > > accidentally break this is more important than reducing harmless > > warning > > messages for things people did intentionally. WARNING might not be > the > > right level for this though. The existing checks like this I showed > > above use LOG for this sort of thing. > > > > The bigger problem is that this message shows up whenever you reload > > the > > config too. Watch this bizarre sequence: > > > > gsmith=# select pg_reload_conf(); > > pg_reload_conf > > ---------------- > > t > > (1 row) > > > > gsmith=# > > gsmith=# select 1; > > WARNING: File "postgresql.auto.conf" is not accessible, either file > > "postgresql.auto.conf" or folder "config" doesn't exist. or default > > "config" is not present in postgresql.conf. > > Configuration parameters changed before start/reload of server with > SET > > PERSISTENT command will not be effective. > > ?column? > > ---------- > > 1 > > > > And as I commented above, shifting more of the checks to SET > PERISTENT > > time could eliminate this check from running at server start and > reload > > altogether. I would be fine with them *also* happening at server > > start. > > But I could understand that other people might not like that. And > > having this pop up on every reload, appearing to a client next to > > another statement altogether, that isn't acceptable though. > > Agreed, I will add the check at time of SET PERSISTENT command. > Also I will try to remove this message at time of reload. Done as explained above. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
2013-03-13 21:28 keltezéssel, Boszormenyi Zoltan írta: > 2013-03-13 13:45 keltezéssel, Andres Freund írta: >> On 2013-03-13 09:09:24 +0100, Boszormenyi Zoltan wrote: >>> 2013-03-13 07:42 keltezéssel, Craig Ringer írta: >>>> On 03/12/2013 06:27 AM, Craig Ringer wrote: >>>>>>> Think also about the case where someone wants to change multiple >>>>>>> values together and having just some set and not others would be >>>>>>> inconsistent. >>>>> Yeah, that's a killer. The reload would need to be scheduled for COMMIT >>>>> time, it can't be done by `SET PERSISTENT` directly. >>>> Thinking about this some more, I'm not sure this is a good idea. >>>> >>>> Right now, SET takes effect immediately. Always, without exception. >>>> Delaying SET PERSISTENT's effects until commit would make it >>>> inconsistent with SET's normal behaviour. >>>> >>>> However, *not* delaying it would make it another quirky >>>> not-transactional not-atomic command. That's OK, but if it's not going >>>> to follow transactional semantics it should not be allowed to run within >>>> a transaction, like VACUUM . >>>> >>>> Writing the changes immediately but deferring the reload until commit >>>> seems to be the worst of those two worlds. >>> I was thinking about it a little. There is a hook that runs at the end >>> of (sub-)transactions. It can be abused for this purpose to make >>> SET PERSISTENT transactional. The subtransactions can also stack >>> these settings, by forgetting settings upon ROLLBACK [ TO SAVEPOINT ] >>> and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT. >>> All it needs is a list and maintaining intermediate pointers when entering >>> into a new level of SAVEPOINT. The functions to register such hooks are >>> in src/include/access/xact.h: >>> >>> extern void RegisterXactCallback(XactCallback callback, void *arg); >>> extern void UnregisterXactCallback(XactCallback callback, void *arg); >>> extern void RegisterSubXactCallback(SubXactCallback callback, void *arg); >>> extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg); >> (sub)xact commit/abort already calls AtEOXact_GUC(commit, nestlevel). So you >> wouldn't even need that. > > Yes, thanks. > >> It seems we could add another value to enum >> GucStackState, like GUC_SET_PERSISTENT - and process those only if commit && >> nestlevel == 1. > > Maybe it's not needed, only enum GucAction needs a new > GUC_ACTION_PERSISTENT value since that's what has any > business in push_old_value(). Adding two new members to > GucStack like these are enough > bool has_persistent; > config_var_value persistent; > and SET PERSISTENT can be treated as GUC_SET. push_old_value() > can merge GUC values set in the same transaction level. It seems both were needed. The attached patch makes SET PERSISTENT transactional and uses one setting per file. It uses the currently existing parsing and validating code and because of this, the patch is half the size of v12 from Amit. Best regards, Zoltán Böszörményi > >> Everytime you see one with commit && nestlevel > 1 you put them into them into >> the stack one level up. >> >> This seems like its somewhat in line with the way SET LOCAL is implemented? >> >>> On the other hand, it would make a lot of sense to implement it >>> as one setting per file or extending the code to allow modifying >>> settings in bulk. The one setting per file should be easier and >>> it would also allow extensions to drop their settings automatically >>> into the automatic config directory. I don't know who mentioned >>> this idea about extensions but it also came up a while back. >> I still think one-setting-per-file is the right way to go, yes. >> >> Greetings, >> >> Andres Freund >> > > -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
Attachment
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
<div class="moz-cite-prefix">2013-03-15 00:48 keltezéssel, Boszormenyi Zoltan írta:<br /></div><blockquote cite="mid:514261B4.6080104@cybertec.at"type="cite">2013-03-13 21:28 keltezéssel, Boszormenyi Zoltan írta: <br /><blockquotetype="cite">2013-03-13 13:45 keltezéssel, Andres Freund írta: <br /><blockquote type="cite">On 2013-03-13 09:09:24+0100, Boszormenyi Zoltan wrote: <br /><blockquote type="cite">2013-03-13 07:42 keltezéssel, Craig Ringer írta: <br/><blockquote type="cite">On 03/12/2013 06:27 AM, Craig Ringer wrote: <br /><blockquote type="cite"><blockquote type="cite"><blockquotetype="cite">Think also about the case where someone wants to change multiple <br /> values togetherand having just some set and not others would be <br /> inconsistent. <br /></blockquote></blockquote> Yeah, that'sa killer. The reload would need to be scheduled for COMMIT <br /> time, it can't be done by `SET PERSISTENT` directly.<br /></blockquote> Thinking about this some more, I'm not sure this is a good idea. <br /><br /> Right now, SETtakes effect immediately. Always, without exception. <br /> Delaying SET PERSISTENT's effects until commit would makeit <br /> inconsistent with SET's normal behaviour. <br /><br /> However, *not* delaying it would make it another quirky<br /> not-transactional not-atomic command. That's OK, but if it's not going <br /> to follow transactional semanticsit should not be allowed to run within <br /> a transaction, like VACUUM . <br /><br /> Writing the changes immediatelybut deferring the reload until commit <br /> seems to be the worst of those two worlds. <br /></blockquote> Iwas thinking about it a little. There is a hook that runs at the end <br /> of (sub-)transactions. It can be abused forthis purpose to make <br /> SET PERSISTENT transactional. The subtransactions can also stack <br /> these settings, byforgetting settings upon ROLLBACK [ TO SAVEPOINT ] <br /> and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT.<br /> All it needs is a list and maintaining intermediate pointers when entering <br /> into a new level of SAVEPOINT.The functions to register such hooks are <br /> in src/include/access/xact.h: <br /><br /> extern void RegisterXactCallback(XactCallbackcallback, void *arg); <br /> extern void UnregisterXactCallback(XactCallback callback, void*arg); <br /> extern void RegisterSubXactCallback(SubXactCallback callback, void *arg); <br /> extern void UnregisterSubXactCallback(SubXactCallbackcallback, void *arg); <br /></blockquote> (sub)xact commit/abort already calls AtEOXact_GUC(commit,nestlevel). So you <br /> wouldn't even need that. <br /></blockquote><br /> Yes, thanks. <br /><br /><blockquotetype="cite"> It seems we could add another value to enum <br /> GucStackState, like GUC_SET_PERSISTENT - andprocess those only if commit && <br /> nestlevel == 1. <br /></blockquote><br /> Maybe it's not needed, only enumGucAction needs a new <br /> GUC_ACTION_PERSISTENT value since that's what has any <br /> business in push_old_value().Adding two new members to <br /> GucStack like these are enough <br /> bool has_persistent; <br /> config_var_value persistent; <br /> and SET PERSISTENT can be treated as GUC_SET. push_old_value() <br /> can merge GUCvalues set in the same transaction level. <br /></blockquote><br /> It seems both were needed. The attached patch makes<br /> SET PERSISTENT transactional and uses one setting per file. <br /> It uses the currently existing parsing andvalidating code <br /> and because of this, the patch is half the size of v12 from Amit. <br /></blockquote><br /> Theonly missing piece is the check for superuser.<br /><br /><br /><blockquote cite="mid:514261B4.6080104@cybertec.at" type="cite"><br/> Best regards, <br /> Zoltán Böszörményi <br /><br /><blockquote type="cite"><br /><blockquote type="cite">Everytimeyou see one with commit && nestlevel > 1 you put them into them into <br /> the stack onelevel up. <br /><br /> This seems like its somewhat in line with the way SET LOCAL is implemented? <br /><br /><blockquotetype="cite">On the other hand, it would make a lot of sense to implement it <br /> as one setting per file orextending the code to allow modifying <br /> settings in bulk. The one setting per file should be easier and <br /> itwould also allow extensions to drop their settings automatically <br /> into the automatic config directory. I don't knowwho mentioned <br /> this idea about extensions but it also came up a while back. <br /></blockquote> I still think one-setting-per-fileis the right way to go, yes. <br /><br /> Greetings, <br /><br /> Andres Freund <br /><br /></blockquote><br/><br /></blockquote><br /><br /><br /><fieldset class="mimeAttachmentHeader"></fieldset><br /><pre wrap=""> </pre></blockquote><br /><br /><pre class="moz-signature" cols="90">-- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: <a class="moz-txt-link-freetext" href="http://www.postgresql-support.de">http://www.postgresql-support.de</a> <aclass="moz-txt-link-freetext" href="http://www.postgresql.at/">http://www.postgresql.at/</a> </pre>
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review] (Fix memory growth)
From
Amit Kapila
Date:
On Thursday, March 14, 2013 6:58 PM Amit Kapila wrote: > On Monday, March 11, 2013 12:18 PM Amit Kapila wrote: > > On Sunday, March 10, 2013 8:43 PM Greg Smith wrote: > > > On 3/7/13 2:42 AM, Amit Kapila wrote: > > > > I also think the tests added for regression may be more than > > > required... > > > > If you think above optimization's to reduce number of tests are > > okay, > > > then I > > > > will update the patch. > > > 7) If I run SET PERSISTENT a lot concurrently, something happens > > > that slows down connecting to the server. Restarting the server > > > makes it > > go > > > away. I have a pgbench test case demonstrating the problem below, > > > in the "Concurrency" section. I haven't tried to replicate it on > > another > > > system yet. > > > > > > = Tested features that work fine = > > > > > > Entries added here are tracked as you'd expect: > > > > > > $ psql -c "set persistent checkpoint_segments='32'" > > > $ pg_ctl reload > > > $ psql -xc "select name,setting,sourcefile,sourceline from > > pg_settings > > > where name='checkpoint_segments'" > > > name | checkpoint_segments > > > setting | 32 > > > sourcefile | > > > /Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf > > > sourceline | 1 > > > > > > When the postgresql.auto.conf file is missing and you use SET > > > PERSISTENT, it quietly creates it when writing out the new setting. > > > > > > = Concurrency and performance slowdown = > > > > > > I made two pgbench scripts that adjust a guc, one user and the > other > > > sighup, to a random value: > > > > > > random-join-limit.sql: > > > > > > \setrandom limit 1 8 > > > set persistent join_collapse_limit=:limit; select pg_reload_conf(); > > > > > > random-segments.sql: > > > > > > \setrandom segments 3 16 > > > set persistent checkpoint_segments=:segments; select > > > pg_reload_conf(); > > > > > > I then fired off a bunch of these in parallel: > > > > > > pgbench -n -f random-join-limit.sql -f random-segments.sql -c 8 -T > > > 60 > > > > > > This ran at 1547 TPS on my 8 core Mac, so that's not bad. No > > > assertions and the config file was still valid at the end, which is > > > a good sign the locking method isn't allowing utter chaos. Without > > > the > > > pg_reload_conf() > > > in the test files, it also completed. With the reload happening in > > one > > > file but not the other, things were also fine. > > > > > > However, one thing I saw is that the server got significantly > slower > > > the more I ran this test script. After a few minutes it was down > to > > > only 400 TPS. The delay shows up between when I run psql and when > I > > > get > > the > > > prompt back. Here's normal behavior: > > > > > > $ time psql -c "select 1" > > > real 0m0.006s > > > > > > And here's what I get after a single run of this pgbench hammering: > > > > > > $ time psql -c "select 1" > > > real 0m0.800s > > > > > > 800ms? The slowdown is all for psql to start and connect, it's not > > in > > > the executor: > > I suspect this is due to pg_reload_conf(). I am looking into it. > However could you please try once without pg_reload_conf(). The memory growth is because of pg_reload_conf(). There were 2 sources of memory growth: 1. Load_ident - Memory context was not getting deleted 2. ProcessConfigFile - Most of the memory for parsing and processing is allocated under TopMemoryContext and is not getting reset, which leads to this problem. For now I havefixed by doing this in separate context. The above 2 fixes are in attached patch 'ctx_growth_fix_v1'. Apart from this, I have fixed issues 1 and 2 reported by you in attached version 'set_persistent_v13', by sending signal at end of command and checking for parameters which cannot be loaded. With this, all the issues reported by you are addressed. Thanks to you for so careful test and review of this patch. With Regards, Amit Kapila.
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Greg Smith
Date:
On 3/14/13 4:48 PM, Boszormenyi Zoltan wrote: > The attached patch makes > SET PERSISTENT transactional and uses one setting per file. > It uses the currently existing parsing and validating code > and because of this, the patch is half the size of v12 from Amit. That's not a completely fair comparison, because you lost all the regression testing code too. This does look like a usefully tighter implementation in a few ways, so good progress on that. I still can't see any reason to prefer this "one setting per file" idea. As I see it, that is pushing the complexity towardthe user in a bad way, seemingly just so it's easier to implement. Most of my customers now use tools like Puppet to manage their PostgreSQL configuration. I do not want to have this conversation: Greg: "You can use SET PERSISTENT to modify settings instead of changing the postgresql.conf" User: "That's useful. How do we adjust Puppet to make sure it picks up the changes?" Greg: "You just scan this config directory and add every file that appears in there! Each setting will be in its own file." User: "<shocked look> It creates new files? That isn't going to fit into our version control approach easily. Can we disable this feature so no one accidentally uses it?" I'm not exaggerating here--"one setting per file" makes this feature less than useless to me. It becomes something where I will have to waste time defending against people using it. I'd prefer to not have this at all than to do it that way. That we're breaking these settings off into their own file, instead of trying to edit the postgresql.conf, to me is a pragmatic trade-off to keep the implementation from being really complicated. It's also a step forward in a larger series for how to improve configuration management. Just because that change introduces an entire directorybeing scanned, I don't see that as an excuse to clutter it with a long list of files too. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Boszormenyi Zoltan
Date:
2013-03-17 17:05 keltezéssel, Greg Smith írta: > On 3/14/13 4:48 PM, Boszormenyi Zoltan wrote: >> The attached patch makes >> SET PERSISTENT transactional and uses one setting per file. >> It uses the currently existing parsing and validating code >> and because of this, the patch is half the size of v12 from Amit. > > That's not a completely fair comparison, because you lost all the regression testing > code too. True. > This does look like a usefully tighter implementation in a few ways, so good progress on > that. > > I still can't see any reason to prefer this "one setting per file" idea. As I see it, > that is pushing the complexity toward the user in a bad way, seemingly just so it's > easier to implement. Most of my customers now use tools like Puppet to manage their > PostgreSQL configuration. I do not want to have this conversation: > > Greg: "You can use SET PERSISTENT to modify settings instead of changing the > postgresql.conf" > > User: "That's useful. How do we adjust Puppet to make sure it picks up the changes?" > > Greg: "You just scan this config directory and add every file that appears in there! > Each setting will be in its own file." > > User: "<shocked look> It creates new files? That isn't going to fit into our version > control approach easily. Can we disable this feature so no one accidentally uses it?" > > I'm not exaggerating here--"one setting per file" makes this feature less than useless > to me. It becomes something where I will have to waste time defending against people > using it. I'd prefer to not have this at all than to do it that way. > > That we're breaking these settings off into their own file, instead of trying to edit > the postgresql.conf, to me is a pragmatic trade-off to keep the implementation from > being really complicated. It's also a step forward in a larger series for how to > improve configuration management. Just because that change introduces an entire > directory being scanned, I don't see that as an excuse to clutter it with a long list of > files too. OK. I just wanted to show an alternative implementation. I admit, I haven't read all mails from this thread so I don't know how important for this feature to be non-transactional, or whether it would be better to be transactional. The one-file-to-rule-them-all approach can also be added to this patch as well, but synchronizing transactions over parsing and rewriting the extra file would decrease the relaxed effects of synchronous_commit. On the other hand, writing out one file per setting, although atomic to the level of one file, cannot be guaranteed to be atomic as a whole for all settings changed in the transaction without some synchronization. As far as I can see, AtEOXact_GUC() is called outside the critical section and is not synchronized any other way. (Currently, there's no need to.) Best regards, Zoltán Böszörményi -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
On Tue, Mar 12, 2013 at 11:36 AM, Greg Stark <stark@mit.edu> wrote: > On Tue, Mar 12, 2013 at 9:06 AM, Greg Smith <greg@2ndquadrant.com> wrote: >> That's jumping right over a few rounds of simpler ways to do this, and just >> going right to the approach we know allows adding more such options later >> with minimal grammar impact. > > As Craig intimated, the minimal grammar impact would be simply > > BEGIN; > set persistent maintenance_work_mem='2GB'; > set persistent work_mem='2GB'; > COMMIT; > > Sending the sighup at transaction end seems like a fairly safe thing > to do too. It's hard to imagine it failing and if it did the worst > case would be that other backends would still have the old values too. I'm late to the party here, but my feeling is that we ought NOT to send a SIGHUP automatically. For various reasons already discussed, that's not going to be completely transparent to the DBA. And when things are not fully transparent, IME, it's better not to do them automatically. DBAs like automation when it's 100% reliable - but they don't like the database to automatically do things that can have unforeseen side effects. Also, while I think that MOST people will probably want a SIGHUP right after SET PERSISTENT, I am not sure that EVERYONE will want that. If you want it and it doesn't happen automatically, you can always do it by hand. If you don't want it and it does happen automatically, you're out of luck: go back to hand-editing config files. I really don't want to leave people with any more reasons to continue hand-editing the config files than we truly must. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 3/21/13 2:38 PM, Robert Haas wrote: > Also, while I think that MOST people will probably want a SIGHUP right > after SET PERSISTENT, I am not sure that EVERYONE will want that. If > you want it and it doesn't happen automatically, you can always do it > by hand. This is a fair position, and since that's how the feature as written right now works that helps. I think proceeding this way needs to hand some sort of hint back to the user though, telling them the change isn't active until SIGHUP. The path I don't want to see if where someone uses SET PERSISTENT and can't figure out why nothing changed. It should be as obvious as we can make it to someone that the explicit reload is necessary. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Alvaro Herrera
Date:
Greg Smith escribió: > On 3/21/13 2:38 PM, Robert Haas wrote: > >Also, while I think that MOST people will probably want a SIGHUP right > >after SET PERSISTENT, I am not sure that EVERYONE will want that. If > >you want it and it doesn't happen automatically, you can always do it > >by hand. > > This is a fair position, and since that's how the feature as written > right now works that helps. I think proceeding this way needs to > hand some sort of hint back to the user though, telling them the > change isn't active until SIGHUP. The path I don't want to see if > where someone uses SET PERSISTENT and can't figure out why nothing > changed. It should be as obvious as we can make it to someone that > the explicit reload is necessary. Maybe add some syntax to prevent the SIGHUP for the rare case where that is wanted, say SET PERSISTENT (reload=off) var=val; (perhaps WITH at the end, dunno) -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Friday, March 22, 2013 12:48 AM Alvaro Herrera wrote: > Greg Smith escribió: > > On 3/21/13 2:38 PM, Robert Haas wrote: > > >Also, while I think that MOST people will probably want a SIGHUP > right > > >after SET PERSISTENT, I am not sure that EVERYONE will want that. > If > > >you want it and it doesn't happen automatically, you can always do > it > > >by hand. > > > > This is a fair position, and since that's how the feature as written > > right now works that helps. I think proceeding this way needs to > > hand some sort of hint back to the user though, telling them the > > change isn't active until SIGHUP. The path I don't want to see if > > where someone uses SET PERSISTENT and can't figure out why nothing > > changed. It should be as obvious as we can make it to someone that > > the explicit reload is necessary. > > Maybe add some syntax to prevent the SIGHUP for the rare case where > that > is wanted, say > > SET PERSISTENT (reload=off) var=val; > > (perhaps WITH at the end, dunno) I think adding new syntax change is little scary for me, not for the matter of implementation but for building consensus on syntax. Can we do it as an enhancement later? Other ways could be to either give Notice after each command or mention the same in documentation clearly. I feel giving Notice after every command doesn't look good, so may be we can mention the same in documentation. With Regards, Amit Kapila.
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Alvaro Herrera
Date:
Amit Kapila escribió: > On Friday, March 22, 2013 12:48 AM Alvaro Herrera wrote: > > Maybe add some syntax to prevent the SIGHUP for the rare case where > > that > > is wanted, say > > > > SET PERSISTENT (reload=off) var=val; > > > > (perhaps WITH at the end, dunno) > > I think adding new syntax change is little scary for me, not for the matter > of implementation but for building consensus on syntax. I cannot but agree on that point. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On 3/21/13 10:39 PM, Amit Kapila wrote: > I think adding new syntax change is little scary for me, not for the matter > of implementation but for building consensus on syntax. > Can we do it as an enhancement later? Yeah, I think the basics of this could be committed, but have later improvements that let you fine-tune when the reload happens better. > Other ways could be to either give Notice after each command or mention the > same in documentation clearly. > I feel giving Notice after every command doesn't look good, so may be we can > mention the same in documentation. I think that NOTICE after every command is the only way we'll make sure to catch every user who should be notified about the feature's limitation. I was thinking of something like this: NOTICE: New configuration is written but not applied yet. Signal the server to reload settings using pg_reload_conf to activate the change. Maybe that's a NOTICE plus a HINT. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
On Friday, March 22, 2013 8:57 AM Alvaro Herrera wrote: > Amit Kapila escribió: > > On Friday, March 22, 2013 12:48 AM Alvaro Herrera wrote: > > > > Maybe add some syntax to prevent the SIGHUP for the rare case where > > > that > > > is wanted, say > > > > > > SET PERSISTENT (reload=off) var=val; > > > > > > (perhaps WITH at the end, dunno) > > > > I think adding new syntax change is little scary for me, not for the > matter > > of implementation but for building consensus on syntax. > > I cannot but agree on that point. Sorry, I don't get your point. Do you mean to say that you don't agree with me and want new syntax as proposed by you to be implemented? With Regards, Amit Kapila.
Greg Smith <greg@2ndQuadrant.com> writes:
> On 3/21/13 10:39 PM, Amit Kapila wrote:
>> I feel giving Notice after every command doesn't look good, so may be we can
>> mention the same in documentation.
> I think that NOTICE after every command is the only way we'll make sure
> to catch every user who should be notified about the feature's limitation.
We have been around on that type of proposal before. A command that
issues a NOTICE as part of its *standard* behavior is really not going
to fly; the annoyance, talking-down-to-the-user factor is too high.
IOW, if you think the command needs that, then its design is broken
and you need to do better.
regards, tom lane
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Alvaro Herrera
Date:
Amit Kapila escribió: > On Friday, March 22, 2013 8:57 AM Alvaro Herrera wrote: > > Amit Kapila escribió: > > > I think adding new syntax change is little scary for me, not for > > > the matter of implementation but for building consensus on syntax. > > > > I cannot but agree on that point. > > Sorry, I don't get your point. > Do you mean to say that you don't agree with me and want new syntax as > proposed by you to be implemented? On the contrary, I was saying I agree with you on the difficulty on getting consensus on this. I don't know where I learned this phrase or even if it's in common usage. After a not-so-quick search I see it explained here: http://www.perfectyourenglish.com/usage/but.htm -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Friday, March 22, 2013 7:33 PM Alvaro Herrera wrote: > Amit Kapila escribió: > > On Friday, March 22, 2013 8:57 AM Alvaro Herrera wrote: > > > Amit Kapila escribió: > > > > > I think adding new syntax change is little scary for me, not for > > > > the matter of implementation but for building consensus on > syntax. > > > > > > I cannot but agree on that point. > > > > Sorry, I don't get your point. > > Do you mean to say that you don't agree with me and want new syntax > as > > proposed by you to be implemented? > > On the contrary, I was saying I agree with you on the difficulty on > getting consensus on this. Now based on final discussion, I will go-ahead and document the behavior rather than using SIGHUP at end or giving NOTICE at end of command. Let me know if anybody still feels otherwise. Apart from this during Greg's testing, he found a performance problem with running pg_reload_conf() along with my patch the reason for which actually turns out to be a memory growth. I have attempted to fix it in the patch ctx_growth_fix_v1.patch which is currently attached in CF queue along with my Patch. http://www.postgresql.org/message-id/004801ce216b$e37c3b30$aa74b190$@kapila@ huawei.com I think that should be done irrespective of SET Persistent Patch. > I don't know where I learned this phrase or even if it's in common > usage. After a not-so-quick search I see it explained here: > http://www.perfectyourenglish.com/usage/but.htm You were right, I was not able to understand. With Regards, Amit Kapila.
On Saturday, March 23, 2013 7:21 AM Amit Kapila wrote: > On Friday, March 22, 2013 7:33 PM Alvaro Herrera wrote: > > Amit Kapila escribió: > > > On Friday, March 22, 2013 8:57 AM Alvaro Herrera wrote: > > > > Amit Kapila escribió: > > > > > > > I think adding new syntax change is little scary for me, not > for > > > > > the matter of implementation but for building consensus on > > syntax. > > > > > > > > I cannot but agree on that point. > > > > > > Sorry, I don't get your point. > > > Do you mean to say that you don't agree with me and want new syntax > > as > > > proposed by you to be implemented? > > > > On the contrary, I was saying I agree with you on the difficulty on > > getting consensus on this. > > Now based on final discussion, I will go-ahead and document the > behavior > rather than using SIGHUP at end or giving NOTICE at end of command. > Let me know if anybody still feels otherwise. Updated the patch to remove SIGHUP at end of command and updated docs accordingly. > Apart from this during Greg's testing, he found a performance problem > with > running pg_reload_conf() along with my patch the reason for which > actually > turns out to be a memory > growth. I have attempted to fix it in the patch ctx_growth_fix_v1.patch > which is currently attached in CF queue along with my Patch. > http://www.postgresql.org/message- > id/004801ce216b$e37c3b30$aa74b190$@kapila@ > huawei.com > I think that should be done irrespective of SET Persistent Patch. With Regards, Amit Kapila.
At this point SET PERSISTENT is looking complete enough that some parts of it might be committed. There's been a lot of useful progress in nailing down the early/obvious problems and what fundamental approach makes sense. Accepting the whole thing still seems a bit too invasive to chew on this late in 9.3 though. Unfortunately it's not until everything goes in that a really useful change happens. What I'll mainly talk about here is how to break this into smaller pieces to consider as potential future commits, hopeful ones that might all go in early for 9.4. There's no identified committer for this feature yet. I think which committer (or committers, plural, given the number of pieces) is willing to take on the GUC changes, and how aggressively they want to consider the pieces of this, is going to determine the timeline for when the moving parts of this idea are adopted. As for the code that seems high risk to me, there's a whole class of subtle problems possible in the memory management in particular. The first stress test I did (just to try and break the feature) shook out what looks so far like a memory allocation concern in the existing codebase, before this new feature is even considered. That didn't feel like a good sign to me. Stepping back, I see four potential commits worth of changes here now, and I would consider them in this order: 1) Fix for pg_reload_conf memory context growth problem uncovered during testing. This is a small patch and minor bug fix that could even go into 9.3. The problem is not so severe it seems worth the disruption of backporting to earlier versions, and it wouldn't really hurt much to kick the problem forward to 9.4. 2) Change the default postgresql.conf to include a config directory. This has been one of my soapbox positions for a while now, and I think this feature is working well enough now to have proven the form of re-arrangement does something useful. This is a small change to the PostgreSQL code that will ripple out to impact packaging. It would be possible to do this part and face the main disruption for 9.3 even if the exact SET PERSISTENT implementation is pushed forward. If there was not much else going on with packaging right now, I might even advocate that myself. Unfortunately, right now "not much else going on" is the exact opposite of the situation all the packagers are in. It's just a bad time to talk about it. 3) Rearrangement of GUC validation into validate_conf_option function. This is ~400 lines of code that could be changed as a zero-impact refactoring, one that just makes the SET PERSISTENT option easier to implement. 4) Implementation of SET PERSISTENT into a single file, config/persistent.auto.conf. As written, a reasonable amount of error checking is done to reduce the odds of someone breaking its implementation without realizing it. The responsibility of activating the new configuration using pg_reload_conf is pushed toward the user, documented but without an explicit warning. All of these changes have different risk vs. reward trade-offs in them. I'd guess the validation_config_option change (3)is the riskiest of this bunch, because it could easily break the standard GUC path in a major way even for people who don't use the feature. It's not that much code, but it is going to take a good bit of committer level review to accept due to its risk. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
On Monday, April 01, 2013 1:03 PM Greg Smith wrote: > At this point SET PERSISTENT is looking complete enough that some parts > of it might be committed. There's been a lot of useful progress in > nailing down the early/obvious problems and what fundamental approach > makes sense. Accepting the whole thing still seems a bit too invasive > to chew on this late in 9.3 though. Unfortunately it's not until > everything goes in that a really useful change happens. > > What I'll mainly talk about here is how to break this into smaller > pieces to consider as potential future commits, hopeful ones that might > all go in early for 9.4. There's no identified committer for this > feature yet. I think which committer (or committers, plural, given the > number of pieces) is willing to take on the GUC changes, and how > aggressively they want to consider the pieces of this, is going to > determine the timeline for when the moving parts of this idea are > adopted. > > As for the code that seems high risk to me, there's a whole class of > subtle problems possible in the memory management in particular. The > first stress test I did (just to try and break the feature) shook out > what looks so far like a memory allocation concern in the existing > codebase, before this new feature is even considered. That didn't feel > like a good sign to me. > > Stepping back, I see four potential commits worth of changes here now, > and I would consider them in this order: > > 1) Fix for pg_reload_conf memory context growth problem uncovered > during > testing. This is a small patch and minor bug fix that could even go > into 9.3. The problem is not so severe it seems worth the disruption > of > backporting to earlier versions, and it wouldn't really hurt much to > kick the problem forward to 9.4. > > 2) Change the default postgresql.conf to include a config directory. > This has been one of my soapbox positions for a while now, and I think > this feature is working well enough now to have proven the form of > re-arrangement does something useful. This is a small change to the > PostgreSQL code that will ripple out to impact packaging. It would be > possible to do this part and face the main disruption for 9.3 even if > the exact SET PERSISTENT implementation is pushed forward. If there > was > not much else going on with packaging right now, I might even advocate > that myself. Unfortunately, right now "not much else going on" is the > exact opposite of the situation all the packagers are in. It's just a > bad time to talk about it. > > 3) Rearrangement of GUC validation into validate_conf_option function. > This is ~400 lines of code that could be changed as a zero-impact > refactoring, one that just makes the SET PERSISTENT option easier to > implement. > > 4) Implementation of SET PERSISTENT into a single file, > config/persistent.auto.conf. As written, a reasonable amount of error > checking is done to reduce the odds of someone breaking its > implementation without realizing it. The responsibility of activating > the new configuration using pg_reload_conf is pushed toward the user, > documented but without an explicit warning. > > All of these changes have different risk vs. reward trade-offs in them. > I'd guess the validation_config_option change (3) is the riskiest of > this bunch, because it could easily break the standard GUC path in a > major way even for people who don't use the feature. It's not that > much > code, but it is going to take a good bit of committer level review to > accept due to its risk. I think in that case we can have 3 separate patches 1. Memory growth defect fix 2. Default postgresql.conf to include config directory and SET Persistent into single file implementation 3. Rearrangement of GUC validation into validate_conf_option function. As already there is review happened for point 2 and point 1 is an existing code defect fix, so in my opinion Patch 1 and 2 should be considered for 9.3. With Regards, Amit Kapila.
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Peter Eisentraut
Date:
I'm going to ignore most of the discussion that led up to this and give
this patch a fresh look.
+ <screen>
+ SET PERSISTENT max_connections To 10;
+ </screen>
The To should probably be capitalized.
I doubt this example actually works because changing max_connections
requires a restart. Try to find a better example.
It's weird that SET LOCAL and SET SESSION actually *set* the value, and
the second key word determines how long the setting will last. SET
PERSISTENT doesn't actually set the value. I predict that this will be
a new favorite help-it-doesn't-work FAQ.
<varlistentry> <term><literal>SCHEMA</literal></term> <listitem>
! <para><literal>SET [ PERSISTENT ] SCHEMA
'<replaceable>value</>'</> is an alias for <literal>SET search_path TO <replaceable>value</></>. Only one
schema can be specified using this syntax. </para>
I don't think [ PERSISTENT ] needs to be added in these and similar
snippets. We don't mention LOCAL etc. here either.
--- 34,41 ---- <filename>postgresql.conf</filename>,
<filename>pg_hba.conf</filename>, and <filename>pg_ident.conf</filename> are traditionally stored in <varname>PGDATA</>
(althoughin <productname>PostgreSQL</productname>
8.0 and
! later, it is possible to place them elsewhere). By default the
directory config is stored
! in <varname>PGDATA</>, however it should be kept along with
<filename>postgresql.conf</filename>. </para>
<table tocentry="1" id="pgdata-contents-table">
This chunk doesn't make any sense to me. "Should" is always tricky.
Why should I, and why might I not?
<row>
+ <entry><filename>config</></entry>
+ <entry>Subdirectory containing automatically generated configuration
files</entry>
+ </row>
+
+ <row> <entry><filename>base</></entry> <entry>Subdirectory containing per-database subdirectories</entry> </row>
Only automatically generated ones?
COPY_STRING_FIELD(name); COPY_NODE_FIELD(args); COPY_SCALAR_FIELD(is_local);
+ COPY_SCALAR_FIELD(is_persistent);
return newnode; }
I suggest changing is_local into a new trivalued field that stores LOCAL
or SESSION or PERSISTENT.
n->is_local = false; $$ = (Node *) n;
}
+ | SET PERSISTENT set_persistent
+ {
+ VariableSetStmt *n = $3;
+ n->is_persistent = true;
+ $$ = (Node *) n;
+ } ;
set_rest:
Why can't you use SET PERSISTENT set_rest?
*** a/src/backend/replication/basebackup.c
--- b/src/backend/replication/basebackup.c
***************
*** 755,760 **** sendDir(char *path, int basepathlen, bool sizeonly)
--- 755,766 ---- strlen(PG_TEMP_FILE_PREFIX)) == 0)
continue;
+ /* skip auto conf temporary file */
+ if (strncmp(de->d_name,
+ PG_AUTOCONF_FILENAME ".",
+ sizeof(PG_AUTOCONF_FILENAME)) == 0)
+ continue;
+
Maybe pg_basebackup should be taught to ignore certain kinds of
temporary files in general. The file name shouldn't be hardcoded into
pg_basebackup. This would effectively make the configuration file
naming scheme part of the replication protocol. See other thread about
pg_basebackup client/server compatibility. This needs to be generalized.
(Side thought: Does pg_basebackup copy editor temp and backup files?)
+ ereport(elevel,
+ (errmsg("Configuration parameters changed with SET
PERSISTENT command before start of server "
+ "will not be effective as \"%s\" file is not
accessible.", PG_AUTOCONF_FILENAME)));
I'm having trouble interpreting this: How can you change something with
SET PERSISTENT before the server starts?
Also: errmsg messages should start with a lowercase letter and should
generally be much shorter. Please review other cases in your patch as well.
+ appendStringInfoString(&buf, "# Do not edit this file manually! "
+ "It is overwritten by the SET PERSISTENT command
\n");
There is some punctuation or something missing at the end.
I suggest you break this into two lines. It's pretty long.
I think the naming of these files is suboptimal:
+ #define PG_AUTOCONF_DIR "config"
"config" would seem to include pg_hba.conf and others. Or
postgresql.conf for that matter. Maybe I should move postgresql.conf
into config/. Another great new FAQ. The normal convention for this is
"postgresql.conf.d". I don't see any reason not to use that. One
reason that was brought up is that this doesn't match other things
currently in PGDATA, but (a) actually it does (hint: postgresql.conf),
and (b) neither does "config".
+ #define PG_AUTOCONF_FILENAME "persistent.auto.conf"
"auto" sounds like the result of an automatic tuning tool. Why not just
persistent.conf. Or set-persistent.conf. That makes the association
clear enough.
I don't know why the concept of a configuration directory is being
introduced here. I mean, I'm all for that, but it seems completely
independent of this. Some people wanted one setting per file, but
that's not what is happening here. The name of the configuration file
is hardcoded throughout, so there is no value in reading all files from
a directory instead of just that one file.
There is a lot of confusion throughout the patch about whether this
config directory is for automatic files only or you can throw your own
stuff in there at will. Another great FAQ. And then the whole dance
about parsing postgresql.conf to make sure the include_dir is not
removed (=> FAQ). Does it care whether it's at the beginning or the
end? Should it?
How does all of this work when you move the configuration files to, say,
/etc/, but want to keep the SET PERSISTENT stuff under $PGDATA? Lots of
moving parts there.
I think this could be much easier if you just read
$PGDATA/persistent.conf or whatever if present. There is perhaps an
analogy to be found with the proposed new recovery.conf handling (e.g.,
always read if present after all other postgresql.conf processing or
something like that).
I'm thinking, this feature is ultimately intended to make things simpler
for average users. But I see so many traps here. It needs to be
simpler yet, and have less moving parts.
I'd almost like an option to turn this off for a server, because it has
so much potential for confusion.
On 4/1/13 5:44 AM, Amit Kapila wrote: > I think in that case we can have 3 separate patches > 1. Memory growth defect fix > 2. Default postgresql.conf to include config directory and SET Persistent > into single file implementation > 3. Rearrangement of GUC validation into validate_conf_option function. > > As already there is review happened for point 2 and point 1 is an existing > code defect fix, so in my opinion > Patch 1 and 2 should be considered for 9.3. They have been considered for 9.3, I just doubt they could get committed right now. In order for this to go in as part of the very last 9.3 feature changes (which is where we're at in the development cycle), you'd need to have a committer volunteer to take on the job of doing a second level of review on it. And then that would have to happen without any other issues popping up. That usually is not how it works. Normally the first round of committer review findsanother round of issues, and there's at least one more update before commit. (Note that this is exactly what just happened today, with the review from Peter Eisentraut) I'm not saying it's impossible for this feature to go in to 9.3, but I'm not seeing any committers volunteer to take on the job either. I do want to see this feature go in--I'll update it for 9.4 even if you don't--but we're already into April. There isn't much time left for 9.3. And the security release this week has gobbled up a good chunk of committer and packager time unexpectedly, which is just bad luck for your submission. From a process perspective, features that enter the last CF of a release that are very close to ready from the start have good odds of being committed. You've done an excellent job of updating this in response to feedback, but it has involved a long list of changes so far. It's fair to say this was still a rough featureat the start of CF 2013-01, and now it's good but can be usefully polished a bit more. For something of this size, going from rough feature to commit quality normally takes more than one CF. I don't have any obvious errors to point out right now. But I think there's still some room to improve on this before commit. Andres mentioned on another thread that he thought merging some of your ideas with the version Zoltan did was useful to look at, and I was thinking of something similar. This is close to being ready, and I hope you won't get discouraged just because it's probably going to slip to 9.4. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
> I'm going to ignore most of the discussion that led up to this and give
> this patch a fresh look.
Thank you.
> + <screen>
> + SET PERSISTENT max_connections To 10;
> + </screen>
>
> The To should probably be capitalized.
Okay, shall fix it.
> I doubt this example actually works because changing max_connections
> requires a restart. Try to find a better example.
Any parameter's changed with this command can be effective either after
restart or SIGHUP.
So I will change it to SIGHUP parameter.
> It's weird that SET LOCAL and SET SESSION actually *set* the value, and
> the second key word determines how long the setting will last. SET
> PERSISTENT doesn't actually set the value. I predict that this will be
> a new favorite help-it-doesn't-work FAQ.
>
> <varlistentry>
> <term><literal>SCHEMA</literal></term>
> <listitem>
> ! <para><literal>SET [ PERSISTENT ] SCHEMA
> '<replaceable>value</>'</> is an alias for
> <literal>SET search_path TO <replaceable>value</></>. Only
> one
> schema can be specified using this syntax.
> </para>
>
> I don't think [ PERSISTENT ] needs to be added in these and similar
> snippets. We don't mention LOCAL etc. here either.
Agreed, shall fix this.
> --- 34,41 ----
> <filename>postgresql.conf</filename>,
> <filename>pg_hba.conf</filename>, and
> <filename>pg_ident.conf</filename> are traditionally stored in
> <varname>PGDATA</> (although in <productname>PostgreSQL</productname>
> 8.0 and
> ! later, it is possible to place them elsewhere). By default the
> directory config is stored
> ! in <varname>PGDATA</>, however it should be kept along with
> <filename>postgresql.conf</filename>.
> </para>
>
> <table tocentry="1" id="pgdata-contents-table">
>
> This chunk doesn't make any sense to me. "Should" is always tricky.
> Why should I, and why might I not?
I mean to say here, that user needs to move config directory and its
contents along with
postgresql.conf.
Shall we change as below:
By default config directory is stored in PGDATA, however it needs to be kept
along with postgresql.conf
> <row>
> + <entry><filename>config</></entry>
> + <entry>Subdirectory containing automatically generated configuration
> files</entry>
> + </row>
> +
> + <row>
> <entry><filename>base</></entry>
> <entry>Subdirectory containing per-database subdirectories</entry>
> </row>
>
> Only automatically generated ones?
No, any other files can also be present.
How about change it as :
Subdirectory containing generated configuration files.
Any other suggestions?
This new directory's will be used to place generated files,
> COPY_STRING_FIELD(name);
> COPY_NODE_FIELD(args);
> COPY_SCALAR_FIELD(is_local);
> + COPY_SCALAR_FIELD(is_persistent);
>
> return newnode;
> }
>
> I suggest changing is_local into a new trivalued field that stores
> LOCAL
> or SESSION or PERSISTENT.
>
> n->is_local = false;
> $$ = (Node *) n;
> }
Okay, I will change it.
> + | SET PERSISTENT set_persistent
> + {
> + VariableSetStmt *n = $3;
> + n->is_persistent = true;
> + $$ = (Node *) n;
> + }
> ;
>
> set_rest:
>
> Why can't you use SET PERSISTENT set_rest?
As SET PERSISTENT cannot be used with some of syntaxes, example (SESSION
AUTHORIZATION) and also it is not supportted inside transaction blocks.
>
> *** a/src/backend/replication/basebackup.c
> --- b/src/backend/replication/basebackup.c
> ***************
> *** 755,760 **** sendDir(char *path, int basepathlen, bool sizeonly)
> --- 755,766 ----
> strlen(PG_TEMP_FILE_PREFIX)) ==
> 0)
> continue;
>
> + /* skip auto conf temporary file */
> + if (strncmp(de->d_name,
> + PG_AUTOCONF_FILENAME ".",
> + sizeof(PG_AUTOCONF_FILENAME))
> == 0)
> + continue;
> +
>
> Maybe pg_basebackup should be taught to ignore certain kinds of
> temporary files in general. The file name shouldn't be hardcoded into
> pg_basebackup. This would effectively make the configuration file
> naming scheme part of the replication protocol. See other thread about
> pg_basebackup client/server compatibility. This needs to be
> generalized.
As we are just trying to ignore the file during backup, why it should effect
replication protocol?
I mean protocol should be effected, if try to send something new or don't
send what is expected on the other side.
Also the same is done for team and backup label file.
> (Side thought: Does pg_basebackup copy editor temp and backup files?)
Currently pg_basebackup doesn't copy temp or backup files. The check I made
in code is similar to check for those two.
>
> + ereport(elevel,
> + (errmsg("Configuration parameters changed with SET
> PERSISTENT command before start of server "
> + "will not be effective as \"%s\" file is not
> accessible.", PG_AUTOCONF_FILENAME)));
>
> I'm having trouble interpreting this: How can you change something with
> SET PERSISTENT before the server starts?
This is for the case when somebody starts the server second time, or i can
say restart server
Example
1. Startup Server, Connect client
2. Change some settings with SET PERSISTENT
3. Shutdown server
4. Start Server; at this step if we don't find the file, the above error
message will appear.
> Also: errmsg messages should start with a lowercase letter and should
> generally be much shorter. Please review other cases in your patch as
> well.
Okay. I shall take care of it.
> + appendStringInfoString(&buf, "# Do not edit this file manually! "
> + "It is overwritten by the SET PERSISTENT command
> \n");
>
> There is some punctuation or something missing at the end.
>
> I suggest you break this into two lines. It's pretty long.
Okay. I shall take care of it.
>
> I think the naming of these files is suboptimal:
>
> + #define PG_AUTOCONF_DIR "config"
>
> "config" would seem to include pg_hba.conf and others. Or
> postgresql.conf for that matter. Maybe I should move postgresql.conf
> into config/. Another great new FAQ. The normal convention for this
> is
> "postgresql.conf.d". I don't see any reason not to use that. One
> reason that was brought up is that this doesn't match other things
> currently in PGDATA, but (a) actually it does (hint: postgresql.conf),
> and (b) neither does "config".
Reason was that "postgresql.conf.d" or "auto.conf.d" doesn't match existing
folder name,
whereas config can have some similarity with global directory as far as I
can see.
However changing name is not a big matter, it's more about for which name
there is more consensus.
> + #define PG_AUTOCONF_FILENAME "persistent.auto.conf"
>
> "auto" sounds like the result of an automatic tuning tool. Why not
> just
> persistent.conf. Or set-persistent.conf. That makes the association
> clear enough.
The 'auto' keyword is kept to make it different from others, as it will be
generated by Command.
It can give indication to user that this is not similar to other conf files
which he can directly edit.
> I don't know why the concept of a configuration directory is being
> introduced here. I mean, I'm all for that, but it seems completely
> independent of this. Some people wanted one setting per file, but
> that's not what is happening here. The name of the configuration file
> is hardcoded throughout, so there is no value in reading all files from
> a directory instead of just that one file.
>
> There is a lot of confusion throughout the patch about whether this
> config directory is for automatic files only or you can throw your own
> stuff in there at will. Another great FAQ. And then the whole dance
> about parsing postgresql.conf to make sure the include_dir is not
> removed (=> FAQ). Does it care whether it's at the beginning or the
> end? Should it?
This will also work as per existing behavior of #include_dir, which means
any configuration parameter values specified after this #include_dir
will override the values set by SET PERSISTENT statement. The same is
mentioned as WARNING just before this include directive.
> How does all of this work when you move the configuration files to,
> say,
> /etc/, but want to keep the SET PERSISTENT stuff under $PGDATA? Lots
> of
> moving parts there.
He needs to move config directory along with postgresql.conf.
> I think this could be much easier if you just read
> $PGDATA/persistent.conf or whatever if present. There is perhaps an
> analogy to be found with the proposed new recovery.conf handling (e.g.,
> always read if present after all other postgresql.conf processing or
> something like that).
In current mechanism, user can revert to old usage (edit every thing in
postgresql.conf) just by commenting or removing
include_dir = 'config' in postgresql.conf.
OTOH if we just try to read if present mechanism, we might not be able to
tell to user that his file is missing and if he has made
any changes by SET PERSISTENT they will not reflect after this startup.
> I'm thinking, this feature is ultimately intended to make things
> simpler
> for average users. But I see so many traps here. It needs to be
> simpler yet, and have less moving parts.
>
> I'd almost like an option to turn this off for a server, because it has
> so much potential for confusion.
If user comments or removes below line in postgresql.conf, then this feature
will be OFF
include_dir = 'config'
With Regards,
Amit Kapila.
On Wednesday, April 03, 2013 2:23 AM Greg Smith wrote: > On 4/1/13 5:44 AM, Amit Kapila wrote: > > > I think in that case we can have 3 separate patches > > 1. Memory growth defect fix > > 2. Default postgresql.conf to include config directory and SET > Persistent > > into single file implementation > > 3. Rearrangement of GUC validation into validate_conf_option > function. > > > > As already there is review happened for point 2 and point 1 is an > existing > > code defect fix, so in my opinion > > Patch 1 and 2 should be considered for 9.3. > > They have been considered for 9.3, I just doubt they could get > committed > right now. In order for this to go in as part of the very last 9.3 > feature changes (which is where we're at in the development cycle), > you'd need to have a committer volunteer to take on the job of doing a > second level of review on it. And then that would have to happen > without any other issues popping up. That usually is not how it works. > Normally the first round of committer review finds another round of > issues, and there's at least one more update before commit. (Note that > this is exactly what just happened today, with the review from Peter > Eisentraut) > > I'm not saying it's impossible for this feature to go in to 9.3, but > I'm > not seeing any committers volunteer to take on the job either. I do > want to see this feature go in--I'll update it for 9.4 even if you > don't--but we're already into April. There isn't much time left for > 9.3. And the security release this week has gobbled up a good chunk of > committer and packager time unexpectedly, which is just bad luck for > your submission. > > From a process perspective, features that enter the last CF of a > release that are very close to ready from the start have good odds of > being committed. You've done an excellent job of updating this in > response to feedback, but it has involved a long list of changes so > far. > It's fair to say this was still a rough feature at the start of CF > 2013-01, and now it's good but can be usefully polished a bit more. > > For something of this size, going from rough feature to commit quality > normally takes more than one CF. I don't have any obvious errors to > point out right now. But I think there's still some room to improve on > this before commit. Andres mentioned on another thread that he thought > merging some of your ideas with the version Zoltan did was useful to > look at, and I was thinking of something similar. This is close to > being ready, and I hope you won't get discouraged just because it's > probably going to slip to 9.4. Thank you for keeping me motivated for this patch now and throughout the last CF by providing valuable suggestions and comments. With Regards, Amit Kapila.
On Tue, Apr 2, 2013 at 12:19 PM, Peter Eisentraut <peter_e@gmx.net> wrote: > It's weird that SET LOCAL and SET SESSION actually *set* the value, and > the second key word determines how long the setting will last. SET > PERSISTENT doesn't actually set the value. I predict that this will be > a new favorite help-it-doesn't-work FAQ. I think this is another argument against this particular syntax. I have always thought that something along the lines of ALTER SYSTEM would be more appropriate. ALTER DATABASE .. SET and ALTER ROLE .. SET don't change the value immediately either, and nobody gets confused about that to my knowledge. But I can see where SET PERSISTENT could cause that sort of confusion. ...Robert
Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Apr 2, 2013 at 12:19 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>> It's weird that SET LOCAL and SET SESSION actually *set* the value, and
>> the second key word determines how long the setting will last. SET
>> PERSISTENT doesn't actually set the value. I predict that this will be
>> a new favorite help-it-doesn't-work FAQ.
> I think this is another argument against this particular syntax. I
> have always thought that something along the lines of ALTER SYSTEM
> would be more appropriate. ALTER DATABASE .. SET and ALTER ROLE ..
> SET don't change the value immediately either, and nobody gets
> confused about that to my knowledge. But I can see where SET
> PERSISTENT could cause that sort of confusion.
Yeah, I think I argued for using the SET syntax to start with, but
I'm coming around to the position that SET PERSISTENT is too much
unlike the behavior of other varieties of SET. ALTER is sounding
more attractive to me now. Not sure about "ALTER SYSTEM" in particular
though --- it's not clear that that has any real merit other than
already existing as a keyword. (Not that that's negligible.)
ALTER CONFIGURATION is another alternative using an existing keyword
that might be worth considering.
regards, tom lane
On Wed, Apr 3, 2013 at 2:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Tue, Apr 2, 2013 at 12:19 PM, Peter Eisentraut <peter_e@gmx.net> wrote: >>> It's weird that SET LOCAL and SET SESSION actually *set* the value, and >>> the second key word determines how long the setting will last. SET >>> PERSISTENT doesn't actually set the value. I predict that this will be >>> a new favorite help-it-doesn't-work FAQ. > >> I think this is another argument against this particular syntax. I >> have always thought that something along the lines of ALTER SYSTEM >> would be more appropriate. ALTER DATABASE .. SET and ALTER ROLE .. >> SET don't change the value immediately either, and nobody gets >> confused about that to my knowledge. But I can see where SET >> PERSISTENT could cause that sort of confusion. > > Yeah, I think I argued for using the SET syntax to start with, but > I'm coming around to the position that SET PERSISTENT is too much > unlike the behavior of other varieties of SET. ALTER is sounding > more attractive to me now. Not sure about "ALTER SYSTEM" in particular > though --- it's not clear that that has any real merit other than > already existing as a keyword. (Not that that's negligible.) > ALTER CONFIGURATION is another alternative using an existing keyword > that might be worth considering. Yeah, I thought about something like that. Aside from saving on keywords, the reason I like ALTER SYSTEM or similar is that I suspect there will be other system-wide things that we may want to let people ALTER in the future, so I think that route might avoid an unnecessary proliferation of top-level commands. I am not, however, deadly attached to the idea, if someone's got a good reason for preferring something else. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Thursday, April 04, 2013 2:52 AM Robert Haas wrote:
> On Wed, Apr 3, 2013 at 2:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Robert Haas <robertmhaas@gmail.com> writes:
> >> On Tue, Apr 2, 2013 at 12:19 PM, Peter Eisentraut <peter_e@gmx.net>
> wrote:
> >>> It's weird that SET LOCAL and SET SESSION actually *set* the value,
> and
> >>> the second key word determines how long the setting will last. SET
> >>> PERSISTENT doesn't actually set the value. I predict that this
> will be
> >>> a new favorite help-it-doesn't-work FAQ.
> >
> >> I think this is another argument against this particular syntax. I
> >> have always thought that something along the lines of ALTER SYSTEM
> >> would be more appropriate. ALTER DATABASE .. SET and ALTER ROLE ..
> >> SET don't change the value immediately either, and nobody gets
> >> confused about that to my knowledge. But I can see where SET
> >> PERSISTENT could cause that sort of confusion.
> >
> > Yeah, I think I argued for using the SET syntax to start with, but
> > I'm coming around to the position that SET PERSISTENT is too much
> > unlike the behavior of other varieties of SET. ALTER is sounding
> > more attractive to me now. Not sure about "ALTER SYSTEM" in
> particular
> > though --- it's not clear that that has any real merit other than
> > already existing as a keyword. (Not that that's negligible.)
> > ALTER CONFIGURATION is another alternative using an existing keyword
> > that might be worth considering.
>
> Yeah, I thought about something like that. Aside from saving on
> keywords, the reason I like ALTER SYSTEM or similar is that I suspect
> there will be other system-wide things that we may want to let people
> ALTER in the future, so I think that route might avoid an unnecessary
> proliferation of top-level commands. I am not, however, deadly
> attached to the idea, if someone's got a good reason for preferring
> something else.
I think second parameter in SET command telling the scope should be fine. As
I could see Oracle
also has similar syntax for it's ALTER SYSTEM Command (Alter System Scope
[Memory|Spfile|Both]).
Description in short:
SPFILE indicates that the change is made in the server parameter file. The
new
setting takes effect when the database is next shut down and started up
again.
MEMORY indicates that the change is made in memory, takes effect
immediately,
and persists until the database is shut down.
The only reason to show above example is that second parameter telling Scope
exists in other databases as well.
However if you are not convinced with above reasoning, then the Alter syntax
can be as follows:
ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
With Regards,
Amit Kapila.
On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
> On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
This feature was discussed for 9.3, but couldn't get committed.
History for this patch is that in the last, Peter Eisentraut has given quite
a few review comments, most of which I have closed in the patch attached.
Apart from that Robert and Tom intended to use ALTER SYSTEM or variant of
ALTER rather than SET. Syntax can be as below:
ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
Some Comments of Peter E which needs to be discussed before changing are as
below:
1) Name of directory should be postgresql.conf.d rather than config.
2) Name of the file persistent.auto.conf should not have auto word.
I have not modified the code, as these had been previously discussed in this
mail chain and as a conclusion of that, I had kept the current names for
file and directory.
> > I'm going to ignore most of the discussion that led up to this and
> give
> > this patch a fresh look.
>
> Thank you.
>
> > + <screen>
> > + SET PERSISTENT max_connections To 10;
> > + </screen>
> >
> > The To should probably be capitalized.
>
> Okay, shall fix it.
Changed to
SET PERSISTENT checkpoint_timeout TO 600
> > I doubt this example actually works because changing max_connections
> > requires a restart. Try to find a better example.
>
> Any parameter's changed with this command can be effective either after
> restart or SIGHUP.
> So I will change it to SIGHUP parameter.
>
> > It's weird that SET LOCAL and SET SESSION actually *set* the value,
> and
> > the second key word determines how long the setting will last. SET
> > PERSISTENT doesn't actually set the value. I predict that this will
> be
> > a new favorite help-it-doesn't-work FAQ.
> >
> > <varlistentry>
> > <term><literal>SCHEMA</literal></term>
> > <listitem>
> > ! <para><literal>SET [ PERSISTENT ] SCHEMA
> > '<replaceable>value</>'</> is an alias for
> > <literal>SET search_path TO <replaceable>value</></>. Only
> > one
> > schema can be specified using this syntax.
> > </para>
> >
> > I don't think [ PERSISTENT ] needs to be added in these and similar
> > snippets. We don't mention LOCAL etc. here either.
>
> Agreed, shall fix this.
Removed from all similar places.
> > --- 34,41 ----
> > <filename>postgresql.conf</filename>,
> > <filename>pg_hba.conf</filename>, and
> > <filename>pg_ident.conf</filename> are traditionally stored in
> > <varname>PGDATA</> (although in
> <productname>PostgreSQL</productname>
> > 8.0 and
> > ! later, it is possible to place them elsewhere). By default the
> > directory config is stored
> > ! in <varname>PGDATA</>, however it should be kept along with
> > <filename>postgresql.conf</filename>.
> > </para>
> >
> > <table tocentry="1" id="pgdata-contents-table">
> >
> > This chunk doesn't make any sense to me. "Should" is always tricky.
> > Why should I, and why might I not?
>
> I mean to say here, that user needs to move config directory and its
> contents along with
> postgresql.conf.
> Shall we change as below:
> By default config directory is stored in PGDATA, however it needs to be
> kept along with postgresql.conf
Changed as proposed above.
>
> > <row>
> > + <entry><filename>config</></entry>
> > + <entry>Subdirectory containing automatically generated
> configuration
> > files</entry>
> > + </row>
> > +
> > + <row>
> > <entry><filename>base</></entry>
> > <entry>Subdirectory containing per-database subdirectories</entry>
> > </row>
> >
> > Only automatically generated ones?
>
> No, any other files can also be present.
> How about change it as :
> Subdirectory containing generated configuration files.
> Any other suggestions?
Changed as above.
> This new directory's will be used to place generated files,
>
> > COPY_STRING_FIELD(name);
> > COPY_NODE_FIELD(args);
> > COPY_SCALAR_FIELD(is_local);
> > + COPY_SCALAR_FIELD(is_persistent);
> >
> > return newnode;
> > }
> >
> > I suggest changing is_local into a new trivalued field that stores
> > LOCAL
> > or SESSION or PERSISTENT.
> >
> > n->is_local = false;
> > $$ = (Node *) n;
> > }
>
> Okay, I will change it.
>
> > + | SET PERSISTENT set_persistent
> > + {
> > + VariableSetStmt *n = $3;
> > + n->is_persistent = true;
> > + $$ = (Node *) n;
> > + }
> > ;
> >
> > set_rest:
> >
> > Why can't you use SET PERSISTENT set_rest?
>
> As SET PERSISTENT cannot be used with some of syntaxes, example
> (SESSION AUTHORIZATION) and also it is not supportted inside
> transaction blocks.
>
> >
> > *** a/src/backend/replication/basebackup.c
> > --- b/src/backend/replication/basebackup.c
> > ***************
> > *** 755,760 **** sendDir(char *path, int basepathlen, bool sizeonly)
> > --- 755,766 ----
> > strlen(PG_TEMP_FILE_PREFIX))
> ==
> > 0)
> > continue;
> >
> > + /* skip auto conf temporary file */
> > + if (strncmp(de->d_name,
> > + PG_AUTOCONF_FILENAME ".",
> > + sizeof(PG_AUTOCONF_FILENAME))
> > == 0)
> > + continue;
> > +
> >
> > Maybe pg_basebackup should be taught to ignore certain kinds of
> > temporary files in general. The file name shouldn't be hardcoded
> into
> > pg_basebackup. This would effectively make the configuration file
> > naming scheme part of the replication protocol. See other thread
> about
> > pg_basebackup client/server compatibility. This needs to be
> > generalized.
>
> As we are just trying to ignore the file during backup, why it should
> effect replication protocol?
> I mean protocol should be effected, if try to send something new or
> don't send what is expected on the other side.
> Also the same is done for team and backup label file.
>
> > (Side thought: Does pg_basebackup copy editor temp and backup files?)
>
> Currently pg_basebackup doesn't copy temp or backup files. The check I
> made in code is similar to check for those two.
>
> >
> > + ereport(elevel,
> > + (errmsg("Configuration parameters changed with SET
> > PERSISTENT command before start of server "
> > + "will not be effective as \"%s\" file is not
> > accessible.", PG_AUTOCONF_FILENAME)));
> >
> > I'm having trouble interpreting this: How can you change something
> with
> > SET PERSISTENT before the server starts?
>
> This is for the case when somebody starts the server second time, or i
> can say restart server
> Example
> 1. Startup Server, Connect client
> 2. Change some settings with SET PERSISTENT
> 3. Shutdown server
> 4. Start Server; at this step if we don't find the file, the above
> error message will appear.
>
> > Also: errmsg messages should start with a lowercase letter and should
> > generally be much shorter. Please review other cases in your patch
> as
> > well.
>
> Okay. I shall take care of it.
Done, all error messages started with lower case letter. Apart from above
message, all other error messages are short, do you feel any other needs to
be shortened.
> > + appendStringInfoString(&buf, "# Do not edit this file manually! "
> > + "It is overwritten by the SET PERSISTENT
> command
> > \n");
> >
> > There is some punctuation or something missing at the end.
> >
> > I suggest you break this into two lines. It's pretty long.
>
> Okay. I shall take care of it.
Done, broken into 2 lines.
>
> >
> > I think the naming of these files is suboptimal:
> >
> > + #define PG_AUTOCONF_DIR "config"
> >
> > "config" would seem to include pg_hba.conf and others. Or
> > postgresql.conf for that matter. Maybe I should move postgresql.conf
> > into config/. Another great new FAQ. The normal convention for this
> > is
> > "postgresql.conf.d". I don't see any reason not to use that. One
> > reason that was brought up is that this doesn't match other things
> > currently in PGDATA, but (a) actually it does (hint:
> postgresql.conf),
> > and (b) neither does "config".
>
> Reason was that "postgresql.conf.d" or "auto.conf.d" doesn't match
> existing folder name,
> whereas config can have some similarity with global directory as far as
> I can see.
>
> However changing name is not a big matter, it's more about for which
> name there is more consensus.
>
> > + #define PG_AUTOCONF_FILENAME "persistent.auto.conf"
> >
> > "auto" sounds like the result of an automatic tuning tool. Why not
> > just
> > persistent.conf. Or set-persistent.conf. That makes the association
> > clear enough.
>
>
> The 'auto' keyword is kept to make it different from others, as it will
> be generated by Command.
> It can give indication to user that this is not similar to other conf
> files which he can directly edit.
>
> > I don't know why the concept of a configuration directory is being
> > introduced here. I mean, I'm all for that, but it seems completely
> > independent of this. Some people wanted one setting per file, but
> > that's not what is happening here. The name of the configuration
> file
> > is hardcoded throughout, so there is no value in reading all files
> from
> > a directory instead of just that one file.
> >
> > There is a lot of confusion throughout the patch about whether this
> > config directory is for automatic files only or you can throw your
> own
> > stuff in there at will. Another great FAQ. And then the whole dance
> > about parsing postgresql.conf to make sure the include_dir is not
> > removed (=> FAQ). Does it care whether it's at the beginning or the
> > end? Should it?
>
> This will also work as per existing behavior of #include_dir, which
> means any configuration parameter values specified after this
> #include_dir
> will override the values set by SET PERSISTENT statement. The same is
> mentioned as WARNING just before this include directive.
>
> > How does all of this work when you move the configuration files to,
> > say,
> > /etc/, but want to keep the SET PERSISTENT stuff under $PGDATA? Lots
> > of
> > moving parts there.
>
> He needs to move config directory along with postgresql.conf.
>
> > I think this could be much easier if you just read
> > $PGDATA/persistent.conf or whatever if present. There is perhaps an
> > analogy to be found with the proposed new recovery.conf handling
> (e.g.,
> > always read if present after all other postgresql.conf processing or
> > something like that).
>
> In current mechanism, user can revert to old usage (edit every thing in
> postgresql.conf) just by commenting or removing
> include_dir = 'config' in postgresql.conf.
> OTOH if we just try to read if present mechanism, we might not be able
> to tell to user that his file is missing and if he has made
> any changes by SET PERSISTENT they will not reflect after this startup.
>
> > I'm thinking, this feature is ultimately intended to make things
> > simpler
> > for average users. But I see so many traps here. It needs to be
> > simpler yet, and have less moving parts.
> >
> > I'd almost like an option to turn this off for a server, because it
> has
> > so much potential for confusion.
>
> If user comments or removes below line in postgresql.conf, then this
> feature will be OFF
> include_dir = 'config'
Do you think above way to switch OFF the feature is acceptable or do you
have any other idea?
Suggestions/Feedback?
With Regards,
Amit Kapila.
On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
> > On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
>
There are 2 options to proceed for this patch for 9.4
1. Upload the SET PERSISTENT syntax patch for coming CF by fixing existing
review comments
2. Implement new syntax ALTER SYSTEM as proposed in below mail
Could you suggest me what could be best way to proceed for this patch?
> This feature was discussed for 9.3, but couldn't get committed.
> History for this patch is that in the last, Peter Eisentraut has given
> quite a few review comments, most of which I have closed in the patch
> attached.
>
> Apart from that Robert and Tom intended to use ALTER SYSTEM or variant
> of ALTER rather than SET. Syntax can be as below:
>
> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>
> Some Comments of Peter E which needs to be discussed before changing
> are as
> below:
>
> 1) Name of directory should be postgresql.conf.d rather than config.
> 2) Name of the file persistent.auto.conf should not have auto word.
>
> I have not modified the code, as these had been previously discussed in
> this mail chain and as a conclusion of that, I had kept the current
> names for file and directory.
>
> > > I'm going to ignore most of the discussion that led up to this and
> > give
> > > this patch a fresh look.
> >
> > Thank you.
> >
> > > + <screen>
> > > + SET PERSISTENT max_connections To 10; </screen>
> > >
> > > The To should probably be capitalized.
> >
> > Okay, shall fix it.
>
> Changed to
> SET PERSISTENT checkpoint_timeout TO 600
>
> > > I doubt this example actually works because changing
> max_connections
> > > requires a restart. Try to find a better example.
> >
> > Any parameter's changed with this command can be effective either
> > after restart or SIGHUP.
> > So I will change it to SIGHUP parameter.
> >
> > > It's weird that SET LOCAL and SET SESSION actually *set* the value,
> > and
> > > the second key word determines how long the setting will last. SET
> > > PERSISTENT doesn't actually set the value. I predict that this
> will
> > be
> > > a new favorite help-it-doesn't-work FAQ.
> > >
> > > <varlistentry>
> > > <term><literal>SCHEMA</literal></term>
> > > <listitem>
> > > ! <para><literal>SET [ PERSISTENT ] SCHEMA
> > > '<replaceable>value</>'</> is an alias for
> > > <literal>SET search_path TO <replaceable>value</></>.
> Only
> > > one
> > > schema can be specified using this syntax.
> > > </para>
> > >
> > > I don't think [ PERSISTENT ] needs to be added in these and similar
> > > snippets. We don't mention LOCAL etc. here either.
> >
> > Agreed, shall fix this.
>
> Removed from all similar places.
>
> > > --- 34,41 ----
> > > <filename>postgresql.conf</filename>,
> > > <filename>pg_hba.conf</filename>, and
> > > <filename>pg_ident.conf</filename> are traditionally stored in
> > > <varname>PGDATA</> (although in
> > <productname>PostgreSQL</productname>
> > > 8.0 and
> > > ! later, it is possible to place them elsewhere). By default the
> > > directory config is stored ! in <varname>PGDATA</>, however it
> > > should be kept along with <filename>postgresql.conf</filename>.
> > > </para>
> > >
> > > <table tocentry="1" id="pgdata-contents-table">
> > >
> > > This chunk doesn't make any sense to me. "Should" is always
> tricky.
> > > Why should I, and why might I not?
> >
> > I mean to say here, that user needs to move config directory and its
> > contents along with postgresql.conf.
> > Shall we change as below:
> > By default config directory is stored in PGDATA, however it needs to
> > be kept along with postgresql.conf
>
> Changed as proposed above.
>
> >
> > > <row>
> > > + <entry><filename>config</></entry>
> > > + <entry>Subdirectory containing automatically generated
> > configuration
> > > files</entry>
> > > + </row>
> > > +
> > > + <row>
> > > <entry><filename>base</></entry>
> > > <entry>Subdirectory containing per-database
> subdirectories</entry>
> > > </row>
> > >
> > > Only automatically generated ones?
> >
> > No, any other files can also be present.
> > How about change it as :
> > Subdirectory containing generated configuration files.
> > Any other suggestions?
>
> Changed as above.
>
> > This new directory's will be used to place generated files,
> >
> > > COPY_STRING_FIELD(name);
> > > COPY_NODE_FIELD(args);
> > > COPY_SCALAR_FIELD(is_local);
> > > + COPY_SCALAR_FIELD(is_persistent);
> > >
> > > return newnode;
> > > }
> > >
> > > I suggest changing is_local into a new trivalued field that stores
> > > LOCAL or SESSION or PERSISTENT.
> > >
> > > n->is_local = false;
> > > $$ = (Node *) n;
> > > }
> >
> > Okay, I will change it.
> >
> > > + | SET PERSISTENT set_persistent
> > > + {
> > > + VariableSetStmt *n = $3;
> > > + n->is_persistent = true;
> > > + $$ = (Node *) n;
> > > + }
> > > ;
> > >
> > > set_rest:
> > >
> > > Why can't you use SET PERSISTENT set_rest?
> >
> > As SET PERSISTENT cannot be used with some of syntaxes, example
> > (SESSION AUTHORIZATION) and also it is not supportted inside
> > transaction blocks.
> >
> > >
> > > *** a/src/backend/replication/basebackup.c
> > > --- b/src/backend/replication/basebackup.c
> > > ***************
> > > *** 755,760 **** sendDir(char *path, int basepathlen, bool
> sizeonly)
> > > --- 755,766 ----
> > >
> strlen(PG_TEMP_FILE_PREFIX))
> > ==
> > > 0)
> > > continue;
> > >
> > > + /* skip auto conf temporary file */
> > > + if (strncmp(de->d_name,
> > > + PG_AUTOCONF_FILENAME ".",
> > > +
> > > + sizeof(PG_AUTOCONF_FILENAME))
> > > == 0)
> > > + continue;
> > > +
> > >
> > > Maybe pg_basebackup should be taught to ignore certain kinds of
> > > temporary files in general. The file name shouldn't be hardcoded
> > into
> > > pg_basebackup. This would effectively make the configuration file
> > > naming scheme part of the replication protocol. See other thread
> > about
> > > pg_basebackup client/server compatibility. This needs to be
> > > generalized.
> >
> > As we are just trying to ignore the file during backup, why it should
> > effect replication protocol?
> > I mean protocol should be effected, if try to send something new or
> > don't send what is expected on the other side.
> > Also the same is done for team and backup label file.
> >
> > > (Side thought: Does pg_basebackup copy editor temp and backup
> > > files?)
> >
> > Currently pg_basebackup doesn't copy temp or backup files. The check
> I
> > made in code is similar to check for those two.
> >
> > >
> > > + ereport(elevel,
> > > + (errmsg("Configuration parameters changed with SET
> > > PERSISTENT command before start of server "
> > > + "will not be effective as \"%s\" file is
> > > + not
> > > accessible.", PG_AUTOCONF_FILENAME)));
> > >
> > > I'm having trouble interpreting this: How can you change something
> > with
> > > SET PERSISTENT before the server starts?
> >
> > This is for the case when somebody starts the server second time, or
> i
> > can say restart server Example 1. Startup Server, Connect client 2.
> > Change some settings with SET PERSISTENT 3. Shutdown server 4. Start
> > Server; at this step if we don't find the file, the above error
> > message will appear.
> >
> > > Also: errmsg messages should start with a lowercase letter and
> > > should generally be much shorter. Please review other cases in
> your
> > > patch
> > as
> > > well.
> >
> > Okay. I shall take care of it.
>
> Done, all error messages started with lower case letter. Apart from
> above message, all other error messages are short, do you feel any
> other needs to be shortened.
>
> > > + appendStringInfoString(&buf, "# Do not edit this file manually!
> "
> > > + "It is overwritten by the SET PERSISTENT
> > command
> > > \n");
> > >
> > > There is some punctuation or something missing at the end.
> > >
> > > I suggest you break this into two lines. It's pretty long.
> >
> > Okay. I shall take care of it.
>
>
> Done, broken into 2 lines.
>
> >
> > >
> > > I think the naming of these files is suboptimal:
> > >
> > > + #define PG_AUTOCONF_DIR "config"
> > >
> > > "config" would seem to include pg_hba.conf and others. Or
> > > postgresql.conf for that matter. Maybe I should move
> > > postgresql.conf into config/. Another great new FAQ. The normal
> > > convention for this is "postgresql.conf.d". I don't see any reason
> > > not to use that. One reason that was brought up is that this
> > > doesn't match other things currently in PGDATA, but (a) actually it
> > > does (hint:
> > postgresql.conf),
> > > and (b) neither does "config".
> >
> > Reason was that "postgresql.conf.d" or "auto.conf.d" doesn't match
> > existing folder name, whereas config can have some similarity with
> > global directory as far as I can see.
> >
> > However changing name is not a big matter, it's more about for which
> > name there is more consensus.
> >
> > > + #define PG_AUTOCONF_FILENAME "persistent.auto.conf"
> > >
> > > "auto" sounds like the result of an automatic tuning tool. Why not
> > > just persistent.conf. Or set-persistent.conf. That makes the
> > > association clear enough.
> >
> >
> > The 'auto' keyword is kept to make it different from others, as it
> > will be generated by Command.
> > It can give indication to user that this is not similar to other conf
> > files which he can directly edit.
> >
> > > I don't know why the concept of a configuration directory is being
> > > introduced here. I mean, I'm all for that, but it seems completely
> > > independent of this. Some people wanted one setting per file, but
> > > that's not what is happening here. The name of the configuration
> > file
> > > is hardcoded throughout, so there is no value in reading all files
> > from
> > > a directory instead of just that one file.
> > >
> > > There is a lot of confusion throughout the patch about whether this
> > > config directory is for automatic files only or you can throw your
> > own
> > > stuff in there at will. Another great FAQ. And then the whole
> > > dance about parsing postgresql.conf to make sure the include_dir is
> > > not removed (=> FAQ). Does it care whether it's at the beginning
> or
> > > the end? Should it?
> >
> > This will also work as per existing behavior of #include_dir, which
> > means any configuration parameter values specified after this
> > #include_dir will override the values set by SET PERSISTENT
> statement.
> > The same is mentioned as WARNING just before this include directive.
> >
> > > How does all of this work when you move the configuration files to,
> > > say, /etc/, but want to keep the SET PERSISTENT stuff under
> $PGDATA?
> > > Lots of moving parts there.
> >
> > He needs to move config directory along with postgresql.conf.
> >
> > > I think this could be much easier if you just read
> > > $PGDATA/persistent.conf or whatever if present. There is perhaps
> an
> > > analogy to be found with the proposed new recovery.conf handling
> > (e.g.,
> > > always read if present after all other postgresql.conf processing
> or
> > > something like that).
> >
> > In current mechanism, user can revert to old usage (edit every thing
> > in
> > postgresql.conf) just by commenting or removing include_dir =
> 'config'
> > in postgresql.conf.
> > OTOH if we just try to read if present mechanism, we might not be
> able
> > to tell to user that his file is missing and if he has made any
> > changes by SET PERSISTENT they will not reflect after this startup.
> >
> > > I'm thinking, this feature is ultimately intended to make things
> > > simpler for average users. But I see so many traps here. It needs
> > > to be simpler yet, and have less moving parts.
> > >
> > > I'd almost like an option to turn this off for a server, because it
> > has
> > > so much potential for confusion.
> >
> > If user comments or removes below line in postgresql.conf, then this
> > feature will be OFF include_dir = 'config'
>
> Do you think above way to switch OFF the feature is acceptable or do
> you have any other idea?
>
>
> Suggestions/Feedback?
With Regards,
Amit Kapila.
On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila <amit.kapila@huawei.com> wrote: > On Monday, May 27, 2013 4:17 PM Amit Kapila wrote: >> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote: >> > On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote: >> > > There are 2 options to proceed for this patch for 9.4 > > 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing existing > review comments > 2. Implement new syntax ALTER SYSTEM as proposed in below mail > > Could you suggest me what could be best way to proceed for this patch? I'm still in favor of some syntax involving ALTER, because it's still true that this behaves more like the existing GUC-setting commands that use ALTER (which change configuration for future sessions) rather the ones that use SET (which change the current settings for some period of time). But I can't speak for people who are not me. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
> On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila <amit.kapila@huawei.com>
> wrote:
> > On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
> >> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
> >> > On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
> >>
> >
> > There are 2 options to proceed for this patch for 9.4
> >
> > 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
> existing
> > review comments
> > 2. Implement new syntax ALTER SYSTEM as proposed in below mail
> >
> > Could you suggest me what could be best way to proceed for this
> patch?
>
> I'm still in favor of some syntax involving ALTER, because it's still
> true that this behaves more like the existing GUC-setting commands
> that use ALTER (which change configuration for future sessions) rather
> the ones that use SET (which change the current settings for some
> period of time).
I will change the patch as per below syntax if there are no objections:
ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
With Regards,
Amit Kapila.
ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Friday, June 07, 2013 9:45 AM Amit Kapila wrote:
> On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
> > On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila <amit.kapila@huawei.com>
> > wrote:
> > > On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
> > >> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
> > >> > On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
> > >>
> > >
> > > There are 2 options to proceed for this patch for 9.4
> > >
> > > 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
> > existing
> > > review comments
> > > 2. Implement new syntax ALTER SYSTEM as proposed in below mail
> > >
> > > Could you suggest me what could be best way to proceed for this
> > patch?
> >
> > I'm still in favor of some syntax involving ALTER, because it's still
> > true that this behaves more like the existing GUC-setting commands
> > that use ALTER (which change configuration for future sessions)
> rather
> > the ones that use SET (which change the current settings for some
> > period of time).
>
>
> I will change the patch as per below syntax if there are no objections:
>
> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
Modified patch contains:
1. Syntax implemented is
ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
DEFAULT};
If user specifies DEFAULT, it will remove entry from auto conf file.
2. File name to store settings set by ALTER SYSTEM command is still
persistent.auto.conf
3. Added a new page for Alter System command in docs. In compatability
section, I had mentioned that this statement is an PostgresQL extension. I had tried to search in
SQL-92 spec but couldn't find such command.
4. During test of this patch, I had observed one problem for PGC_BACKEND
parameters like log_connections. If I change these parameters directly in postgresql.conf and then do
pg_reload_conf() and reconnect, it will still show the old value. This is observed only on WINDOWS. I will
investigate this problem and update you. Due to this problem, I had removed few test cases.
Kindly let me know your suggestions.
With Regards,
Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 06/13/2013 05:35 AM, Amit Kapila wrote:
> On Friday, June 07, 2013 9:45 AM Amit Kapila wrote:
>> On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
>>> On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila <amit.kapila@huawei.com>
>>> wrote:
>>>> On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
>>>>> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
>>>>>> On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
>>>>>
>>>>
>>>> There are 2 options to proceed for this patch for 9.4
>>>>
>>>> 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
>>> existing
>>>> review comments
>>>> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
>>>>
>>>> Could you suggest me what could be best way to proceed for this
>>> patch?
>>>
>>> I'm still in favor of some syntax involving ALTER, because it's still
>>> true that this behaves more like the existing GUC-setting commands
>>> that use ALTER (which change configuration for future sessions)
>> rather
>>> the ones that use SET (which change the current settings for some
>>> period of time).
>>
>>
>> I will change the patch as per below syntax if there are no objections:
>>
>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>
> Modified patch contains:
>
> 1. Syntax implemented is
> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
> DEFAULT};
>
> If user specifies DEFAULT, it will remove entry from auto conf file.
>
> 2. File name to store settings set by ALTER SYSTEM command is still
> persistent.auto.conf
Why? Shouldn't it just be auto.conf? Or system.auto.conf?
I prefer auto.conf, personally.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Friday, June 14, 2013 3:17 AM Josh Berkus wrote:
> On 06/13/2013 05:35 AM, Amit Kapila wrote:
> > On Friday, June 07, 2013 9:45 AM Amit Kapila wrote:
> >> On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
> >>> On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila
> <amit.kapila@huawei.com>
> >>> wrote:
> >>>> On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
> >>>>> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
> >>>>>> On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
> >>>>>
> >>>>
> >>>> There are 2 options to proceed for this patch for 9.4
> >>>>
> >>>> 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
> >>> existing
> >>>> review comments
> >>>> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
> >>>>
> >>>> Could you suggest me what could be best way to proceed for this
> >>> patch?
> >>>
> >>> I'm still in favor of some syntax involving ALTER, because it's
> still
> >>> true that this behaves more like the existing GUC-setting commands
> >>> that use ALTER (which change configuration for future sessions)
> >> rather
> >>> the ones that use SET (which change the current settings for some
> >>> period of time).
> >>
> >>
> >> I will change the patch as per below syntax if there are no
> objections:
> >>
> >> ALTER SYSTEM SET configuration_parameter {TO | =} {value, |
> 'value'};
> >
> > Modified patch contains:
> >
> > 1. Syntax implemented is
> > ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
> > DEFAULT};
> >
> > If user specifies DEFAULT, it will remove entry from auto conf file.
> >
> > 2. File name to store settings set by ALTER SYSTEM command is still
> > persistent.auto.conf
>
> Why? Shouldn't it just be auto.conf? Or system.auto.conf?
I had kept same name as it was decided for 9.3, but now as command has changed so it makes more sense to change name
aswell. I think it can be one of what you suggested or postgresql.auto.conf.
> I prefer auto.conf, personally. Thanks. if no body has any other suggestion I will change it. I think one of
system.auto.confor postgresql.auto.conf is more appropriate because it either resembles command or existing
configurationsettings file.
With Regards,
Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Boszormenyi Zoltan
Date:
<div class="moz-cite-prefix">Hi,<br /><br /> review below.<br /><br /> 2013-06-13 14:35 keltezéssel, Amit Kapila
írta:<br/></div><blockquote cite="mid:009401ce6832$76653200$632f9600$@kapila@huawei.com" type="cite"><pre wrap="">On
Friday,June 07, 2013 9:45 AM Amit Kapila wrote:
</pre><blockquote type="cite"><pre wrap="">On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
</pre><blockquote type="cite"><pre wrap="">On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila <a class="moz-txt-link-rfc2396E"
href="mailto:amit.kapila@huawei.com"><amit.kapila@huawei.com></a>
wrote:
</pre><blockquote type="cite"><pre wrap="">On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
</pre><blockquote type="cite"><pre wrap="">On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
</pre><blockquote type="cite"><pre wrap="">On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
</pre></blockquote><pre wrap="">
</pre></blockquote><pre wrap="">
There are 2 options to proceed for this patch for 9.4
1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
</pre></blockquote><pre wrap="">existing
</pre><blockquote type="cite"><pre wrap="">review comments
2. Implement new syntax ALTER SYSTEM as proposed in below mail
Could you suggest me what could be best way to proceed for this
</pre></blockquote><pre wrap="">patch?
I'm still in favor of some syntax involving ALTER, because it's still
true that this behaves more like the existing GUC-setting commands
that use ALTER (which change configuration for future sessions)
</pre></blockquote><pre wrap="">rather
</pre><blockquote type="cite"><pre wrap="">the ones that use SET (which change the current settings for some
period of time).
</pre></blockquote><pre wrap="">
I will change the patch as per below syntax if there are no objections:
ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
</pre></blockquote><pre wrap="">
Modified patch contains:
1. Syntax implemented is
ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
DEFAULT};
If user specifies DEFAULT, it will remove entry from auto conf file.
2. File name to store settings set by ALTER SYSTEM command is still
persistent.auto.conf
3. Added a new page for Alter System command in docs. In compatability
section, I had mentioned that this statement is an PostgresQL extension. I had tried to search in
SQL-92 spec but couldn't find such command.
4. During test of this patch, I had observed one problem for PGC_BACKEND
parameters like log_connections. If I change these parameters directly in postgresql.conf and then do
pg_reload_conf() and reconnect, it will still show the old value. This is observed only on WINDOWS. I will
investigate this problem and update you. Due to this problem, I had removed few test cases.</pre></blockquote><br /> I
can'treproduce this error under Linux (Fedora 19/x86_64).<br /><br /> The bug might be somewhere else and not caused by
yourpatch<br /> if manually adding/removing "log_connections = on" from postgresql.conf<br /> exhibits the same
behaviourunder Windows. (If I read it correctly,<br /> you tested it exactly this way.) Does the current GIT version
behave<br/> the same way?<br /><br /><blockquote cite="mid:009401ce6832$76653200$632f9600$@kapila@huawei.com"
type="cite"><prewrap="">
Kindly let me know your suggestions.
With Regards,
Amit Kapila.
</pre></blockquote><br /> * Is the patch in a patch format which has context?<br /><br /> Yes.<br /><br /> * Does it
applycleanly to the current git master?<br /><br /> Yes.<br /><br /> * Does it include reasonable tests, necessary doc
patches,etc? <br /><br /> Yes.<br /><br /><p>Read what the patch is supposed to do, and consider:<br /><p>* Does the
patchactually implement that?<br /><p>Yes.<br /> * Do we want that?<br /><br /> Yes.<br /><br /> * Do we already have
it?<br/><br /> No.<br /><br /> * Does it follow SQL spec, or the community-agreed behavior?<br /><br /> No such spec,
followsthe agreed behaviour.<br /><br /> * Does it include pg_dump support (if applicable)?<br /><br /> N/A<br /><br />
*Are there dangers?<br /><br /> No.<br /><br /> * Have all the bases been covered?<br /><br /> According to the above
noteunder Windows, not yet.<br /><br /> The name "persistent.auto.conf" is not yet set in stone.<br />
postgresql.auto.confmight be better.<br /><br /> Apply the patch, compile it and test:<br /><br /> * Does the feature
workas advertised?<br /><br /> Yes, with the noted exception above for log_connections.<br /><br /> * Are there corner
casesthe author has failed to consider?<br /><br /> I can't see any. It is through <br /><br /> * Are there any
assertionfailures or crashes? <br /><br /> No.<br /><br /> * Does the patch slow down simple tests?<br /><br /> No.<br
/><br/> * If it claims to improve performance, does it?<br /><br /> N/A<br /><br /> * Does it slow down other things?
<br/><br /> No.<br /><br /> * Does it follow the project <a class="external text"
href="http://developer.postgresql.org/pgdocs/postgres/source.html"rel="nofollow">coding guidelines</a>?<br /><br />
Mostly,yes. Some nits, though. At some places, you do:<br /><br /> - ParseConfigFile(ConfigFileName, NULL, true, 0,
elevel,&head, &tail);<br /> + ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head,
&tail,NULL);<br/><br /> without a space between the last comma and the NULL keyword.<br /><br /> Also, in the
commentabove ParseConfigFile() there are unnecessary<br /> white space changes and spaces at the end of the line.<br
/><br/> * Are there portability issues?<br /><br /> No.<br /><br /> * Will it work on Windows/BSD etc?<br /><br /> It
should.It doesn't use any platform-dependent code.<br /><br /> * Are the comments sufficient and accurate?<br /><br />
Yes.<br/><br /> * Does it do what it says, correctly?<br /><br /> Yes.<br /><br /> * Does it produce compiler
warnings?<br/><br /> No.<br /><br /> * Can you make it crash?<br /><br /> No.<br /><br /> Best regards,<br /> Zoltán
Böszörményi<br/><br /><pre class="moz-signature" cols="90">--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: <a class="moz-txt-link-freetext" href="http://www.postgresql-support.de">http://www.postgresql-support.de</a>
<aclass="moz-txt-link-freetext" href="http://www.postgresql.at/">http://www.postgresql.at/</a>
</pre>
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Boszormenyi Zoltan
Date:
2013-06-14 05:12 keltezéssel, Amit Kapila írta:
> On Friday, June 14, 2013 3:17 AM Josh Berkus wrote:
>> On 06/13/2013 05:35 AM, Amit Kapila wrote:
>>> On Friday, June 07, 2013 9:45 AM Amit Kapila wrote:
>>>> On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
>>>>> On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila
>> <amit.kapila@huawei.com>
>>>>> wrote:
>>>>>> On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
>>>>>>> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
>>>>>>>> On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
>>>>>> There are 2 options to proceed for this patch for 9.4
>>>>>>
>>>>>> 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
>>>>> existing
>>>>>> review comments
>>>>>> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
>>>>>>
>>>>>> Could you suggest me what could be best way to proceed for this
>>>>> patch?
>>>>>
>>>>> I'm still in favor of some syntax involving ALTER, because it's
>> still
>>>>> true that this behaves more like the existing GUC-setting commands
>>>>> that use ALTER (which change configuration for future sessions)
>>>> rather
>>>>> the ones that use SET (which change the current settings for some
>>>>> period of time).
>>>>
>>>> I will change the patch as per below syntax if there are no
>> objections:
>>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, |
>> 'value'};
>>> Modified patch contains:
>>>
>>> 1. Syntax implemented is
>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
>>> DEFAULT};
>>>
>>> If user specifies DEFAULT, it will remove entry from auto conf file.
>>>
>>> 2. File name to store settings set by ALTER SYSTEM command is still
>>> persistent.auto.conf
>> Why? Shouldn't it just be auto.conf? Or system.auto.conf?
> I had kept same name as it was decided for 9.3, but now as command has changed so it makes more
> sense to change name as well. I think it can be one of what you suggested or postgresql.auto.conf.
>
>
>> I prefer auto.conf, personally.
> Thanks. if no body has any other suggestion I will change it.
>
> I think one of system.auto.conf or postgresql.auto.conf is more appropriate because it either resembles command or
existingconfiguration settings
> file.
I like postgresql.auto.conf better.
>
> With Regards,
> Amit Kapila.
>
>
>
--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de http://www.postgresql.at/
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tuesday, June 18, 2013 3:26 PM Boszormenyi Zoltan wrote:
> Hi,
> review below. Thanks for the review.
>>>>>There are 2 options to proceed for this patch for 9.4
>>>>>1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
>>>>>existing
>>>>>review comments
>>>>>2. Implement new syntax ALTER SYSTEM as proposed in below mail
>>>>>Could you suggest me what could be best way to proceed for this
>>>>>patch?
>>>>I'm still in favor of some syntax involving ALTER, because it's still
>>>>true that this behaves more like the existing GUC-setting commands
>>>>that use ALTER (which change configuration for future sessions)
>>>>rather
>>>>the ones that use SET (which change the current settings for some
>>>>period of time).
>>>I will change the patch as per below syntax if there are no objections:
>>>ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>>Modified patch contains:
>>1. Syntax implemented is
>>ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
>>DEFAULT};
>>If user specifies DEFAULT, it will remove entry from auto conf file.
>>2. File name to store settings set by ALTER SYSTEM command is still
>>persistent.auto.conf
>>3. Added a new page for Alter System command in docs. In compatability
>>section, I had mentioned that
>> this statement is an PostgresQL extension. I had tried to search in
>>SQL-92 spec but couldn't find such command.
>>4. During test of this patch, I had observed one problem for PGC_BACKEND
>>parameters like log_connections.>> If I change these parameters directly in postgresql.conf and then do
>>pg_reload_conf() and reconnect, it will
>> still show the old value. This is observed only on WINDOWS. I will
>>investigate this problem and update you.
>> Due to this problem, I had removed few test cases.
> I can't reproduce this error under Linux (Fedora 19/x86_64).
> The bug might be somewhere else and not caused by your patch
> if manually adding/removing "log_connections = on" from postgresql.conf
> exhibits the same behaviour under Windows. (If I read it correctly,
> you tested it exactly this way.) Does the current GIT version behave
> the same way?
Yes, the current Git has problem which I had reported in below mail:
http://www.postgresql.org/message-id/009801ce68f7$3a746340$af5d29c0$@kapila@
huawei.com
This problem is not related to this patch, it occurs only on WINDOWS or
under EXEC_BACKEND flag.
I think we can discuss this problem in above mail thread.
> * Have all the bases been covered?
> According to the above note under Windows, not yet.
> The name "persistent.auto.conf" is not yet set in stone.
> postgresql.auto.conf might be better.
I think, the decision of name, we can leave to committer with below
possibilities,
as it is very difficult to build consensus on any particular name.
Auto.conf
System.auto.conf
Postgresql.auto.conf
Persistent.auto.conf
> Apply the patch, compile it and test:
> * Does it follow the project coding guidelines?
> Mostly, yes. Some nits, though. At some places, you do:
> - ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head, &tail);
> + ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head,
&tail,NULL);
I think you mean ParseConfigFp()?
without a space between the last comma and the NULL keyword.
> Also, in the comment above ParseConfigFile() there are unnecessary
> white space changes and spaces at the end of the line.
Do you mean to say whitespaces in below text?
! * and absolute-ifying the path name if necessary.
! *
! * While parsing, it records if it has parsed persistent.auto.conf file.
! * This information can be used by the callers to ensure if the parameters
! * set by ALTER SYSTEM SET command will be effective. */
With Regards,
Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Boszormenyi Zoltan
Date:
2013-06-18 14:11 keltezéssel, Amit Kapila írta:
> On Tuesday, June 18, 2013 3:26 PM Boszormenyi Zoltan wrote:
>> Hi,
>> review below.
> Thanks for the review.
>
>
>>>>>> There are 2 options to proceed for this patch for 9.4
>>>>>> 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
>>>>>> existing
>>>>>> review comments
>>>>>> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
>>>>>> Could you suggest me what could be best way to proceed for this
>>>>>> patch?
>>>>> I'm still in favor of some syntax involving ALTER, because it's still
>>>>> true that this behaves more like the existing GUC-setting commands
>>>>> that use ALTER (which change configuration for future sessions)
>>>>> rather
>>>>> the ones that use SET (which change the current settings for some
>>>>> period of time).
>
>>>> I will change the patch as per below syntax if there are no objections:
>>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>>> Modified patch contains:
>>> 1. Syntax implemented is
>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
>>> DEFAULT};
>>> If user specifies DEFAULT, it will remove entry from auto conf file.
>>> 2. File name to store settings set by ALTER SYSTEM command is still
>>> persistent.auto.conf
>>> 3. Added a new page for Alter System command in docs. In compatability
>>> section, I had mentioned that
>>> this statement is an PostgresQL extension. I had tried to search in
>>> SQL-92 spec but couldn't find such command.
>>> 4. During test of this patch, I had observed one problem for PGC_BACKEND
>>> parameters like log_connections.
> >> If I change these parameters directly in postgresql.conf and then do
>>> pg_reload_conf() and reconnect, it will
>>> still show the old value. This is observed only on WINDOWS. I will
>>> investigate this problem and update you.
>>> Due to this problem, I had removed few test cases.
>> I can't reproduce this error under Linux (Fedora 19/x86_64).
>> The bug might be somewhere else and not caused by your patch
>> if manually adding/removing "log_connections = on" from postgresql.conf
>> exhibits the same behaviour under Windows. (If I read it correctly,
>> you tested it exactly this way.) Does the current GIT version behave
>> the same way?
> Yes, the current Git has problem which I had reported in below mail:
>
> http://www.postgresql.org/message-id/009801ce68f7$3a746340$af5d29c0$@kapila@
> huawei.com
>
> This problem is not related to this patch, it occurs only on WINDOWS or
> under EXEC_BACKEND flag.
> I think we can discuss this problem in above mail thread.
>
>
>
>> * Have all the bases been covered?
>> According to the above note under Windows, not yet.
>> The name "persistent.auto.conf" is not yet set in stone.
>> postgresql.auto.conf might be better.
> I think, the decision of name, we can leave to committer with below
> possibilities,
> as it is very difficult to build consensus on any particular name.
>
> Auto.conf
> System.auto.conf
> Postgresql.auto.conf
> Persistent.auto.conf
>
>> Apply the patch, compile it and test:
>
>> * Does it follow the project coding guidelines?
>> Mostly, yes. Some nits, though. At some places, you do:
>> - ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head, &tail);
>> + ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head,
> &tail,NULL);
>
> I think you mean ParseConfigFp()?
Sorry, yes.
>
> without a space between the last comma and the NULL keyword.
>
>> Also, in the comment above ParseConfigFile() there are unnecessary
>> white space changes and spaces at the end of the line.
> Do you mean to say whitespaces in below text?
>
> ! * and absolute-ifying the path name if necessary.
>
> ! *
> ! * While parsing, it records if it has parsed persistent.auto.conf file.
> ! * This information can be used by the callers to ensure if the parameters
> ! * set by ALTER SYSTEM SET command will be effective.
> */
Yes.
Anyway, both would be fixed by a pgindent run. (I think.)
>
> With Regards,
> Amit Kapila.
>
--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de http://www.postgresql.at/
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
Amit, > I think, the decision of name, we can leave to committer with below > possibilities, > as it is very difficult to build consensus on any particular name. > > Auto.conf > System.auto.conf > Postgresql.auto.conf > Persistent.auto.conf Reasons for "auto.conf" as a choice above all of the previous: 1) short 2) significantly different from "postgresql.conf" 3) near the beginning of the alphabet, so it should sort near the top if there are other files in conf/ directory -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tuesday, June 18, 2013 11:25 PM Josh Berkus wrote: > Amit, > > > I think, the decision of name, we can leave to committer with below > > possibilities, > > as it is very difficult to build consensus on any particular name. > > > > Auto.conf > > System.auto.conf > > Postgresql.auto.conf > > Persistent.auto.conf > > Reasons for "auto.conf" as a choice above all of the previous: > > 1) short > 2) significantly different from "postgresql.conf" What is the advantage to keep name significantly different from "postgresql.conf", rather will it be not helpful for use if it is similar to "postgresql.conf"? > 3) near the beginning of the alphabet, so it should sort near the top > if > there are other files in conf/ directory How would user correlate auto.conf contains configuration parameters set by Alter System command? I had kept persistent.auto.conf based on suggestion in this thread that user can easily correlate it with SET PERSISTENT command. So based on that theory I think it would be better to name it as System.auto.conf or Postgresql.auto.conf With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tuesday, June 18, 2013 7:39 PM Boszormenyi Zoltan wrote:
> 2013-06-18 14:11 keltezéssel, Amit Kapila írta:
> > On Tuesday, June 18, 2013 3:26 PM Boszormenyi Zoltan wrote:
> >> Hi,
> >> review below.
> > Thanks for the review.
> >
> >
> >>>>>> There are 2 options to proceed for this patch for 9.4
> >>>>>> 1. Upload the SET PERSISTENT syntax patch for coming CF by
> fixing
> >>>>>> existing
> >>>>>> review comments
> >>>>>> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
> >>>>>> Could you suggest me what could be best way to proceed for this
> >>>>>> patch?
> >>>>> I'm still in favor of some syntax involving ALTER, because it's
> still
> >>>>> true that this behaves more like the existing GUC-setting
> commands
> >>>>> that use ALTER (which change configuration for future sessions)
> >>>>> rather
> >>>>> the ones that use SET (which change the current settings for some
> >>>>> period of time).
> >
> >>>> I will change the patch as per below syntax if there are no
> objections:
> >>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, |
> 'value'};
> >>> Modified patch contains:
> >>> 1. Syntax implemented is
> >>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'
> |
> >>> DEFAULT};
> >>> If user specifies DEFAULT, it will remove entry from auto conf
> file.
> >>> 2. File name to store settings set by ALTER SYSTEM command is still
> >>> persistent.auto.conf
> >>> 3. Added a new page for Alter System command in docs. In
> compatability
> >>> section, I had mentioned that
> >>> this statement is an PostgresQL extension. I had tried to search
> in
> >>> SQL-92 spec but couldn't find such command.
> >>> 4. During test of this patch, I had observed one problem for
> PGC_BACKEND
> >>> parameters like log_connections.
> > >> If I change these parameters directly in postgresql.conf and
> then do
> >>> pg_reload_conf() and reconnect, it will
> >>> still show the old value. This is observed only on WINDOWS. I
> will
> >>> investigate this problem and update you.
> >>> Due to this problem, I had removed few test cases.
> >> I can't reproduce this error under Linux (Fedora 19/x86_64).
> >> The bug might be somewhere else and not caused by your patch
> >> if manually adding/removing "log_connections = on" from
> postgresql.conf
> >> exhibits the same behaviour under Windows. (If I read it correctly,
> >> you tested it exactly this way.) Does the current GIT version behave
> >> the same way?
> > Yes, the current Git has problem which I had reported in below mail:
> >
> > http://www.postgresql.org/message-
> id/009801ce68f7$3a746340$af5d29c0$@kapila@
> > huawei.com
> >
> > This problem is not related to this patch, it occurs only on WINDOWS
> or
> > under EXEC_BACKEND flag.
> > I think we can discuss this problem in above mail thread.
> >
> >
> >
> >> * Have all the bases been covered?
> >> According to the above note under Windows, not yet.
> >> The name "persistent.auto.conf" is not yet set in stone.
> >> postgresql.auto.conf might be better.
> > I think, the decision of name, we can leave to committer with below
> > possibilities,
> > as it is very difficult to build consensus on any particular name.
> >
> > Auto.conf
> > System.auto.conf
> > Postgresql.auto.conf
> > Persistent.auto.conf
> >
> >> Apply the patch, compile it and test:
> >
> >> * Does it follow the project coding guidelines?
> >> Mostly, yes. Some nits, though. At some places, you do:
> >> - ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head,
> &tail);
> >> + ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head,
> > &tail,NULL);
> >
> > I think you mean ParseConfigFp()?
>
> Sorry, yes.
>
> >
> > without a space between the last comma and the NULL keyword.
> >
> >> Also, in the comment above ParseConfigFile() there are unnecessary
> >> white space changes and spaces at the end of the line.
> > Do you mean to say whitespaces in below text?
> >
> > ! * and absolute-ifying the path name if necessary.
> >
> > ! *
> > ! * While parsing, it records if it has parsed persistent.auto.conf
> file.
> > ! * This information can be used by the callers to ensure if the
> parameters
> > ! * set by ALTER SYSTEM SET command will be effective.
> > */
>
> Yes.
>
> Anyway, both would be fixed by a pgindent run. (I think.)
I think pgindent doesn't work on .l file, in anycase I had corrected it
manually.
Find the updated patch attached with this mail.
Kindly let me know if there is anything more needs to be done for this
patch.
With Regards,
Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Peter Eisentraut
Date:
On 6/13/13 5:47 PM, Josh Berkus wrote: >> 2. File name to store settings set by ALTER SYSTEM command is still >> > persistent.auto.conf > Why? Shouldn't it just be auto.conf? Or system.auto.conf? > > I prefer auto.conf, personally. Well, not much about it is automatic, really. It's just set elsewhere.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 06/19/2013 10:48 AM, Peter Eisentraut wrote: > On 6/13/13 5:47 PM, Josh Berkus wrote: >>> 2. File name to store settings set by ALTER SYSTEM command is still >>>> persistent.auto.conf >> Why? Shouldn't it just be auto.conf? Or system.auto.conf? >> >> I prefer auto.conf, personally. > > Well, not much about it is automatic, really. It's just set elsewhere. > True, but can you think of a better word to mean "don't edit this by hand"? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Peter Eisentraut
Date:
On 6/19/13 1:49 PM, Josh Berkus wrote: > On 06/19/2013 10:48 AM, Peter Eisentraut wrote: >> On 6/13/13 5:47 PM, Josh Berkus wrote: >>>> 2. File name to store settings set by ALTER SYSTEM command is still >>>>> persistent.auto.conf >>> Why? Shouldn't it just be auto.conf? Or system.auto.conf? >>> >>> I prefer auto.conf, personally. >> >> Well, not much about it is automatic, really. It's just set elsewhere. >> > > True, but can you think of a better word to mean "don't edit this by hand"? generated_by_server.conf
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Magnus Hagander
Date:
<p dir="ltr"><br /> On Jun 19, 2013 7:55 PM, "Peter Eisentraut" <<a href="mailto:peter_e@gmx.net">peter_e@gmx.net</a>>wrote:<br /> ><br /> > On 6/19/13 1:49 PM, Josh Berkus wrote:<br/> > > On 06/19/2013 10:48 AM, Peter Eisentraut wrote:<br /> > >> On 6/13/13 5:47 PM, Josh Berkuswrote:<br /> > >>>> 2. File name to store settings set by ALTER SYSTEM command is still<br /> > >>>>>persistent.auto.conf<br /> > >>> Why? Shouldn't it just be auto.conf? Or system.auto.conf?<br/> > >>><br /> > >>> I prefer auto.conf, personally.<br /> > >><br />> >> Well, not much about it is automatic, really. It's just set elsewhere.<br /> > >><br /> > ><br/> > > True, but can you think of a better word to mean "don't edit this by hand"?<br /> ><br /> > generated_by_server.conf<pdir="ltr">System.conf? <p dir="ltr">/Magnus
Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]
From
Peter Eisentraut
Date:
On 6/7/13 12:14 AM, Amit Kapila wrote:
> I will change the patch as per below syntax if there are no objections:
>
> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
I do like using ALTER SYSTEM in general, but now that I think about it,
the 9.3 feature to create global settings in pg_db_role_setting would
also have been a candidate for exactly that same syntax if it had been
available. In fact, if we do add ALTER SYSTEM, it might make sense to
recast that feature into that syntax.
It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
or something like that. It's only a small syntax change, so don't worry
about it too much, but let's keep thinking about it.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Magnus Hagander wrote: > On Jun 19, 2013 7:55 PM, "Peter Eisentraut" <peter_e@gmx.net> wrote: > > generated_by_server.conf > > System.conf? alter_system.conf ? -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Josh Berkus <josh@agliodbs.com> writes:
> True, but can you think of a better word to mean "don't edit this by hand"?
The file name is not nearly as important for that as putting in a
header comment "# Don't edit this file by hand."
regards, tom lane
On Wednesday, June 19, 2013 11:30 PM Peter Eisentraut wrote:
> On 6/7/13 12:14 AM, Amit Kapila wrote:
> > I will change the patch as per below syntax if there are no
> objections:
> >
> > ALTER SYSTEM SET configuration_parameter {TO | =} {value, |
> 'value'};
>
> I do like using ALTER SYSTEM in general, but now that I think about it,
> the 9.3 feature to create global settings in pg_db_role_setting would
> also have been a candidate for exactly that same syntax if it had been
> available. In fact, if we do add ALTER SYSTEM, it might make sense to
> recast that feature into that syntax.
>
> It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
> or something like that. It's only a small syntax change, so don't
> worry
> about it too much, but let's keep thinking about it.
Is your intention to extend the syntax by EXTERNAL FILE is for specifying
SCOPE for configuration parameter?
ALTER SYSTEM SET SCOPE FILE
ALTER SYSTEM SET EXTENSION FILE
With Regards,
Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Thursday, June 20, 2013 3:53 AM Tom Lane wrote: > Josh Berkus <josh@agliodbs.com> writes: > > True, but can you think of a better word to mean "don't edit this by > hand"? > > The file name is not nearly as important for that as putting in a > header comment "# Don't edit this file by hand." Currently header comment is: # Do not edit this file manually! # It will be overwritten by ALTER SYSTEM command. Kindly let me know your suggestion about the above comment in header? With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Thursday, June 20, 2013 12:32 AM Alvaro Herrera wrote: > Magnus Hagander wrote: > > On Jun 19, 2013 7:55 PM, "Peter Eisentraut" <peter_e@gmx.net> wrote: > > > > generated_by_server.conf > > > > System.conf? > > alter_system.conf ? All the names proposed in this thread are as follows: Auto.conf - 1 Vote (Josh) System.auto.conf - 1 Vote (Josh) Postgresql.auto.conf - 2 Votes (Zoltan, Amit) Persistent.auto.conf - 0 Vote generated_by_server.conf - 1 Vote (Peter E) System.conf - 1 Vote (Magnus) alter_system.conf - 1 Vote (Alvaro) I had consolidated the list, so that it would be helpful for committer to decide among proposed names for this file. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Amit Kapila wrote: > Currently header comment is: > # Do not edit this file manually! > # It will be overwritten by ALTER SYSTEM command. > > Kindly let me know your suggestion about the above comment in header? This seems perfect to me. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Wed, Jun 19, 2013 at 1:59 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On 6/7/13 12:14 AM, Amit Kapila wrote:
>> I will change the patch as per below syntax if there are no objections:
>>
>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>
> I do like using ALTER SYSTEM in general, but now that I think about it,
> the 9.3 feature to create global settings in pg_db_role_setting would
> also have been a candidate for exactly that same syntax if it had been
> available. In fact, if we do add ALTER SYSTEM, it might make sense to
> recast that feature into that syntax.
>
> It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
> or something like that. It's only a small syntax change, so don't worry
> about it too much, but let's keep thinking about it.
I think that anything we want to add should either go before SET or
after the value, such as:
ALTER SYSTEM SET configuration_parameter = 'value' IN FILE 'myconf.conf';
ALTER SYSTEM CONFIGURATION FILE SET configuration_parameter = 'file';
ALTER SYSTEM CONFIGURATION FILE 'myconf.conf' SET
configuration_parameter = 'file';
I agree we shouldn't back ourselves into a syntactic corner.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Sat, Jun 22, 2013 at 12:11 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, Jun 19, 2013 at 1:59 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>> On 6/7/13 12:14 AM, Amit Kapila wrote:
>>> I will change the patch as per below syntax if there are no objections:
>>>
>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>>
>> I do like using ALTER SYSTEM in general, but now that I think about it,
>> the 9.3 feature to create global settings in pg_db_role_setting would
>> also have been a candidate for exactly that same syntax if it had been
>> available. In fact, if we do add ALTER SYSTEM, it might make sense to
>> recast that feature into that syntax.
>>
>> It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
>> or something like that. It's only a small syntax change, so don't worry
>> about it too much, but let's keep thinking about it.
>
> I think that anything we want to add should either go before SET or
> after the value, such as:
>
> ALTER SYSTEM SET configuration_parameter = 'value' IN FILE 'myconf.conf';
> ALTER SYSTEM CONFIGURATION FILE SET configuration_parameter = 'file';
> ALTER SYSTEM CONFIGURATION FILE 'myconf.conf' SET
> configuration_parameter = 'file';
>
> I agree we shouldn't back ourselves into a syntactic corner.
Maybe this idea has been already discussed, but why don't we just
add the function like update_config_file(parameter, value)? We can
commit the core of the patch with that function API first, and then
we can discuss the syntax and add another API like ALTER SYSTEM
later.
We now have set_config() function to set the parameter,
so adding the function for this feature should not be a surprise.
If the name of the function is, for example, update_conf_file,
most users would think that it's intuitive even if SIGHUP is not
automatically sent immediately. We don't need to emit
the message like 'This setting change will not be loaded until SIGHUP'.
Regards,
--
Fujii Masao
On Fri, Jun 21, 2013 at 12:56 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Sat, Jun 22, 2013 at 12:11 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Wed, Jun 19, 2013 at 1:59 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>>> On 6/7/13 12:14 AM, Amit Kapila wrote:
>>>> I will change the patch as per below syntax if there are no objections:
>>>>
>>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>>>
>>> I do like using ALTER SYSTEM in general, but now that I think about it,
>>> the 9.3 feature to create global settings in pg_db_role_setting would
>>> also have been a candidate for exactly that same syntax if it had been
>>> available. In fact, if we do add ALTER SYSTEM, it might make sense to
>>> recast that feature into that syntax.
>>>
>>> It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
>>> or something like that. It's only a small syntax change, so don't worry
>>> about it too much, but let's keep thinking about it.
>>
>> I think that anything we want to add should either go before SET or
>> after the value, such as:
>>
>> ALTER SYSTEM SET configuration_parameter = 'value' IN FILE 'myconf.conf';
>> ALTER SYSTEM CONFIGURATION FILE SET configuration_parameter = 'file';
>> ALTER SYSTEM CONFIGURATION FILE 'myconf.conf' SET
>> configuration_parameter = 'file';
>>
>> I agree we shouldn't back ourselves into a syntactic corner.
>
> Maybe this idea has been already discussed, but why don't we just
> add the function like update_config_file(parameter, value)? We can
> commit the core of the patch with that function API first, and then
> we can discuss the syntax and add another API like ALTER SYSTEM
> later.
>
> We now have set_config() function to set the parameter,
> so adding the function for this feature should not be a surprise.
> If the name of the function is, for example, update_conf_file,
> most users would think that it's intuitive even if SIGHUP is not
> automatically sent immediately. We don't need to emit
> the message like 'This setting change will not be loaded until SIGHUP'.
I could certainly support that plan. I'm satisfied with the ALTER
SYSTEM syntax and feel that might find other plausible uses, but
functional notation would be OK with me, too.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Thu, Jun 20, 2013 at 12:18 AM, Amit Kapila <amit.kapila@huawei.com> wrote: > Auto.conf - 1 Vote (Josh) > System.auto.conf - 1 Vote (Josh) > Postgresql.auto.conf - 2 Votes (Zoltan, Amit) > Persistent.auto.conf - 0 Vote > generated_by_server.conf - 1 Vote (Peter E) > System.conf - 1 Vote (Magnus) > alter_system.conf - 1 Vote (Alvaro) > > I had consolidated the list, so that it would be helpful for committer to > decide among proposed names for this file. I'll also vote for postgresql.auto.conf. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit kapila
Date:
On Friday, June 21, 2013 11:48 PM Robert Haas wrote: On Thu, Jun 20, 2013 at 12:18 AM, Amit Kapila <amit.kapila@huawei.com> wrote: >> Auto.conf - 1 Vote (Josh) >> System.auto.conf - 1 Vote (Josh) >> Postgresql.auto.conf - 2 Votes (Zoltan, Amit) >> Persistent.auto.conf - 0 Vote >> generated_by_server.conf - 1 Vote (Peter E) >> System.conf - 1 Vote (Magnus) >> alter_system.conf - 1 Vote (Alvaro) > >> I had consolidated the list, so that it would be helpful for committer to >> decide among proposed names for this file. > I'll also vote for postgresql.auto.conf. Thanks to all of you for suggesting meaningful names. I will change the name of file to postgresql.auto.conf. Kindly letme know if there is any objection to it. With Regards, Amit Kapila.
On Friday, June 21, 2013 11:43 PM Robert Haas wrote:
On Fri, Jun 21, 2013 at 12:56 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Sat, Jun 22, 2013 at 12:11 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Wed, Jun 19, 2013 at 1:59 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>>> On 6/7/13 12:14 AM, Amit Kapila wrote:
>>>>> I will change the patch as per below syntax if there are no objections:
>>>>>
>>>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>>>>
>>>> I do like using ALTER SYSTEM in general, but now that I think about it,
>>>> the 9.3 feature to create global settings in pg_db_role_setting would
>>>> also have been a candidate for exactly that same syntax if it had been
>>>> available. In fact, if we do add ALTER SYSTEM, it might make sense to
>>>> recast that feature into that syntax.
>>>>
>>>> It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
>>>> or something like that. It's only a small syntax change, so don't worry
>>>> about it too much, but let's keep thinking about it.
>>>
>>> I think that anything we want to add should either go before SET or
>>> after the value, such as:
>>>
>>> ALTER SYSTEM SET configuration_parameter = 'value' IN FILE 'myconf.conf';
>>> ALTER SYSTEM CONFIGURATION FILE SET configuration_parameter = 'file';
>>> ALTER SYSTEM CONFIGURATION FILE 'myconf.conf' SET
>>> configuration_parameter = 'file';
>>>
>>> I agree we shouldn't back ourselves into a syntactic corner.
>>
>> Maybe this idea has been already discussed, but why don't we just
>> add the function like update_config_file(parameter, value)? We can
>> commit the core of the patch with that function API first, and then
>> we can discuss the syntax and add another API like ALTER SYSTEM
>> later.
>>
>> We now have set_config() function to set the parameter,
>> so adding the function for this feature should not be a surprise.
>> If the name of the function is, for example, update_conf_file,
>> most users would think that it's intuitive even if SIGHUP is not
>> automatically sent immediately. We don't need to emit
>> the message like 'This setting change will not be loaded until SIGHUP'.
> I could certainly support that plan. I'm satisfied with the ALTER
> SYSTEM syntax and feel that might find other plausible uses, but
> functional notation would be OK with me, too.
I can update the patch to have a function as suggested by Fujii-san if it can be decided
that for the first committed version it will be sufficient.
OTOH already we already have consensus on ALTER SYSTEM syntax apart from few extra keywords to make it more
meaningful/extendible.
I think we can consider the current syntax (ALTER SYSTEM SET ..) and make that behavior as default for writing to auto
file.
In future we can extend it with other keywords depending on usage.
With Regards,
Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Saturday, June 22, 2013 9:47 AM Amit kapila wrote: > On Friday, June 21, 2013 11:48 PM Robert Haas wrote: > On Thu, Jun 20, 2013 at 12:18 AM, Amit Kapila <amit.kapila@huawei.com> > wrote: > >> Auto.conf - 1 Vote (Josh) > >> System.auto.conf - 1 Vote (Josh) > >> Postgresql.auto.conf - 2 Votes (Zoltan, Amit) > >> Persistent.auto.conf - 0 Vote > >> generated_by_server.conf - 1 Vote (Peter E) > >> System.conf - 1 Vote (Magnus) > >> alter_system.conf - 1 Vote (Alvaro) > > > >> I had consolidated the list, so that it would be helpful for > committer to > >> decide among proposed names for this file. > > > I'll also vote for postgresql.auto.conf. > > Thanks to all of you for suggesting meaningful names. I will change > the name of file to postgresql.auto.conf. > Kindly let me know if there is any objection to it. I have changed the file name to postgresql.auto.conf and I have changed the name of SetPersistentLock to AutoFileLock. Zoltan, Could you please once check the attached Patch if you have time, as now all the things are resolved except for small doubt in syntax extensibility which can be changed based on final decision. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
>> I'll also vote for postgresql.auto.conf. > > Thanks to all of you for suggesting meaningful names. I will change the name of file to postgresql.auto.conf. > Kindly let me know if there is any objection to it. There is no name you could come up with for which there would not be an objection. Please plow ahead. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Wednesday, June 26, 2013 5:03 AM Josh Berkus wrote: > > >> I'll also vote for postgresql.auto.conf. > > > > Thanks to all of you for suggesting meaningful names. I will change > the name of file to postgresql.auto.conf. > > Kindly let me know if there is any objection to it. > > There is no name you could come up with for which there would not be an > objection. Please plow ahead. Thanks. I had updated the patch and sent it. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Amit Kapila escribió:
> I have changed the file name to postgresql.auto.conf and I have changed the
> name of SetPersistentLock to AutoFileLock.
>
> Zoltan,
>
> Could you please once check the attached Patch if you have time, as now all
> the things are resolved except for small doubt in syntax extensibility
> which can be changed based on final decision.
There are a bunch of whitespace problems, as you would notice with "git
diff --check" or "git show --color". Could you please clean that up?
Also, some of the indentation in various places is not right; perhaps
you could fix that by running pgindent over the source files you
modified (this is easily visible by eyeballing the git diff output;
stuff is misaligned in pretty obvious ways.)
Random minor other comments:
+ use of <xref linkend="SQL-ALTER SYSTEM">
this SGML link cannot possibly work. Please run "make check" in the
doc/src/sgml directory.
Examples in alter_system.sgml have a typo "SYTEM". Same file has "or or".
Also in that file, The name of an configuration parameter that exist in <filename>postgresql.conf</filename>.
the parameter needn't exist in that file to be settable, right?
<refnamediv> <refname>ALTER SYSTEM</refname> <refpurpose>change a server configuration
parameter</refpurpose></refnamediv>
Perhaps "permanently change .."?
Also, some parameters require a restart, not a reload; maybe we should
direct the user somewhere else to learn how to freshen up the
configuration after calling the command.
+ /* skip auto conf temporary file */
+ if (strncmp(de->d_name,
+ PG_AUTOCONF_FILENAME ".",
+ sizeof(PG_AUTOCONF_FILENAME)) == 0)
+ continue;
What's the dot doing there?
+ if ((stat(AutoConfFileName, &st) == -1))
+ ereport(elevel,
+ (errmsg("configuration parameters changed with ALTER SYSTEM command before restart of server "
+ "will not be effective as \"%s\" file is not accessible.", PG_AUTOCONF_FILENAME)));
+ else
+ ereport(elevel,
+ (errmsg("configuration parameters changed with ALTER SYSTEM command before restart of server "
+ "will not be effective as default include directive for \"%s\" folder is not present in
postgresql.conf",PG_AUTOCONF_DIR)));
These messages should be split. Perhaps have the "will not be
effective" in the errmsg() and the reason as part of errdetail()? Also,
I'm not really sure about doing another stat() on the file after parsing
failed; I think the parse routine should fill some failure context
information, instead of having this code trying to reproduce the failure
to know what to report. Maybe something like the SlruErrorCause enum,
not sure.
Similarly,
+ /*
+ * record if the file currently being parsed is postgresql.auto.conf,
+ * so that it can be later used to give warning if it doesn't parse
+ * it.
+ */
+ snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR, PG_AUTOCONF_FILENAME);
+ ConfigAutoFileName = AbsoluteConfigLocation(Filename, ConfigFileName);
+ if (depth == 1 && strcmp(ConfigAutoFileName, config_file) == 0)
+ *is_config_dir_parsed = true;
this seems very odd. The whole "is_config_dir_parsed" mechanism smells
strange to me, really. I think the caller should be in charge of
keeping track of this, but I'm not sure. ParseConfigFp() would have no
way of knowing, and in one place we're calling that routine precisely to
parse the auto file.
Thanks,
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Simon Riggs
Date:
On 22 June 2013 05:17, Amit kapila <amit.kapila@huawei.com> wrote:
On Friday, June 21, 2013 11:48 PM Robert Haas wrote:Thanks to all of you for suggesting meaningful names. I will change the name of file to postgresql.auto.conf.
On Thu, Jun 20, 2013 at 12:18 AM, Amit Kapila <amit.kapila@huawei.com> wrote:
>> Auto.conf - 1 Vote (Josh)
>> System.auto.conf - 1 Vote (Josh)
>> Postgresql.auto.conf - 2 Votes (Zoltan, Amit)
>> Persistent.auto.conf - 0 Vote
>> generated_by_server.conf - 1 Vote (Peter E)
>> System.conf - 1 Vote (Magnus)
>> alter_system.conf - 1 Vote (Alvaro)
>
>> I had consolidated the list, so that it would be helpful for committer to
>> decide among proposed names for this file.
> I'll also vote for postgresql.auto.conf.
Kindly let me know if there is any objection to it.
postgresql.auto.conf is similar enough to postgresql.conf that it will prevent tab completion from working as it does now. As a result it will cause people to bring that file up for editing when we wish to avoid that.
So for me the name is not arbitrary because of that point and we should avoid the current choice.
Can we go for auto.postgresql.conf instead .... almost identical, just significantly visually different and not interfering with tab completion?
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Wednesday, July 03, 2013 11:42 AM Simon Riggs wrote: On 22 June 2013 05:17, Amit kapila <amit.kapila@huawei.com> wrote: On Friday, June 21, 2013 11:48 PM Robert Haas wrote: On Thu, Jun 20, 2013 at 12:18 AM, Amit Kapila <amit.kapila@huawei.com> wrote: >>>> Auto.conf - 1 Vote (Josh) >>>> System.auto.conf - 1 Vote (Josh) >>>> Postgresql.auto.conf - 2 Votes (Zoltan, Amit) >>>> Persistent.auto.conf - 0 Vote >>>> generated_by_server.conf - 1 Vote (Peter E) >>>> System.conf - 1 Vote (Magnus) >>>> alter_system.conf - 1 Vote (Alvaro) > >>>> I had consolidated the list, so that it would be helpful for committer to >>>> decide among proposed names for this file. >>> I'll also vote for postgresql.auto.conf. >> Thanks to all of you for suggesting meaningful names. I will change the name of file to postgresql.auto.conf. >> Kindly let me know if there is any objection to it. > postgresql.auto.conf is similar enough to postgresql.conf that it will prevent tab completion from working as it does now. As a result it will cause > people to bring that file up for editing when we wish to avoid that. This new file postgresql.auto.conf will be created inside a new directory "config", so both will be in different directories. Will there be any confusion with tab completion for different directories? > So for me the name is not arbitrary because of that point and we should avoid the current choice. > Can we go for auto.postgresql.conf instead .... almost identical, just significantly visually different and not interfering with tab completion? With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote: > Amit Kapila escribió: > > > I have changed the file name to postgresql.auto.conf and I have > changed the > > name of SetPersistentLock to AutoFileLock. > > > > Zoltan, > > > > Could you please once check the attached Patch if you have time, as > now all > > the things are resolved except for small doubt in syntax > extensibility > > which can be changed based on final decision. > > There are a bunch of whitespace problems, as you would notice with "git > diff --check" or "git show --color". Could you please clean that up? > Also, some of the indentation in various places is not right; perhaps > you could fix that by running pgindent over the source files you > modified (this is easily visible by eyeballing the git diff output; > stuff is misaligned in pretty obvious ways.) > > Random minor other comments: > > + use of <xref linkend="SQL-ALTER SYSTEM"> > > this SGML link cannot possibly work. Please run "make check" in the > doc/src/sgml directory. > > Examples in alter_system.sgml have a typo "SYTEM". Same file has "or > or". I will fix above issues. > Also in that file, > The name of an configuration parameter that exist in > <filename>postgresql.conf</filename>. > the parameter needn't exist in that file to be settable, right? > > <refnamediv> > <refname>ALTER SYSTEM</refname> > <refpurpose>change a server configuration parameter</refpurpose> > </refnamediv> Yes you are right. How about below line, same parameter description as SET command Name of a settable run-time parameter. Available parameters are documented in Chapter 18 > Perhaps "permanently change .."? I am not sure, if adding "permanently" is appropriate here, as one could also interpret it, that after parameter is changed with this command, it can never be changed again. > Also, some parameters require a restart, not a reload; maybe we should > direct the user somewhere else to learn how to freshen up the > configuration after calling the command. Below link contains useful information in this regard: http://www.postgresql.org/docs/devel/static/config-setting.html Particularly refer below para in above link (the below text is for your reference to see if above link is useful): The configuration file is reread whenever the main server process receives a SIGHUP signal; this is most easily done by running pg_ctl reload from the command-line or by calling the SQL function pg_reload_conf(). The main server process also propagates this signal to all currently running server processes so that existing sessions also get the new value. Alternatively, you can send the signal to a single server process directly. Some parameters can only be set at server start; any changes to their entries in the configuration file will be ignored until the server is restarted. Invalid parameter settings in the configuration file are likewise ignored (but logged) during SIGHUP processing. > > + /* skip auto conf temporary file */ > + if (strncmp(de->d_name, > + PG_AUTOCONF_FILENAME ".", > + sizeof(PG_AUTOCONF_FILENAME)) == 0) > + continue; > > What's the dot doing there? This was from previous version of patch, now it is not required, I will remove it. > > + if ((stat(AutoConfFileName, &st) == -1)) > + ereport(elevel, > + (errmsg("configuration parameters changed with ALTER > SYSTEM command before restart of server " > + "will not be effective as \"%s\" file is not > accessible.", PG_AUTOCONF_FILENAME))); > + else > + ereport(elevel, > + (errmsg("configuration parameters changed with ALTER > SYSTEM command before restart of server " > + "will not be effective as default include > directive for \"%s\" folder is not present in postgresql.conf", > PG_AUTOCONF_DIR))); > > These messages should be split. Perhaps have the "will not be > effective" in the errmsg() and the reason as part of errdetail()? How about changing to below messages ereport(elevel, (errmsg("configuration parameters changed with ALTER SYSTEM command will not be effective") errdetail("file \"%s\" containing configuration parameters is not accessible.", PG_AUTOCONF_FILENAME))); ereport(elevel, (errmsg("configuration parameters changed with ALTER SYSTEM command will not be effective") errdetail("default include directive for \"%s\" folder is not present in postgresql.conf", PG_AUTOCONF_DIR))); > Also, > I'm not really sure about doing another stat() on the file after > parsing > failed; I think the parse routine should fill some failure context > information, instead of having this code trying to reproduce the > failure > to know what to report. Maybe something like the SlruErrorCause enum, > not sure. It might not be feasible with current implementation as currently it just note down whether it has parsed file with name postgresql.auto.conf and then in outer function takes decision based on that parameter. However if we change current implementation for knowing whether it has parsed postgresql.auto.conf, then might be it is possible. See in below point one suggestion to achieve the same. > Similarly, > > + /* > + * record if the file currently being parsed is > postgresql.auto.conf, > + * so that it can be later used to give warning if it doesn't > parse > + * it. > + */ > + snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR, > PG_AUTOCONF_FILENAME); > + ConfigAutoFileName = AbsoluteConfigLocation(Filename, > ConfigFileName); > + if (depth == 1 && strcmp(ConfigAutoFileName, config_file) == 0) > + *is_config_dir_parsed = true; > > this seems very odd. The whole "is_config_dir_parsed" mechanism smells > strange to me, really. I think the caller should be in charge of > keeping track of this, but I'm not sure. ParseConfigFp() would have no > way of knowing, and in one place we're calling that routine precisely > to > parse the auto file. How about changing it in way such that 1. we introduce a new enum ConfigDirParseError (3 values a. config_dir_not_parsed, config_dir_parsed, config_file_not_accessible) and a new static variable confdirerr. 2. variable should be initialized with config_dir_not_parsed in function ParseConfigFile() 3. Now in function ParseConfigDirectory, we compare that passed directory name is "config" , then we set confdirerr = config_dir_parsed 4. Further in function ParseConfigDirectory, check during ReadDir, if it has read file postgresql.auto.conf I think above should handle both error cases a. include directive for config file is commented or not present; b. file postgresql.auto.conf is not accessible With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Simon Riggs
Date:
On 3 July 2013 07:45, Amit Kapila <amit.kapila@huawei.com> wrote:
Simon Riggs http://www.2ndQuadrant.com/> postgresql.auto.conf is similar enough to postgresql.conf that it willThis new file postgresql.auto.conf will be created inside a new directory
prevent tab completion from working as it does now. As a result it will
cause
> people to bring that file up for editing when we wish to avoid that.
"config", so both will be in different directories.
Will there be any confusion with tab completion for different directories?
Tab completion will not be confused, no.
But I think everything else will be. How will I know that some settings have been set by ALTER SYSTEM and some by other means?
Sounds to me like a recipe for chaos. I hope nobody is calling for a commit on this anytime soon. I think the whole thing needs some careful thought and review before we commit this.
--
PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Wednesday, July 03, 2013 6:29 PM Simon Riggs wrote: On 3 July 2013 07:45, Amit Kapila <amit.kapila@huawei.com> wrote: >>> postgresql.auto.conf is similar enough to postgresql.conf that it will prevent tab completion from working as it does now. As a result it will cause >>> people to bring that file up for editing when we wish to avoid that. >>This new file postgresql.auto.conf will be created inside a new directory >>"config", so both will be in different directories. >> Will there be any confusion with tab completion for different directories? > Tab completion will not be confused, no. > But I think everything else will be. >How will I know that some settings have been set by ALTER SYSTEM and some by other means? Other means by hand editing postgresql.auto.conf? If not we can check in pg_settings, it shows sourcefile. So if the setting is from new file postgresql.auto.conf, this means it is set by ALTER SYSTEM. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
> Amit Kapila escribió:
>
> > I have changed the file name to postgresql.auto.conf and I have
> changed the
> > name of SetPersistentLock to AutoFileLock.
> >
> > Zoltan,
> >
> > Could you please once check the attached Patch if you have time, as
> now all
> > the things are resolved except for small doubt in syntax
> extensibility
> > which can be changed based on final decision.
>
> There are a bunch of whitespace problems, as you would notice with "git
> diff --check" or "git show --color". Could you please clean that up?
Fixed all whitespaces.
> Also, some of the indentation in various places is not right; perhaps
> you could fix that by running pgindent over the source files you
> modified (this is easily visible by eyeballing the git diff output;
> stuff is misaligned in pretty obvious ways.)
Fixed, by running pgindent
> Random minor other comments:
>
> + use of <xref linkend="SQL-ALTER SYSTEM">
>
> this SGML link cannot possibly work. Please run "make check" in the
> doc/src/sgml directory.
Fixed, make check passes now.
> Examples in alter_system.sgml have a typo "SYTEM". Same file has "or
> or".
Fixed.
> Also in that file,
> The name of an configuration parameter that exist in
> <filename>postgresql.conf</filename>.
> the parameter needn't exist in that file to be settable, right?
Changed to below text:
Name of a settable run-time parameter. Available parameters are documented
in <xref linkend="runtime-config">.
> <refnamediv>
> <refname>ALTER SYSTEM</refname>
> <refpurpose>change a server configuration parameter</refpurpose>
> </refnamediv>
>
> Perhaps "permanently change .."?
Not changed.
>
> Also, some parameters require a restart, not a reload; maybe we should
> direct the user somewhere else to learn how to freshen up the
> configuration after calling the command.
>
> + /* skip auto conf temporary file */
> + if (strncmp(de->d_name,
> + PG_AUTOCONF_FILENAME ".",
> + sizeof(PG_AUTOCONF_FILENAME)) == 0)
> + continue;
>
> What's the dot doing there?
Fixed, removed dot.
>
>
> + if ((stat(AutoConfFileName, &st) == -1))
> + ereport(elevel,
> + (errmsg("configuration parameters changed with ALTER
> SYSTEM command before restart of server "
> + "will not be effective as \"%s\" file is not
> accessible.", PG_AUTOCONF_FILENAME)));
> + else
> + ereport(elevel,
> + (errmsg("configuration parameters changed with ALTER
> SYSTEM command before restart of server "
> + "will not be effective as default include
> directive for \"%s\" folder is not present in postgresql.conf",
> PG_AUTOCONF_DIR)));
>
> These messages should be split. Perhaps have the "will not be
> effective" in the errmsg() and the reason as part of errdetail()?
Okay, changed as per suggestion.
> Also,
> I'm not really sure about doing another stat() on the file after
> parsing
> failed; I think the parse routine should fill some failure context
> information, instead of having this code trying to reproduce the
> failure
> to know what to report. Maybe something like the SlruErrorCause enum,
> not sure.
>
> Similarly,
>
> + /*
> + * record if the file currently being parsed is
> postgresql.auto.conf,
> + * so that it can be later used to give warning if it doesn't
> parse
> + * it.
> + */
> + snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR,
> PG_AUTOCONF_FILENAME);
> + ConfigAutoFileName = AbsoluteConfigLocation(Filename,
> ConfigFileName);
> + if (depth == 1 && strcmp(ConfigAutoFileName, config_file) == 0)
> + *is_config_dir_parsed = true;
>
> this seems very odd. The whole "is_config_dir_parsed" mechanism smells
> strange to me, really. I think the caller should be in charge of
> keeping track of this, but I'm not sure. ParseConfigFp() would have no
> way of knowing, and in one place we're calling that routine precisely
> to
> parse the auto file.
Changed by adding new enum AutoConfErrorCause. Now is_config_dir_parsed is
removed from code.
Kindly let me know if this way is better than previous?
Apart from above I have fixed one issue in function
AlterSystemSetConfigFile(), called FreeConfigVariables().
With Regards,
Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Fujii Masao
Date:
On Fri, Jul 5, 2013 at 2:19 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
>> Amit Kapila escribió:
>>
>> > I have changed the file name to postgresql.auto.conf and I have
>> changed the
>> > name of SetPersistentLock to AutoFileLock.
>> >
>> > Zoltan,
>> >
>> > Could you please once check the attached Patch if you have time, as
>> now all
>> > the things are resolved except for small doubt in syntax
>> extensibility
>> > which can be changed based on final decision.
>>
>> There are a bunch of whitespace problems, as you would notice with "git
>> diff --check" or "git show --color". Could you please clean that up?
>
> Fixed all whitespaces.
>
>> Also, some of the indentation in various places is not right; perhaps
>> you could fix that by running pgindent over the source files you
>> modified (this is easily visible by eyeballing the git diff output;
>> stuff is misaligned in pretty obvious ways.)
>
> Fixed, by running pgindent
>
>
>
>> Random minor other comments:
>>
>> + use of <xref linkend="SQL-ALTER SYSTEM">
>>
>> this SGML link cannot possibly work. Please run "make check" in the
>> doc/src/sgml directory.
>
> Fixed, make check passes now.
>
>> Examples in alter_system.sgml have a typo "SYTEM". Same file has "or
>> or".
>
> Fixed.
>
>> Also in that file,
>> The name of an configuration parameter that exist in
>> <filename>postgresql.conf</filename>.
>> the parameter needn't exist in that file to be settable, right?
>
> Changed to below text:
> Name of a settable run-time parameter. Available parameters are documented
> in <xref linkend="runtime-config">.
>
>
>> <refnamediv>
>> <refname>ALTER SYSTEM</refname>
>> <refpurpose>change a server configuration parameter</refpurpose>
>> </refnamediv>
>>
>> Perhaps "permanently change .."?
>
> Not changed.
>
>>
>> Also, some parameters require a restart, not a reload; maybe we should
>> direct the user somewhere else to learn how to freshen up the
>> configuration after calling the command.
>>
>> + /* skip auto conf temporary file */
>> + if (strncmp(de->d_name,
>> + PG_AUTOCONF_FILENAME ".",
>> + sizeof(PG_AUTOCONF_FILENAME)) == 0)
>> + continue;
>>
>> What's the dot doing there?
>
> Fixed, removed dot.
>>
>>
>> + if ((stat(AutoConfFileName, &st) == -1))
>> + ereport(elevel,
>> + (errmsg("configuration parameters changed with ALTER
>> SYSTEM command before restart of server "
>> + "will not be effective as \"%s\" file is not
>> accessible.", PG_AUTOCONF_FILENAME)));
>> + else
>> + ereport(elevel,
>> + (errmsg("configuration parameters changed with ALTER
>> SYSTEM command before restart of server "
>> + "will not be effective as default include
>> directive for \"%s\" folder is not present in postgresql.conf",
>> PG_AUTOCONF_DIR)));
>>
>> These messages should be split. Perhaps have the "will not be
>> effective" in the errmsg() and the reason as part of errdetail()?
>
> Okay, changed as per suggestion.
>
>> Also,
>> I'm not really sure about doing another stat() on the file after
>> parsing
>> failed; I think the parse routine should fill some failure context
>> information, instead of having this code trying to reproduce the
>> failure
>> to know what to report. Maybe something like the SlruErrorCause enum,
>> not sure.
>>
>> Similarly,
>>
>> + /*
>> + * record if the file currently being parsed is
>> postgresql.auto.conf,
>> + * so that it can be later used to give warning if it doesn't
>> parse
>> + * it.
>> + */
>> + snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR,
>> PG_AUTOCONF_FILENAME);
>> + ConfigAutoFileName = AbsoluteConfigLocation(Filename,
>> ConfigFileName);
>> + if (depth == 1 && strcmp(ConfigAutoFileName, config_file) == 0)
>> + *is_config_dir_parsed = true;
>>
>> this seems very odd. The whole "is_config_dir_parsed" mechanism smells
>> strange to me, really. I think the caller should be in charge of
>> keeping track of this, but I'm not sure. ParseConfigFp() would have no
>> way of knowing, and in one place we're calling that routine precisely
>> to
>> parse the auto file.
>
> Changed by adding new enum AutoConfErrorCause. Now is_config_dir_parsed is
> removed from code.
> Kindly let me know if this way is better than previous?
>
> Apart from above I have fixed one issue in function
> AlterSystemSetConfigFile(), called FreeConfigVariables().
I got the following compile warnings.
guc.c:5187: warning: no previous prototype for 'validate_conf_option'
preproc.y:7746.2-31: warning: type clash on default action: <str> != <>
The make installcheck failed when I ran it against the server with
wal_level = hot_standby. The regression.diff is
------------------------------------
*** 27,35 **** (1 row)
show wal_level;
! wal_level
! -----------
! minimal (1 row)
show authentication_timeout;
--- 27,35 ---- (1 row)
show wal_level;
! wal_level
! -------------
! hot_standby (1 row)
show authentication_timeout;
------------------------------------
The regression test of ALTER SYSTEM calls pg_sleep(1) six times.
Those who dislike the regression test patches which were proposed
in this CF might dislike this repeat of pg_sleep(1) because it would
increase the time of regression test.
+ /* skip auto conf temporary file */
+ if (strncmp(de->d_name,
+ PG_AUTOCONF_FILENAME,
+ sizeof(PG_AUTOCONF_FILENAME)) == 0)
+ continue;
Why do we need to exclude the auto conf file from the backup?
I think that it should be included in the backup as well as normal
postgresql.conf.
+ autofile = fopen(path, PG_BINARY_W);
+ if (autofile == NULL)
+ {
+ fprintf(stderr, _("%s: could not open file \"%s\" for writing: %s\n"),
+ progname, path, strerror(errno));
+ exit_nicely();
+ }
+
+ if (fputs("# Do not edit this file manually! It is overwritten by
the ALTER SYSTEM command \n",
+ autofile) < 0)
+ {
+ fprintf(stderr, _("%s: could not write file \"%s\": %s\n"),
+ progname, path, strerror(errno));
+ exit_nicely();
+ }
+
+ if (fclose(autofile))
+ {
+ fprintf(stderr, _("%s: could not close file \"%s\": %s\n"),
+ progname, path, strerror(errno));
+ exit_nicely();
+ }
You can simplify the code by using writefile().
Regards,
--
Fujii Masao
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit kapila
Date:
On Friday, July 12, 2013 12:02 AM Fujii Masao wrote:
On Fri, Jul 5, 2013 at 2:19 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
>> Amit Kapila escribió:
>>
> I got the following compile warnings.
> guc.c:5187: warning: no previous prototype for 'validate_conf_option'
> preproc.y:7746.2-31: warning: type clash on default action: <str> != <>
I generally use windows as dev environment, it hasn't shown these warnings.
I shall check in linux and correct the same.
> The make installcheck failed when I ran it against the server with
> wal_level = hot_standby. The regression.diff is
> ------------------------------------
> *** 27,35 ****
> (1 row)
> show wal_level;
> ! wal_level
> ! -----------
> ! minimal
> (1 row)
> show authentication_timeout;
> --- 27,35 ----
> (1 row)
> show wal_level;
> ! wal_level
> ! -------------
> ! hot_standby
> (1 row)
> show authentication_timeout;
> ------------------------------------
The tests in alter_system.sql are dependent on default values of postgresql.conf, which is okay for make check
but not for installcheck. I shall remove that dependency from testcases.
> The regression test of ALTER SYSTEM calls pg_sleep(1) six times.
> Those who dislike the regression test patches which were proposed
> in this CF might dislike this repeat of pg_sleep(1) because it would
> increase the time of regression test.
The sleep is used to ensure the effects of pg_reload_conf() can be visible.
To avoid increasing time of regression tests, we can do one of below:
1) decrease the time for sleep, but not sure how to safely decide how much to sleep.
2) combine the tests such that only 1 or 2 sleep calls should be used.
what's your opinion?
> + /* skip auto conf temporary file */
> + if (strncmp(de->d_name,
> + PG_AUTOCONF_FILENAME,
> + sizeof(PG_AUTOCONF_FILENAME)) == 0)
> + continue;
> Why do we need to exclude the auto conf file from the backup?
> I think that it should be included in the backup as well as normal
> postgresql.conf.
The original intention was to skip the autoconf temporary file which is created during AlterSystemSetConfigFile() for
crashsafety. I will change to exclude temp autoconf file.
> + autofile = fopen(path, PG_BINARY_W);
> + if (autofile == NULL)
> + {
> + fprintf(stderr, _("%s: could not open file \"%s\" for writing: %s\n"),
> + progname, path, strerror(errno));
> + exit_nicely();
> + }
> +
> + if (fputs("# Do not edit this file manually! It is overwritten by
> the ALTER SYSTEM command \n",
> + autofile) < 0)
> + {
> + fprintf(stderr, _("%s: could not write file \"%s\": %s\n"),
> + progname, path, strerror(errno));
> + exit_nicely();
> + }
> +
> + if (fclose(autofile))
> + {
> + fprintf(stderr, _("%s: could not close file \"%s\": %s\n"),
> + progname, path, strerror(errno));
> + exit_nicely();
> + }
> You can simplify the code by using writefile().
Yes, it is better to use writefile(). I shall update the patch for same.
With Regards,
Amit Kapila
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andrew Dunstan
Date:
On 07/12/2013 09:15 AM, Amit kapila wrote: >> guc.c:5187: warning: no previous prototype for 'validate_conf_option' >> preproc.y:7746.2-31: warning: type clash on default action: <str> != <> > I generally use windows as dev environment, it hasn't shown these warnings. Hackers, please take note. Assertions that nobody uses Windows for development are false. cheers andrew
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 07/12/2013 06:15 AM, Amit kapila wrote: > I generally use windows as dev environment, it hasn't shown these warnings. > I shall check in linux and correct the same. Really? Hey, I'm gonna send you a lot of Windows-specific patches for testing in the future ... -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit kapila
Date:
On Friday, July 12, 2013 10:07 PM Josh Berkus wrote: On 07/12/2013 06:15 AM, Amit kapila wrote: >> I generally use windows as dev environment, it hasn't shown these warnings. >> I shall check in linux and correct the same. > Really? Yes. > Hey, I'm gonna send you a lot of Windows-specific patches for >testing in the future ... I would be happy to participate as much as I can. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Friday, July 12, 2013 6:46 PM Amit kapila wrote:
> On Friday, July 12, 2013 12:02 AM Fujii Masao wrote:
> On Fri, Jul 5, 2013 at 2:19 PM, Amit Kapila <amit.kapila@huawei.com>
> wrote:
> > On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
> >> Amit Kapila escribió:
> >>
>
> > I got the following compile warnings.
>
> > guc.c:5187: warning: no previous prototype for 'validate_conf_option'
> > preproc.y:7746.2-31: warning: type clash on default action: <str> !=
> <> In gram.y semicolon (';') was missing, due to which it was not
generating proper preproc.y
> I generally use windows as dev environment, it hasn't shown these
> warnings.
> I shall check in linux and correct the same.
Fixed.
>
> > The make installcheck failed when I ran it against the server with
> > wal_level = hot_standby. The regression.diff is
>
> > ------------------------------------
> > *** 27,35 ****
> > (1 row)
>
> > show wal_level;
> > ! wal_level
> > ! -----------
> > ! minimal
> > (1 row)
>
> > show authentication_timeout;
> > --- 27,35 ----
> > (1 row)
>
> > show wal_level;
> > ! wal_level
> > ! -------------
> > ! hot_standby
> > (1 row)
>
> > show authentication_timeout;
> > ------------------------------------
>
> The tests in alter_system.sql are dependent on default values of
> postgresql.conf, which is okay for make check
> but not for installcheck. I shall remove that dependency from
> testcases.
Removed all tests for which values were dependent on postgresql.conf a. removed check of values before reload b.
removedparameters which can only set after server restart c. removed check for values after setting to default
>
> > The regression test of ALTER SYSTEM calls pg_sleep(1) six times.
> > Those who dislike the regression test patches which were proposed
> > in this CF might dislike this repeat of pg_sleep(1) because it would
> > increase the time of regression test.
>
> The sleep is used to ensure the effects of pg_reload_conf() can be
> visible.
> To avoid increasing time of regression tests, we can do one of below:
>
> 1) decrease the time for sleep, but not sure how to safely decide how
> much to sleep.
> 2) combine the tests such that only 1 or 2 sleep calls should be used.
>
> what's your opinion?
Modified to use 2 sleep calls.
> > + /* skip auto conf temporary file */
> > + if (strncmp(de->d_name,
> > + PG_AUTOCONF_FILENAME,
> > + sizeof(PG_AUTOCONF_FILENAME))
> == 0)
> > + continue;
>
> > Why do we need to exclude the auto conf file from the backup?
> > I think that it should be included in the backup as well as normal
> > postgresql.conf.
>
> The original intention was to skip the autoconf temporary file which
> is created during AlterSystemSetConfigFile()
> for crash safety. I will change to exclude temp autoconf file.
I had modified the check to exclude postgresql.auto.conf.temp file. I
have used hardcoded ".temp" instead of macro,
as I don't find other use of same in code.
> > + autofile = fopen(path, PG_BINARY_W);
> > + if (autofile == NULL)
> > + {
> > + fprintf(stderr, _("%s: could not open file \"%s\" for
> writing: %s\n"),
> > + progname, path, strerror(errno));
> > + exit_nicely();
> > + }
> > +
> > + if (fputs("# Do not edit this file manually! It is
> overwritten by
> > the ALTER SYSTEM command \n",
> > + autofile) < 0)
> > + {
> > + fprintf(stderr, _("%s: could not write file \"%s\":
> %s\n"),
> > + progname, path, strerror(errno));
> > + exit_nicely();
> > + }
> > +
> > + if (fclose(autofile))
> > + {
> > + fprintf(stderr, _("%s: could not close file \"%s\":
> %s\n"),
> > + progname, path, strerror(errno));
> > + exit_nicely();
> > + }
>
> > You can simplify the code by using writefile().
>
> Yes, it is better to use writefile(). I shall update the patch for
> same.
Modified to use writefile().
With Regards,
Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Fujii Masao
Date:
On Mon, Jul 15, 2013 at 10:43 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Friday, July 12, 2013 6:46 PM Amit kapila wrote:
>> On Friday, July 12, 2013 12:02 AM Fujii Masao wrote:
>> On Fri, Jul 5, 2013 at 2:19 PM, Amit Kapila <amit.kapila@huawei.com>
>> wrote:
>> > On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
>> >> Amit Kapila escribió:
>> >>
>>
>> > I got the following compile warnings.
>>
>> > guc.c:5187: warning: no previous prototype for 'validate_conf_option'
>> > preproc.y:7746.2-31: warning: type clash on default action: <str> !=
>> <>
>
> In gram.y semicolon (';') was missing, due to which it was not
> generating proper preproc.y
>
>> I generally use windows as dev environment, it hasn't shown these
>> warnings.
>> I shall check in linux and correct the same.
>
> Fixed.
>>
>> > The make installcheck failed when I ran it against the server with
>> > wal_level = hot_standby. The regression.diff is
>>
>> > ------------------------------------
>> > *** 27,35 ****
>> > (1 row)
>>
>> > show wal_level;
>> > ! wal_level
>> > ! -----------
>> > ! minimal
>> > (1 row)
>>
>> > show authentication_timeout;
>> > --- 27,35 ----
>> > (1 row)
>>
>> > show wal_level;
>> > ! wal_level
>> > ! -------------
>> > ! hot_standby
>> > (1 row)
>>
>> > show authentication_timeout;
>> > ------------------------------------
>>
>> The tests in alter_system.sql are dependent on default values of
>> postgresql.conf, which is okay for make check
>> but not for installcheck. I shall remove that dependency from
>> testcases.
>
> Removed all tests for which values were dependent on postgresql.conf
> a. removed check of values before reload
> b. removed parameters which can only set after server restart
> c. removed check for values after setting to default
>
>>
>> > The regression test of ALTER SYSTEM calls pg_sleep(1) six times.
>> > Those who dislike the regression test patches which were proposed
>> > in this CF might dislike this repeat of pg_sleep(1) because it would
>> > increase the time of regression test.
>>
>> The sleep is used to ensure the effects of pg_reload_conf() can be
>> visible.
>> To avoid increasing time of regression tests, we can do one of below:
>>
>> 1) decrease the time for sleep, but not sure how to safely decide how
>> much to sleep.
>> 2) combine the tests such that only 1 or 2 sleep calls should be used.
>>
>> what's your opinion?
>
> Modified to use 2 sleep calls.
>
>> > + /* skip auto conf temporary file */
>> > + if (strncmp(de->d_name,
>> > + PG_AUTOCONF_FILENAME,
>> > + sizeof(PG_AUTOCONF_FILENAME))
>> == 0)
>> > + continue;
>>
>> > Why do we need to exclude the auto conf file from the backup?
>> > I think that it should be included in the backup as well as normal
>> > postgresql.conf.
>>
>> The original intention was to skip the autoconf temporary file which
>> is created during AlterSystemSetConfigFile()
>> for crash safety. I will change to exclude temp autoconf file.
>
> I had modified the check to exclude postgresql.auto.conf.temp file. I
> have used hardcoded ".temp" instead of macro,
> as I don't find other use of same in code.
>
>> > + autofile = fopen(path, PG_BINARY_W);
>> > + if (autofile == NULL)
>> > + {
>> > + fprintf(stderr, _("%s: could not open file \"%s\" for
>> writing: %s\n"),
>> > + progname, path, strerror(errno));
>> > + exit_nicely();
>> > + }
>> > +
>> > + if (fputs("# Do not edit this file manually! It is
>> overwritten by
>> > the ALTER SYSTEM command \n",
>> > + autofile) < 0)
>> > + {
>> > + fprintf(stderr, _("%s: could not write file \"%s\":
>> %s\n"),
>> > + progname, path, strerror(errno));
>> > + exit_nicely();
>> > + }
>> > +
>> > + if (fclose(autofile))
>> > + {
>> > + fprintf(stderr, _("%s: could not close file \"%s\":
>> %s\n"),
>> > + progname, path, strerror(errno));
>> > + exit_nicely();
>> > + }
>>
>> > You can simplify the code by using writefile().
>>
>> Yes, it is better to use writefile(). I shall update the patch for
>> same.
>
> Modified to use writefile().
Thanks for updating the patch!
In the patch, ALTER SYSTEM SET validates the postgresql.conf.
I think this is overkill. While ALTER SYSTEM SET is being executed,
a user might be modifying the postgresql.conf. That validation
breaks this use case.
+# This includes the default configuration directory included to support
+# ALTER SYSTEM statement.
+#
+# WARNING: User should not remove below include_dir or directory config,
+# as both are required to make ALTER SYSTEM command work.
+# Any configuration parameter values specified after this line
+# will override the values set by ALTER SYSTEM statement.
+#include_dir = 'config'
Why do we need to expose this setting to a user? As the warning says,
if a user *should not* remove this setting, I think we should not expose
it from the beginning and should always treat the postgresql.auto.conf
as included configuration file even if that setting is not in postgresql.conf.
Regards,
--
Fujii Masao
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit kapila
Date:
On Monday, July 15, 2013 11:51 PM Fujii Masao wrote:
On Mon, Jul 15, 2013 at 10:43 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Friday, July 12, 2013 6:46 PM Amit kapila wrote:
>> On Friday, July 12, 2013 12:02 AM Fujii Masao wrote:
>> On Fri, Jul 5, 2013 at 2:19 PM, Amit Kapila <amit.kapila@huawei.com>
>> wrote:
>> > On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
>> >> Amit Kapila escribió:
>> >>
>>
> Thanks for updating the patch!
> In the patch, ALTER SYSTEM SET validates the postgresql.conf.
> I think this is overkill.
I assume you are talking about below code, if I am wrong please point me to the exact code you are reffering:
/* * Verify if include_dir for postgresql.auto.conf file exists in * postgresql.conf. If it doesn't exist then
warnthe user that parameters * changed with this command will not be effective. */ autoconf_errcause =
AUTOCONF_DIR_NOT_PARSED;
if (!ParseConfigFile(ConfigFileName, NULL, false, 0, LOG, &head, &tail)) ereport(ERROR,
(errmsg("configurationfile \"%s\" contains errors;
parameters changed by " "this command will not be effective", ConfigFileName)));
/* * ignore the list of options as we are intersted to just ensure the * existence of include directive for
configfolder. */ head = tail = NULL;
if (autoconf_errcause == AUTOCONF_DIR_NOT_PARSED) ereport(WARNING, (errmsg("configuration
parameterschanged by this
command will not be effective"), errdetail("default include directive for \"%s\"
folder is not present in postgresql.conf", PG_AUTOCONF_DIR)));
This code is to parse postgresql.conf to check if include directive for config directory is present.
> While ALTER SYSTEM SET is being executed,
> a user might be modifying the postgresql.conf. That validation
> breaks this use case.
> +# This includes the default configuration directory included to support
> +# ALTER SYSTEM statement.
> +#
> +# WARNING: User should not remove below include_dir or directory config,
> +# as both are required to make ALTER SYSTEM command work.
> +# Any configuration parameter values specified after this line
> +# will override the values set by ALTER SYSTEM statement.
> +#include_dir = 'config'
> Why do we need to expose this setting to a user?
a) This can be a knob to turn this feature off. This has been asked by few people, one of the mail link is mentioned
below(refer towards end of mail in the link): http://www.postgresql.org/message-id/515B04F9.30900@gmx.net
b) In case user wants to change priority of parameters set by Alter System, he can move the include_dir up or down in
postgresql.conf.
> As the warning says,
> if a user *should not* remove this setting, I think we should not expose
> it from the beginning and should always treat the postgresql.auto.conf
> as included configuration file even if that setting is not in postgresql.conf.
I think it will be usefull to keep include_dir for some users.
If you think that above use for keeping include_dir is not worth or it can be provided in some
another way, then we can change this behavior and remove parsing of postgresql.conf from
AlterSystemSetConfigFile() function.
Thank you very much for reviewing this patch so carefully and giving valuable feedback.
With Regards,
Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Amit kapila escribió: > > +# This includes the default configuration directory included to support > > +# ALTER SYSTEM statement. > > +# > > +# WARNING: User should not remove below include_dir or directory config, > > +# as both are required to make ALTER SYSTEM command work. > > +# Any configuration parameter values specified after this line > > +# will override the values set by ALTER SYSTEM statement. > > +#include_dir = 'config' > > > Why do we need to expose this setting to a user? > > a) This can be a knob to turn this feature off. This has been asked by few people, > one of the mail link is mentioned below (refer towards end of mail in the link): > http://www.postgresql.org/message-id/515B04F9.30900@gmx.net > > b) In case user wants to change priority of parameters set by Alter System, he can move the > include_dir up or down in postgresql.conf. Both of these seem like they would make troubleshooters' lives more difficult. I think we should just parse the auto file automatically after parsing postgresql.conf, without requiring the include directive to be there. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 07/17/2013 12:01 PM, Alvaro Herrera wrote: > Both of these seem like they would make troubleshooters' lives more > difficult. I think we should just parse the auto file automatically > after parsing postgresql.conf, without requiring the include directive > to be there. Wait, I thought the auto file was going into the conf.d directory? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Fri, Jul 12, 2013 at 9:15 AM, Amit kapila <amit.kapila@huawei.com> wrote: > The sleep is used to ensure the effects of pg_reload_conf() can be visible. That strikes me as almost certain to result in random regression test failures. My experience, as a man who helps to maintain a very large suite of regression tests, is that all time-based assumptions about how long it will take some operation to complete are false, and that failures are a lot more frequent than you think they will be. This feature doesn't strike me as a good candidate for regression testing anyway. Keep in mind that people are intended to be able to run make installcheck against a deployed system without messing it up. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Robert Haas escribió: > This feature doesn't strike me as a good candidate for regression > testing anyway. Keep in mind that people are intended to be able to > run make installcheck against a deployed system without messing it up. This is my opinion as well. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-07-17 17:05:40 -0400, Alvaro Herrera wrote: > Robert Haas escribió: > > > This feature doesn't strike me as a good candidate for regression > > testing anyway. Keep in mind that people are intended to be able to > > run make installcheck against a deployed system without messing it up. > > This is my opinion as well. Setting and resetting a GUC like pg_regress.test_var or so shouldn't be too invasive. Some minimal testing seems like a good idea to me. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 07/17/2013 03:17 PM, Andres Freund wrote: > On 2013-07-17 17:05:40 -0400, Alvaro Herrera wrote: >> Robert Haas escribió: >> >>> This feature doesn't strike me as a good candidate for regression >>> testing anyway. Keep in mind that people are intended to be able to >>> run make installcheck against a deployed system without messing it up. >> >> This is my opinion as well. > > Setting and resetting a GUC like pg_regress.test_var or so shouldn't be > too invasive. Some minimal testing seems like a good idea to me. Also, we can "make check" even if we don't "make installcheck". -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Peter Eisentraut
Date:
If we are going to add ALTER SYSTEM, then I'd like to consider recasting ALTER ROLE ALL SET in terms of ALTER SYSTEM as well, because the ALTER ROLE ALL syntax was a hack. At least it should be possible to consider various levels of "system". Who knows, maybe a future version will allow propagating settings to standby servers? I don't know what a good terminology would be, but let's keep some options open.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Peter Eisentraut
Date:
On Tue, 2013-07-16 at 04:23 +0000, Amit kapila wrote: > > > Why do we need to expose this setting to a user? > > a) This can be a knob to turn this feature off. This has been asked by > few people, > one of the mail link is mentioned below (refer towards end of mail > in the link): > http://www.postgresql.org/message-id/515B04F9.30900@gmx.net > > b) In case user wants to change priority of parameters set by Alter > System, he can move the > include_dir up or down in postgresql.conf. In that message I argue that the auto file should always be read implicitly. The comment about being able to turn this feature off was because as presented it was too complicated and error-prone. Removing the include would incidentally not be a way to turn the feature off. Turning the feature of would require disabling the ALTER SYSTEM command. I'm not really asking for that functionality, but I'm really against having to manually include the auto file, and also against the system nagging me about including the file. There is the slightly interesting question whether the auto file should notionally be processed before or after the main postgresql.conf, but that's a question we should answer now, not let it the users work it out.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Thursday, July 18, 2013 2:36 AM Alvaro Herrera wrote: > Robert Haas escribió: > > > This feature doesn't strike me as a good candidate for regression > > testing anyway. Keep in mind that people are intended to be able to > > run make installcheck against a deployed system without messing it > up. > > This is my opinion as well. Summarization of all tests in this patch: 1. Set of SIGHUP parameters in config file, use reload and sleep and then check value of parameters to validate whether they are set. 2. Reset all parameters so that the system returns to previous state, use reload and sleep 3. Negative scenario tests As suggested by Robert that having timing based tests are not good candidate for regression tests, which essentially means that all tests as part of point 1 & 2 are not good candidates for testing. If tests corresponding to point 1 && 2 are to be removed, then keeping some negative tests also doesn't seem to make much sense. To cover Alter System functionality and code, we can have Alter System commands without reload and sleep, but I think it might not be of much use until we verify the same using Show. So I will remove the tests from this patch in next updated version if there is no other useful way to achieve it. On a side note, I had checked that in regression suite already few tests use pg_sleep() and one of them in stats.sql is something similar. It also does wait using pg_sleep() to let prior tests action get completed. I understand that having already at one place doesn't allow to use it at other places, but still I think that if some tests using similar mechanism are running successfully, we can add new one provided they ensure the timing doesn't cause problem. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Thursday, July 18, 2013 12:31 AM Alvaro Herrera wrote: > Amit kapila escribió: > > > > > +# This includes the default configuration directory included to > support > > > +# ALTER SYSTEM statement. > > > +# > > > +# WARNING: User should not remove below include_dir or directory > config, > > > +# as both are required to make ALTER SYSTEM command work. > > > +# Any configuration parameter values specified after this > line > > > +# will override the values set by ALTER SYSTEM statement. > > > +#include_dir = 'config' > > > > > Why do we need to expose this setting to a user? > > > > a) This can be a knob to turn this feature off. This has been asked > by few people, > > one of the mail link is mentioned below (refer towards end of mail > in the link): > > http://www.postgresql.org/message-id/515B04F9.30900@gmx.net > > > > b) In case user wants to change priority of parameters set by Alter > System, he can move the > > include_dir up or down in postgresql.conf. > > Both of these seem like they would make troubleshooters' lives more > difficult. I think we should just parse the auto file automatically > after parsing postgresql.conf, without requiring the include directive > to be there. Okay, I shall modify the patch that way. However still the file will be in config directory. Now the 2 things regarding this needs to be taken care: a. As raised by Peter Eisentraut, shall we do parsing of autofile before or after postgresql.conf. By default I will keep it after until there is some reason to do it before. b. If the directory/file is missing while parsing issue warning message. Kindly let me know your opinion about the same. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Thursday, July 18, 2013 12:38 AM Josh Berkus wrote: > On 07/17/2013 12:01 PM, Alvaro Herrera wrote: > > Both of these seem like they would make troubleshooters' lives more > > difficult. I think we should just parse the auto file automatically > > after parsing postgresql.conf, without requiring the include > directive > > to be there. > > Wait, I thought the auto file was going into the conf.d directory? Auto file is going into config directory, but will that make any difference if we have to parse it automatically after postgresql.conf. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 07/18/2013 04:25 AM, Amit Kapila wrote: > On Thursday, July 18, 2013 12:38 AM Josh Berkus wrote: >> On 07/17/2013 12:01 PM, Alvaro Herrera wrote: >>> Both of these seem like they would make troubleshooters' lives more >>> difficult. I think we should just parse the auto file automatically >>> after parsing postgresql.conf, without requiring the include >> directive >>> to be there. >> >> Wait, I thought the auto file was going into the conf.d directory? > > Auto file is going into config directory, but will that make any difference > if we have to parse it automatically after postgresql.conf. Well, I thought that the whole conf.d directory automatically got parsed after postgresql.conf. No? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Fujii Masao
Date:
On Fri, Jul 19, 2013 at 2:45 AM, Josh Berkus <josh@agliodbs.com> wrote: > On 07/18/2013 04:25 AM, Amit Kapila wrote: >> On Thursday, July 18, 2013 12:38 AM Josh Berkus wrote: >>> On 07/17/2013 12:01 PM, Alvaro Herrera wrote: >>>> Both of these seem like they would make troubleshooters' lives more >>>> difficult. I think we should just parse the auto file automatically >>>> after parsing postgresql.conf, without requiring the include >>> directive >>>> to be there. >>> >>> Wait, I thought the auto file was going into the conf.d directory? >> >> Auto file is going into config directory, but will that make any difference >> if we have to parse it automatically after postgresql.conf. > > Well, I thought that the whole conf.d directory automatically got parsed > after postgresql.conf. No? No, in the previous patch. We needed to set include_dir to 'config' (though that's the default). Regards, -- Fujii Masao
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Fujii Masao escribió: > On Fri, Jul 19, 2013 at 2:45 AM, Josh Berkus <josh@agliodbs.com> wrote: > > On 07/18/2013 04:25 AM, Amit Kapila wrote: > >> On Thursday, July 18, 2013 12:38 AM Josh Berkus wrote: > >>> On 07/17/2013 12:01 PM, Alvaro Herrera wrote: > >>>> Both of these seem like they would make troubleshooters' lives more > >>>> difficult. I think we should just parse the auto file automatically > >>>> after parsing postgresql.conf, without requiring the include > >>> directive > >>>> to be there. > >>> > >>> Wait, I thought the auto file was going into the conf.d directory? > >> > >> Auto file is going into config directory, but will that make any difference > >> if we have to parse it automatically after postgresql.conf. > > > > Well, I thought that the whole conf.d directory automatically got parsed > > after postgresql.conf. No? > > No, in the previous patch. We needed to set include_dir to 'config' (though > that's the default). I know this has been discussed already, but I want to do a quick summary of existing proposals, point out their drawbacks, and present a proposal of my own. Note I ignore the whole file-per-setting vs. one-file-to-rule-them-all, because the latter has already been shot down. So live ideas floated here are: 1. have a config/postgresql.auto.conf file, require include_dir or include in postgresql.conf This breaks if user removesthe config/ directory, or if the user removes the include_dir directive from postgresql.conf. ALTER SYSTEM is inthe business of doing mkdir() or failing altogether if the dir is not present. Doesn't seem friendly. 2. have a conf.d/ directory, put the file therein; no include or include_dir directive is necessary. I think this is aseparate patch and merits its own discussion. This might be a good idea, but I don't think that this is the way to implementALTER SYSTEM. If users don't want to allow conf.d they can remove it, but that would break ALTER SYSTEM unnecessarily. Since they are separate features it seems to me that they should function independently. I think we should just put the config directives of ALTER SYSTEM into a file, not dir, alongside postgresql.conf; I would suggest postgresql.auto.conf. This file is parsed automatically after postgresql.conf, without requiring an "include" directive in postgresql.conf. If the user does not want that file, they can remove it; but a future ALTER SYSTEM will create the file again. No need to parse stuff to find out whether the directory exists, or the file exists, or such nonsense. I think the only drawback of this is that there's no way to disable ALTER SYSTEM (i.e. there's no directory you can remove to prevent the thing from working). But is this a desirable property? I mean, if we want to disallow ALTER SYSTEM I think we should provide a direct way to do that, perhaps allow_alter_system = off in postgresql.conf; but I don't think we need to provide this, really, at least not in the first implementation. Note that the Debian packaging uses postgres-writable directories for its configuration files, so the daemon is always able to create the file in the config dir. This seems the simplest way; config tools such as Puppet know they always need to consider the postgresql.auto.conf file. I think the whole business of parsing the file, and then verifying whether the auto file has been parsed, is nonsensical and should be removed from the patch. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
Alvaro, > I think the only drawback of this is that there's no way to disable > ALTER SYSTEM (i.e. there's no directory you can remove to prevent the > thing from working). But is this a desirable property? I mean, if we > want to disallow ALTER SYSTEM I think we should provide a direct way to > do that, perhaps allow_alter_system = off in postgresql.conf; but I > don't think we need to provide this, really, at least not in the first > implementation. Agreed that turning alter system off by deleting the directory is NOT a desireable feature. I'm also unclear on the desire to disable ALTER SYSTEM; if someone has superuser access, then they can just use pg_write_file to add config directives anyway, no? So there's not any security value in disabling it. Maybe there's a case I'm not thinking of. Of course, people *can* disable it by creating a blank postgresql.auto.conf file in the right place and making it non-writeable, if they want. We are missing one feature, which is the ability to relocate the postgresql.auto.conf file if relocating it is desireable according to some sysadmin spec. This kind of ties into another patch which was discussed on this list -- the ability to relocate the recovery.conf file. Personally, I think that it would be better for the users if we provided a way to relocate *all* conf files to the same master directory, but that we don't need a way to relocate each config file individually, but that's a longer discussion. In other words, let's accept an ALTER SYSTEM patch which doesn't support relocating, and then argue whether a second patch which supports relocating is needed. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Josh Berkus escribió: > We are missing one feature, which is the ability to relocate the > postgresql.auto.conf file if relocating it is desireable according to > some sysadmin spec. This kind of ties into another patch which was > discussed on this list -- the ability to relocate the recovery.conf > file. Well, postgresql.conf is already relocatable. Is there any advantage in being able to move postgresql.auto.conf to a different location than postgresql.conf? I don't see any. I didn't follow the recovery.conf discussion, but I imagine that the reason for wanting to be able to relocate it is related to standby vs. master distinction. This doesn't apply to postgresql.auto.conf, I think, does it? If people want to drill real down, they can always have a postgresql.auto.conf that's a symlink. (In this light, we would ship an empty postgresql.auto.conf in a freshly initdb'd system. IIRC the current patch already does that.) -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Smith
Date:
On 7/18/13 4:02 PM, Alvaro Herrera wrote: > I think we should just put the config directives of ALTER SYSTEM into a > file, not dir, alongside postgresql.conf; I would suggest > postgresql.auto.conf. This file is parsed automatically after > postgresql.conf, without requiring an "include" directive in > postgresql.conf. It is possible to build ALTER SYSTEM SET this way. One of the goals of the implementation style used here wasn't just to accomplish that narrow feature though. We keep running into situations where a standard, directory based configuration system would make things easier. Each time that happens, someone comes up with another way to avoid doing it. I think that approach contributes to stagnationin the terrible status quo of PostgreSQL configuration management. I thought this was a good spot to try and re-draw this line because I don't want just one program that is able to create new configuration entries easily. I want to see a whole universe of them. ALTER SYSTEM SET, tuning helpers, replication helpers, logging helpers, vacuum schedulers. All of them *could* just dump a simple file into a config directory with code anyone can write. And having ALTER SYSTEM SET do that provides a strong precedent for how it can be done. (I'd like to see initdb do that instead of hacking the system postgresql.conf as if sed-style edits were still the new hotness, but that's a future change) My claim, and that's one informed by writing pgtune, is that by far the hardest part of writing a configuration addition tool is parsing a postgresql.conf file. The expectation right now is that all changes must happen there. Creating a new snippet of configuration settings is easy, but no tools know where to put one right now. Instead we just keep coming up with single, hard-coded file names that people have to know in order to manage their installations. > This seems the simplest way; config tools such as Puppet know they > always need to consider the postgresql.auto.conf file. Like this idea. What administrators really want, whether they realize it or not, is to point Puppet at a configuration directory. Then the problem of "what are the config files in the new version of Postgres" happens once more and then it's over. Exactly what the files in there are named should be completely under control of the administrator. We're never going to reach that unless we lead by example though. The database's configuration pushes people toward using a small number of files with magic names--postgresql.conf, recovery.conf, and now postgresql.auto.conf in your proposal. Meanwhile, all sensible UNIX-style projects with complicated configurations in text files has moved toward a configuration directory full of them. Here are some directories on the last RHEL6 system I was editing configuration on this week: /etc/httpd/conf.d/ /etc/munin/conf.d/ /etc/ld.so.conf.d/ /etc/munin/conf.d/ /etc/dovecot/conf.d/ /etc/yum/pluginconf.d/ Some of them didn't get the memo that the right standard name is conf.d now, but they're the minority. It's fine to have a postgresql.conf file that you *can* make all your changes to, for people who want to stay in the old monolithic approach. But if there were also a conf.d directory underthere, many classes of administrator would breath a sign of relief. With all the precedents people may have already ran into, with the right naming can be made discoverable and familiar to a lot of administrators. Telling them instead that there's this new postgresql.auto.conf file that suddenly they have to worry about--I'd say don't even bother if that's how you want to do it. That's making the problem I've been trying to simplify for five years now even worse. > I think the whole business of parsing the file, and then verifying > whether the auto file has been parsed, is nonsensical and should be > removed from the patch. That was just trying to keep people from screwing up their configuration while they get used to things. That and some of the other sanity checking is not necessary, it was just trying to make the transition to the new configuration approach less error-prone. I don't think anyone would disagree that the current patch is doing enough of that error checking work that the error checking itself is the most likely thing to break. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
Greg, > I thought this was a good spot to try and re-draw this line because I > don't want just one program that is able to create new configuration > entries easily. I want to see a whole universe of them. ALTER SYSTEM > SET, tuning helpers, replication helpers, logging helpers, vacuum > schedulers. All of them *could* just dump a simple file into a config > directory with code anyone can write. And having ALTER SYSTEM SET do > that provides a strong precedent for how it can be done. (I'd like to > see initdb do that instead of hacking the system postgresql.conf as if > sed-style edits were still the new hotness, but that's a future change) Thank you. I wanted to say this, but I couldn't find a way to express it. > Some of them didn't get the memo that the right standard name is conf.d > now, but they're the minority. Apparently we didn't get the memo either. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Friday, July 19, 2013 1:33 AM Alvaro Herrera wrote: > Fujii Masao escribió: > > On Fri, Jul 19, 2013 at 2:45 AM, Josh Berkus <josh@agliodbs.com> > wrote: > > > On 07/18/2013 04:25 AM, Amit Kapila wrote: > > >> On Thursday, July 18, 2013 12:38 AM Josh Berkus wrote: > > >>> On 07/17/2013 12:01 PM, Alvaro Herrera wrote: > > >>>> Both of these seem like they would make troubleshooters' lives > more > > >>>> difficult. I think we should just parse the auto file > automatically > > >>>> after parsing postgresql.conf, without requiring the include > > >>> directive > > >>>> to be there. > > >>> > > >>> Wait, I thought the auto file was going into the conf.d > directory? > > >> > > >> Auto file is going into config directory, but will that make any > difference > > >> if we have to parse it automatically after postgresql.conf. > > > > > > Well, I thought that the whole conf.d directory automatically got > parsed > > > after postgresql.conf. No? > > > > No, in the previous patch. We needed to set include_dir to 'config' > (though > > that's the default). > > I know this has been discussed already, but I want to do a quick > summary > of existing proposals, point out their drawbacks, and present a > proposal > of my own. Note I ignore the whole file-per-setting vs. > one-file-to-rule-them-all, because the latter has already been shot > down. So live ideas floated here are: > > 1. have a config/postgresql.auto.conf file, require include_dir or > include in postgresql.conf > This breaks if user removes the config/ directory, or if the user > removes the include_dir directive from postgresql.conf. ALTER > SYSTEM > is in the business of doing mkdir() or failing altogether if the dir > is not present. Doesn't seem friendly. > > 2. have a conf.d/ directory, put the file therein; no include or > include_dir directive is necessary. > I think this is a separate patch and merits its own discussion. > This > might be a good idea, but I don't think that this is the > way to implement ALTER SYSTEM. If users don't want to allow conf.d > they can remove it, but that would break ALTER SYSTEM unnecessarily. > Since they are separate features it seems to me that they should > function independently. > > I think we should just put the config directives of ALTER SYSTEM into a > file, not dir, alongside postgresql.conf; I would suggest > postgresql.auto.conf. This file is parsed automatically after > postgresql.conf, without requiring an "include" directive in > postgresql.conf. If the user does not want that file, they can remove > it; but a future ALTER SYSTEM will create the file again. > No need to > parse stuff to find out whether the directory exists, or the file > exists, or such nonsense. I think this will be removed from patch, once we implement automatic parsing of config/postgresql.auto.conf after parsing postgresql.conf > I think the only drawback of this is that there's no way to disable > ALTER SYSTEM (i.e. there's no directory you can remove to prevent the > thing from working). But is this a desirable property? I mean, if we > want to disallow ALTER SYSTEM I think we should provide a direct way to > do that, perhaps allow_alter_system = off in postgresql.conf; but I > don't think we need to provide this, really, at least not in the first > implementation. > > Note that the Debian packaging uses postgres-writable directories for > its configuration files, so the daemon is always able to create the > file > in the config dir. > > This seems the simplest way; config tools such as Puppet know they > always need to consider the postgresql.auto.conf file. > I think the whole business of parsing the file, and then verifying > whether the auto file has been parsed, is nonsensical and should be > removed from the patch. Okay, I understood your concern about checking during parsing to find error whether directory/file exist. In the next version,I will remove this error. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit kapila
Date:
Amit Kapila wrote: On Thursday, July 18, 2013 12:31 AM Alvaro Herrera wrote: > Amit kapila escribió: > > >> > > +# This includes the default configuration directory included to >> support >> > > +# ALTER SYSTEM statement. >> > > +# >> > > +# WARNING: User should not remove below include_dir or directory >> config, >> > > +# as both are required to make ALTER SYSTEM command work. >> > > +# Any configuration parameter values specified after this >> line >> > > +# will override the values set by ALTER SYSTEM statement. >> > > +#include_dir = 'config' >> > >> > > Why do we need to expose this setting to a user? >> > >> > a) This can be a knob to turn this feature off. This has been asked >> by few people, >> > one of the mail link is mentioned below (refer towards end of mail >> in the link): >> > http://www.postgresql.org/message-id/515B04F9.30900@gmx.net >> > >> > b) In case user wants to change priority of parameters set by Alter >> System, he can move the >> > include_dir up or down in postgresql.conf. >> >> Both of these seem like they would make troubleshooters' lives more >> difficult. I think we should just parse the auto file automatically >> after parsing postgresql.conf, without requiring the include directive >> to be there. > Okay, I shall modify the patch that way. However still the file will be in > config directory. > Now the 2 things regarding this needs to be taken care: > a. As raised by Peter Eisentraut, shall we do parsing of autofile before or > after postgresql.conf. By default I will keep it after until there is some > reason to do it before. > b. If the directory/file is missing while parsing issue warning message. As per discussion, updated patch contains following changes: 1. Regression tests for Alter System are removed 2. Parsed the auto file automatically after parsing postgresql.conf 3. Removed addition of include directive in postgresql.conf 4. Removed error handling for parsing errors With Regards, Amit Kapila.
Attachment
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Smith
Date:
On 7/22/13 12:58 PM, Amit kapila wrote: > As per discussion, updated patch contains following changes: > 1. Regression tests for Alter System are removed > 2. Parsed the auto file automatically after parsing postgresql.conf > 3. Removed addition of include directive in postgresql.conf > 4. Removed error handling for parsing errors These changes have shrunk the diff down to 1411 lines of code. I'd like to identify which committer might take this on at this point. In a few respects this is "Ready For Committer" now, because the committer who takes this on is going to get a strong vote on how to resolve most of the remaining fundamental issues: -Is this the point to finally commit to the config directory approach? -If this does set the config directory usage precedent, is the name used for that appropriate? Whoever suggested the change from "conf.d" to "config" made an error IMHO. Every example I've found of other projects doing this style of config refactoring picked either "conf.d" or a unique, no two are alike name. I'd rather not see Postgres add yet another unique one. (I think the 'postgresql' part of postgresql.auto.conf as the name of the file is redundant too--what else would be in the Postgres config directory but postgresql files?--but that's not a major issue) This could definitely use a round of committer level review of how the GUC handling is being done now too. That part of the code seems to have settled, and things like using the new validate_conf_option could be committed even with other parts still being discussed. Exactly how to best break this out into useful commits is another decision that really needs some input from the potential committer though. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Greg Smith escribió: > On 7/22/13 12:58 PM, Amit kapila wrote: > >As per discussion, updated patch contains following changes: > >1. Regression tests for Alter System are removed > >2. Parsed the auto file automatically after parsing postgresql.conf > >3. Removed addition of include directive in postgresql.conf > >4. Removed error handling for parsing errors > > These changes have shrunk the diff down to 1411 lines of code. Nice. > I'd like to identify which committer might take this on at this > point. I'm considering take a stab at it soonish, FWIW, if no other committer steps up. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
All, Christophe just discovered something with include files which is going to cause issues with ALTER SYSTEM SET. So, take as a hypothetical that you use the default postgresql.conf file, which sets shared_buffers = 32MB. Instead of editing this file, you do ALTER SYSTEM SET shared_buffers = '1GB', which updates config/postgresql.auto.conf. Then you restart PostgreSQL. Everything is hunky-dory, until a later occasion where you *reload* postgresql. Then postgres startup hits the first "shared_buffers=32MB" (in postgresql.conf), says "I can't change shared buffers on a reload", and aborts before ever getting to postgresql.auto.conf. This is a problem that exists now with includes in postgresql.conf, but having ALTER SYSTEM SET will cause more users to hit it. Seems like we'll need to fix it before releasing 9.4. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Josh Berkus <josh@agliodbs.com> writes:
> Christophe just discovered something with include files which is going
> to cause issues with ALTER SYSTEM SET.
> So, take as a hypothetical that you use the default postgresql.conf
> file, which sets shared_buffers = 32MB.
> Instead of editing this file, you do ALTER SYSTEM SET shared_buffers =
> '1GB', which updates config/postgresql.auto.conf.
> Then you restart PostgreSQL.
> Everything is hunky-dory, until a later occasion where you *reload*
> postgresql.
Everything was already *not* hunky-dory as soon as you did that, since
a SIGHUP would have had the same problem.
I'd be inclined to think that ALTER SYSTEM SET should not be allowed to
modify any PGC_POSTMASTER parameters.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
> On Tuesday, July 23, 2013 5:26 AM Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
> > Christophe just discovered something with include files which is
> going
> > to cause issues with ALTER SYSTEM SET.
>
> > So, take as a hypothetical that you use the default postgresql.conf
> > file, which sets shared_buffers = 32MB.
>
> > Instead of editing this file, you do ALTER SYSTEM SET shared_buffers
> =
> > '1GB', which updates config/postgresql.auto.conf.
>
> > Then you restart PostgreSQL.
>
> > Everything is hunky-dory, until a later occasion where you *reload*
> > postgresql.
>
> Everything was already *not* hunky-dory as soon as you did that, since
> a SIGHUP would have had the same problem.
>
> I'd be inclined to think that ALTER SYSTEM SET should not be allowed to
> modify any PGC_POSTMASTER parameters.
One way to fix the problem is that while parsing if the option already
exists, replace it. Something like below code
ParseConfigFp()
{
..
..
/* replace the option value, if already exists in list */ for (item = *head_p; item != NULL;
item=
item->next) { if (strcmp(item->name, opt_name) == 0)
{ pfree(item->value);
item->value= pstrdup(opt_value); replaced = true;
break; } } if(!replaced)
{ /* ordinary variable, append to list */ item
=palloc(sizeof *item); item->name = opt_name; item->value
=opt_value; item->filename = pstrdup(config_file);
item->sourceline= ConfigFileLineno-1; item->next = NULL;
if(*head_p == NULL) *head_p = item; else
(*tail_p)->next = item; *tail_p = item;
}
..
..
}
There is overhead of traversing the list each time, but as this path is
traversed in less and non-performance critical operations, it can be
considered to fix the problem.
If you consider above as a non-trivial or not a right way to fix the
problem,
then I can update the patch to disallow PGC_POSTMASTER parameters by ALTER
SYSTEM SET command.
With Regards,
Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
> Everything was already *not* hunky-dory as soon as you did that, since > a SIGHUP would have had the same problem. > > I'd be inclined to think that ALTER SYSTEM SET should not be allowed to > modify any PGC_POSTMASTER parameters. That makes alter system set a bunch less useful, but might be insurmountable. Anyway, I also regard this as a problem we need to resolve now that we have the config directory if we expect people to build autotuning tools.Once someone is relying on an autotuning tool whichdrops a file in config/, it becomes a real problem that there are uncommented PGC_POSTMASTER options in the default postgresql.conf. I'm not sure what the solution to that problem should be, but I do know that we're going to hear it a lot from users as includes and the config directory get more common. Certainly any solution which says "first, manually edit postgresql.conf" makes the config directory into a kind of joke. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tuesday, July 23, 2013 5:06 AM Josh Berkus wrote: > All, > > Christophe just discovered something with include files which is going > to cause issues with ALTER SYSTEM SET. > > So, take as a hypothetical that you use the default postgresql.conf > file, which sets shared_buffers = 32MB. > > Instead of editing this file, you do ALTER SYSTEM SET shared_buffers = > '1GB', which updates config/postgresql.auto.conf. > > Then you restart PostgreSQL. > > Everything is hunky-dory, until a later occasion where you *reload* > postgresql. Then postgres startup hits the first "shared_buffers=32MB" > (in postgresql.conf), says "I can't change shared buffers on a reload", > and aborts before ever getting to postgresql.auto.conf. It doesn't abort after showing initial message "parameter %s cannot be changed without restart", rather it processes allremaining parameters. We can even test it by setting 1 postmaster and 1 sighup variable, after reload, even though it log message for postmastervariable, but it will set the sighup variable. So in this the real problem is that there is some message in log which can create confusion, otherwise the behavior willbe exactly what user wants as per Postgres specs. To avoid the message, following can be done: 1. We need to make sure that in function ProcessConfigFile before calling set_config_option(), list of parameters is unique. a. it can be done while storing the elements in list as suggested by me in my yesterday's mail. http://www.postgresql.org/message-id/004c01ce8761$b8a4ab20$29ee0160$@kapila@huawei.com b. after parsing all the config files,make sure the list contain one entry for any variable (giving priority to elements that come later) 2. As this is just a confusing message issue, we can even update the docs to explain this behavior. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Thu, Jul 18, 2013 at 6:03 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > Josh Berkus escribió: > >> We are missing one feature, which is the ability to relocate the >> postgresql.auto.conf file if relocating it is desireable according to >> some sysadmin spec. This kind of ties into another patch which was >> discussed on this list -- the ability to relocate the recovery.conf >> file. > > Well, postgresql.conf is already relocatable. Is there any advantage in > being able to move postgresql.auto.conf to a different location than > postgresql.conf? I don't see any. I didn't follow the recovery.conf > discussion, but I imagine that the reason for wanting to be able to > relocate it is related to standby vs. master distinction. This doesn't > apply to postgresql.auto.conf, I think, does it? > > If people want to drill real down, they can always have a > postgresql.auto.conf that's a symlink. (In this light, we would ship an > empty postgresql.auto.conf in a freshly initdb'd system. IIRC the > current patch already does that.) My thought is that people might put postgresql.conf in a directory that only contains configuration files and isn't writeable by the postgres user. So I would expect to find postgresql.auto.conf in the data directory always. But in general +1 for your proposal. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Mon, Jul 22, 2013 at 7:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Josh Berkus <josh@agliodbs.com> writes: >> Christophe just discovered something with include files which is going >> to cause issues with ALTER SYSTEM SET. > >> So, take as a hypothetical that you use the default postgresql.conf >> file, which sets shared_buffers = 32MB. > >> Instead of editing this file, you do ALTER SYSTEM SET shared_buffers = >> '1GB', which updates config/postgresql.auto.conf. > >> Then you restart PostgreSQL. > >> Everything is hunky-dory, until a later occasion where you *reload* >> postgresql. > > Everything was already *not* hunky-dory as soon as you did that, since > a SIGHUP would have had the same problem. > > I'd be inclined to think that ALTER SYSTEM SET should not be allowed to > modify any PGC_POSTMASTER parameters. That significantly decreases the usefulness of ALTER SYSTEM without actually preventing the underlying problem. If having multiple conflicting values for parameters in the config files doesn't work cleanly, then we should fix that, not cripple this feature. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Robert Haas escribió: > On Mon, Jul 22, 2013 at 7:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > I'd be inclined to think that ALTER SYSTEM SET should not be allowed to > > modify any PGC_POSTMASTER parameters. > > That significantly decreases the usefulness of ALTER SYSTEM without > actually preventing the underlying problem. If having multiple > conflicting values for parameters in the config files doesn't work > cleanly, then we should fix that, not cripple this feature. I think the only solution that makes sense here is to comment out the entry in postgresql.conf. We weren't supposed to modify that file, but in light of this problem I'm starting to be of the opinion that we ought to allow this implementation do this one change. I admit this is somewhat problematic: consider that somebody having postgresql.conf open in an editor while somebody else runs ALTER SYSTEM would see the file changing underneath. But at least in Vim, what happens is that the editor warns you that the file has changed underneath, and asks you whether you want to load the new version. Now, this is not likely to be an easy thing to implement ... Another option would be to separate config loading in two stages; first read all the files, and only check and throw errors and warnings later, when we know the definite place that's going to be the authoritative definition of each variable. I'm not comfortable with the idea that PGC_POSTMASTER variables should be treated differently from others. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Robert Haas escribi�:
>> On Mon, Jul 22, 2013 at 7:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> I'd be inclined to think that ALTER SYSTEM SET should not be allowed to
>>> modify any PGC_POSTMASTER parameters.
>> That significantly decreases the usefulness of ALTER SYSTEM without
>> actually preventing the underlying problem. If having multiple
>> conflicting values for parameters in the config files doesn't work
>> cleanly, then we should fix that, not cripple this feature.
> I think the only solution that makes sense here is to comment out the
> entry in postgresql.conf.
Actually, this is much ado about nothing. If you actually put two
conflicting values for shared_buffers into postgresql.conf, you'll
find out that:
1. on initial start, the server just silently adopts the later one.
2. if you SIGHUP, you get:
LOG: autovacuum launcher started
LOG: received SIGHUP, reloading configuration files
LOG: parameter "shared_buffers" cannot be changed without restarting the server
LOG: configuration file "/home/postgres/version93/data/postgresql.conf" contains errors; unaffected changes were
applied
We could possibly improve the code to not produce that log bleat, but
it's hardly a show-stopper as is.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Jul 18, 2013 at 6:03 PM, Alvaro Herrera
> <alvherre@2ndquadrant.com> wrote:
>> Josh Berkus escribi�:
>>> We are missing one feature, which is the ability to relocate the
>>> postgresql.auto.conf file if relocating it is desireable according to
>>> some sysadmin spec.
> My thought is that people might put postgresql.conf in a directory
> that only contains configuration files and isn't writeable by the
> postgres user. So I would expect to find postgresql.auto.conf in the
> data directory always.
Yeah, exactly. I think putting it anywhere but $PGDATA is a bad idea,
and a sysadmin who thinks he knows better probably doesn't.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 07/25/2013 02:02 PM, Tom Lane wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Thu, Jul 18, 2013 at 6:03 PM, Alvaro Herrera >> <alvherre@2ndquadrant.com> wrote: >>> Josh Berkus escribi�: >>>> We are missing one feature, which is the ability to relocate the >>>> postgresql.auto.conf file if relocating it is desireable according to >>>> some sysadmin spec. > >> My thought is that people might put postgresql.conf in a directory >> that only contains configuration files and isn't writeable by the >> postgres user. So I would expect to find postgresql.auto.conf in the >> data directory always. > > Yeah, exactly. I think putting it anywhere but $PGDATA is a bad idea, > and a sysadmin who thinks he knows better probably doesn't. Please see Greg Smith's fairly strong arguments for putting it in the config/ directory. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Josh Berkus escribió: > On 07/25/2013 02:02 PM, Tom Lane wrote: > > Robert Haas <robertmhaas@gmail.com> writes: > > > >> My thought is that people might put postgresql.conf in a directory > >> that only contains configuration files and isn't writeable by the > >> postgres user. So I would expect to find postgresql.auto.conf in the > >> data directory always. > > > > Yeah, exactly. I think putting it anywhere but $PGDATA is a bad idea, > > and a sysadmin who thinks he knows better probably doesn't. > > Please see Greg Smith's fairly strong arguments for putting it in the > config/ directory. As far as I see, there are two argument he makes: 1. We ought to have conf.d/ (not config/) so that tools have a way to deploy snippets (e.g. pgtune) 2. we ought to have all the config files together so that versioning tools (Puppet) can just say "keep all files withindirectory X versioned" and not have to care about specific file names, etc. I can buy (1), because that's a pretty common design for daemons nowadays. But I think that's its own patch, and there's no reason that this patch should be messing with this. I don't care all that much about (2), but I have no problem with doing that. So we could have two patches, first one that introduces a conf.d subdir that's automatically parsed after postgresql.conf, and another one that implements ALTER SYSTEM by using a file within conf.d. The reason I say we need a separate patch for conf.d is that I think it'd be easier to argue about it in isolation, than having it be entangled with ALTER SYSTEM stuff. The main contention point I see is where conf.d lives; the two options are in $PGDATA or together with postgresql.conf. Tom and Robert, above, say it should be in $PGDATA; but this goes against Debian packaging and the Linux FHS (or whatever that thing is called). -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Friday, July 26, 2013 10:35 AM Alvaro Herrera wrote: > Josh Berkus escribió: > > On 07/25/2013 02:02 PM, Tom Lane wrote: > > > Robert Haas <robertmhaas@gmail.com> writes: > > > > > >> My thought is that people might put postgresql.conf in a directory > > >> that only contains configuration files and isn't writeable by the > > >> postgres user. So I would expect to find postgresql.auto.conf in > the > > >> data directory always. > > > > > > Yeah, exactly. I think putting it anywhere but $PGDATA is a bad > idea, > > > and a sysadmin who thinks he knows better probably doesn't. > > > > Please see Greg Smith's fairly strong arguments for putting it in the > > config/ directory. > > As far as I see, there are two argument he makes: > > 1. We ought to have conf.d/ (not config/) so that tools have a way to > deploy snippets (e.g. pgtune) > > 2. we ought to have all the config files together so that versioning > tools (Puppet) can just say "keep all files within directory X > versioned" and not have to care about specific file names, etc. > > > I can buy (1), because that's a pretty common design for daemons > nowadays. But I think that's its own patch, and there's no reason that > this patch should be messing with this. I don't care all that much > about (2), but I have no problem with doing that. > > So we could have two patches, first one that introduces a conf.d subdir > that's automatically parsed after postgresql.conf, and another one that > implements ALTER SYSTEM by using a file within conf.d. The reason I > say > we need a separate patch for conf.d is that I think it'd be easier to > argue about it in isolation, than having it be entangled with ALTER > SYSTEM stuff. The main contention point I see is where conf.d lives; > the two options are in $PGDATA or together with postgresql.conf. Tom > and Robert, above, say it should be in $PGDATA; but this goes against > Debian packaging and the Linux FHS (or whatever that thing is called). Sure, I think we can split into 2 patches, but I doubt it will make it any easier to get this feature completed. The contention point can delay the first patch (automatic parse of conf.d) which can lead to further delay of ALTER SYSTEM SET patch and that will eventually lead to its death. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> The main contention point I see is where conf.d lives;
> the two options are in $PGDATA or together with postgresql.conf. Tom
> and Robert, above, say it should be in $PGDATA; but this goes against
> Debian packaging and the Linux FHS (or whatever that thing is called).
Ordinarily, if postgresql.conf is not in $PGDATA, it will be somewhere
that the postmaster does not (and should not) have write permissions
for. I have no objection to inventiny a conf.d subdirectory, I just say
that it must be under $PGDATA. The argument that this is against FHS
is utter nonsense, because anything we write there is not static
configuration, it's just data.
Come to think of it, maybe part of the reason we're having such a hard
time getting to consensus is that people are conflating the "snippet"
part with the "writable" part? I mean, if you are thinking you want
system-management tools to be able to drop in configuration fragments as
separate files, there's a case to be made for a conf.d subdirectory that
lives somewhere that the postmaster can't necessarily write. We just
mustn't confuse that with support for ALTER SYSTEM SET. I strongly
believe that ALTER SYSTEM SET must not be designed to write anywhere
outside $PGDATA.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Come to think of it, maybe part of the reason we're having such a hard
> time getting to consensus is that people are conflating the "snippet"
> part with the "writable" part? I mean, if you are thinking you want
> system-management tools to be able to drop in configuration fragments as
> separate files, there's a case to be made for a conf.d subdirectory that
> lives somewhere that the postmaster can't necessarily write. We just
> mustn't confuse that with support for ALTER SYSTEM SET. I strongly
> believe that ALTER SYSTEM SET must not be designed to write anywhere
> outside $PGDATA.
Agreed. To continue that thought, I find it *very* unlikely that a
given environment would use *both* a tool like puppet to manage the
files in their conf.d *and* have people using ALTER SYSTEM SET. You're
going to do one or the other, almost certainly; not the least of which
is because those are very likely two different teams and only one of
them is going to be responsible for the PG system config.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Tom Lane <tgl@sss.pgh.pa.us> writes: > Come to think of it, maybe part of the reason we're having such a hard > time getting to consensus is that people are conflating the "snippet" > part with the "writable" part? I mean, if you are thinking you want > system-management tools to be able to drop in configuration fragments as > separate files, there's a case to be made for a conf.d subdirectory that > lives somewhere that the postmaster can't necessarily write. We just > mustn't confuse that with support for ALTER SYSTEM SET. I strongly > believe that ALTER SYSTEM SET must not be designed to write anywhere > outside $PGDATA. Agreed, +1. I think we should have conf.d in the same place as postgresql.conf, wherever that is for your distribution of choice (e.g. /etc), and certainly ALTER SYSTEM SET should stay away from those places. Come to think of it, the one-file-per-guc approach that received some push back when targeting the conf.d place might well be acceptable when targeting some place in $PGDATA, right? I still believe that's the best technical way to have at it. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit kapila
Date:
On Friday, July 26, 2013 6:18 PM Tom Lane wrote: Alvaro Herrera <alvherre@2ndquadrant.com> writes: >> The main contention point I see is where conf.d lives; >> the two options are in $PGDATA or together with postgresql.conf. Tom >> and Robert, above, say it should be in $PGDATA; but this goes against >> Debian packaging and the Linux FHS (or whatever that thing is called). > Ordinarily, if postgresql.conf is not in $PGDATA, it will be somewhere > that the postmaster does not (and should not) have write permissions > for. I have no objection to inventiny a conf.d subdirectory, I just say > that it must be under $PGDATA. The argument that this is against FHS > is utter nonsense, because anything we write there is not static > configuration, it's just data. > Come to think of it, maybe part of the reason we're having such a hard > time getting to consensus is that people are conflating the "snippet" > part with the "writable" part? I mean, if you are thinking you want > system-management tools to be able to drop in configuration fragments as > separate files, there's a case to be made for a conf.d subdirectory that > lives somewhere that the postmaster can't necessarily write. We just > mustn't confuse that with support for ALTER SYSTEM SET. I strongly > believe that ALTER SYSTEM SET must not be designed to write anywhere > outside $PGDATA. I think if we can design conf.d separately for config files of management tools, then it is better to have postgresql.auto.conf to be in $PGDATA rather than in $PGDATA/conf.d Kindly let me know if you feel otherwise, else I will update and send patch tomorrow. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
> I think if we can design conf.d separately for config files of management tools, then > it is better to have postgresql.auto.conf to be in $PGDATA rather than in > $PGDATA/conf.d One of the biggest current complaints about recovery.conf from Debian/Ubuntu users is the fact that it lives in $PGDATA. Won't we just get the same thing here? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Josh Berkus <josh@agliodbs.com> writes:
>> I think if we can design conf.d separately for config files of management tools, then
>> it is better to have postgresql.auto.conf to be in $PGDATA rather than in
>> $PGDATA/conf.d
> One of the biggest current complaints about recovery.conf from
> Debian/Ubuntu users is the fact that it lives in $PGDATA. Won't we just
> get the same thing here?
I don't think that's the same case, but ... why exactly don't they like
recovery.conf, and can you show that the location of the file has
anything to do with the underlying complaint? Personally I bet it's
more about the confusion between configuration and triggering functions.
Until we can get those things separated, using recovery.conf to argue
about file locations will result in nothing but confusion and bad
design.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Tom Lane <tgl@sss.pgh.pa.us> writes: >> One of the biggest current complaints about recovery.conf from >> Debian/Ubuntu users is the fact that it lives in $PGDATA. Won't we just >> get the same thing here? I don't think so, see below. > I don't think that's the same case, but ... why exactly don't they like > recovery.conf, and can you show that the location of the file has > anything to do with the underlying complaint? Personally I bet it's > more about the confusion between configuration and triggering functions. > Until we can get those things separated, using recovery.conf to argue > about file locations will result in nothing but confusion and bad > design. I think it's all about having a user-edited configuration file somewhere else than /etc, and that keeping the triggering functions separated from the setup itself would be a good thing to have. I bet that if some facts and links are necessary they will appear soon, now is not a right time for me to be searching for those facts… Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Sunday, July 28, 2013 11:12 AM Amit kapila wrote: > On Friday, July 26, 2013 6:18 PM Tom Lane wrote: > Alvaro Herrera <alvherre@2ndquadrant.com> writes: > >> The main contention point I see is where conf.d lives; > >> the two options are in $PGDATA or together with postgresql.conf. > Tom > >> and Robert, above, say it should be in $PGDATA; but this goes > against > >> Debian packaging and the Linux FHS (or whatever that thing is > called). > > > Ordinarily, if postgresql.conf is not in $PGDATA, it will be > somewhere > > that the postmaster does not (and should not) have write permissions > > for. I have no objection to inventiny a conf.d subdirectory, I just > say > > that it must be under $PGDATA. The argument that this is against FHS > > is utter nonsense, because anything we write there is not static > > configuration, it's just data. > > > Come to think of it, maybe part of the reason we're having such a > hard > > time getting to consensus is that people are conflating the "snippet" > > part with the "writable" part? I mean, if you are thinking you want > > system-management tools to be able to drop in configuration fragments > as > > separate files, there's a case to be made for a conf.d subdirectory > that > > lives somewhere that the postmaster can't necessarily write. We just > > mustn't confuse that with support for ALTER SYSTEM SET. I strongly > > believe that ALTER SYSTEM SET must not be designed to write anywhere > > outside $PGDATA. > > I think if we can design conf.d separately for config files of > management tools, then > it is better to have postgresql.auto.conf to be in $PGDATA rather than > in > $PGDATA/conf.d > > Kindly let me know if you feel otherwise, else I will update and send > patch > tomorrow. Modified patch to have postgresql.auto.conf in $PGDATA. Changes are as below: 1. initdb to create auto file in $PGDATA 2. ProcessConfigFile to open auto file from data directory, special case handling for initdb 3. AlterSystemSetConfigFile function to consider data directory as reference for operating on auto file 4. modified comments in code and docs to remove usage of config directory 5. modified function write_auto_conf_file() such that even if there is no configuration item to write, it should write header message. This is to handle case when there is only one parameter valueand user set it to default, before this modification ,it will write empty file. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Cédric Villemain
Date:
Le lundi 29 juillet 2013 13:47:57, Amit Kapila a écrit : > On Sunday, July 28, 2013 11:12 AM Amit kapila wrote: > > On Friday, July 26, 2013 6:18 PM Tom Lane wrote: > > > > Alvaro Herrera <alvherre@2ndquadrant.com> writes: > > >> The main contention point I see is where conf.d lives; > > >> the two options are in $PGDATA or together with postgresql.conf. > > > > Tom > > > > >> and Robert, above, say it should be in $PGDATA; but this goes > > > > against > > > > >> Debian packaging and the Linux FHS (or whatever that thing is > > > > called). > > > > > Ordinarily, if postgresql.conf is not in $PGDATA, it will be > > > > somewhere > > > > > that the postmaster does not (and should not) have write permissions > > > for. I have no objection to inventiny a conf.d subdirectory, I just > > > > say > > > > > that it must be under $PGDATA. The argument that this is against FHS > > > is utter nonsense, because anything we write there is not static > > > configuration, it's just data. > > > > > > Come to think of it, maybe part of the reason we're having such a > > > > hard > > > > > time getting to consensus is that people are conflating the "snippet" > > > part with the "writable" part? I mean, if you are thinking you want > > > system-management tools to be able to drop in configuration fragments > > > > as > > > > > separate files, there's a case to be made for a conf.d subdirectory > > > > that > > > > > lives somewhere that the postmaster can't necessarily write. We just > > > mustn't confuse that with support for ALTER SYSTEM SET. I strongly > > > believe that ALTER SYSTEM SET must not be designed to write anywhere > > > outside $PGDATA. > > > > I think if we can design conf.d separately for config files of > > management tools, then > > it is better to have postgresql.auto.conf to be in $PGDATA rather than > > in > > $PGDATA/conf.d > > > > Kindly let me know if you feel otherwise, else I will update and send > > patch > > tomorrow. > > Modified patch to have postgresql.auto.conf in $PGDATA. Changes are as > below: > > 1. initdb to create auto file in $PGDATA > 2. ProcessConfigFile to open auto file from data directory, special case > handling for initdb > 3. AlterSystemSetConfigFile function to consider data directory as > reference for operating on auto file > 4. modified comments in code and docs to remove usage of config directory > 5. modified function write_auto_conf_file() such that even if there is no > configuration item to write, it should write header message. > This is to handle case when there is only one parameter value and user > set it to default, before this modification ,it > will write empty file. I just read the patch, quickly. You may split the patch thanks to validate_conf_option(), however it is not a rule on postgresql-hacker. Why not harcode in ParseConfigFp() that we should parse the auto.conf file at the end (and/or if USE_AUTO_CONF is not OFF) instead of hacking ProcessConfigFile() with data_directory ? (data_directory should be set at this point) ... just thinking, a very convenient way to enable/disable that is just to add/remove the include directive in postgresql.conf. So no change should be required in ParseConf at all. Except maybe AbsoluteConfigLocation which should prefix the path to auto.conf.d with data_directory. What I like with the include directive is that Sysadmin can define some GUC *after* the auto.conf so he is sure those are not 'erased' by auto.conf (or by the DBA). Also, it looks very interesting to stick to an one-file-for-many-GUC when we absolutely don't care : this file should (MUST ?) not be edited by hand. The thing achieve is that it limits the access to ALTER SYSTEM. One file per GUC allows to LWlock only this GUC, isn't it ? (and also does not require machinery for holding old/new auto GUC, or at least more simple). It also prevent usage of ALTER SYSTEM for a cluster (as in replication) because this is not WAL logged. But it can be easier if trying to manage only one GUC at a time. I agree with Tom comment that this file(s) must be in data_directory. postgresql.auto.conf is useless, a "data_directory/auto.conf" (.d/ ?) is enough. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
> Why not harcode in ParseConfigFp() that we should parse the auto.conf file at > the end (and/or if USE_AUTO_CONF is not OFF) instead of hacking > ProcessConfigFile() with data_directory ? (data_directory should be set at this > point) ... just thinking, a very convenient way to enable/disable that is just > to add/remove the include directive in postgresql.conf. So no change should be > required in ParseConf at all. Except maybe AbsoluteConfigLocation which should > prefix the path to auto.conf.d with data_directory. What I like with the > include directive is that Sysadmin can define some GUC *after* the auto.conf so > he is sure those are not 'erased' by auto.conf (or by the DBA). Why do you think DBAs would like an option to disable this feature? I see no point in that. And being able to relocate the parsing of auto.conf to be in the middle of postgresql.conf instead of at the end ... that seems nightmarish. I mean, things are *already* nontrivial to follow, I don't see what would can come from a DBA running ALTER SYSTEM and wondering why their changes don't take. > Also, it looks very interesting to stick to an one-file-for-many-GUC when we > absolutely don't care : this file should (MUST ?) not be edited by hand. > The thing achieve is that it limits the access to ALTER SYSTEM. One file per > GUC allows to LWlock only this GUC, isn't it ? (and also does not require > machinery for holding old/new auto GUC, or at least more simple). This has already been debated, and we have already reached consensus (one file to rule them all). I don't think it's a good idea to go over all that discussion again. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
All, Based on the ongoing discussion of this patch, I have moved it to 9.4CF2 (9-2013). Mind you, it would be good to commit some version of it before September. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Euler Taveira
Date:
On 18-07-2013 17:02, Alvaro Herrera wrote: > I think we should just put the config directives of ALTER SYSTEM into a > file, not dir, alongside postgresql.conf; I would suggest > postgresql.auto.conf. This file is parsed automatically after > postgresql.conf, without requiring an "include" directive in > postgresql.conf. If the user does not want that file, they can remove > it; but a future ALTER SYSTEM will create the file again. No need to > parse stuff to find out whether the directory exists, or the file > exists, or such nonsense. > +1. Let's keep it as simple as possible. There will be the possibility to revisit the directory and/or include* ideas. > I think the only drawback of this is that there's no way to disable > ALTER SYSTEM (i.e. there's no directory you can remove to prevent the > thing from working). > Disable ALTER SYSTEM? No, thanks. Change the postgresql.auto.conf permissions or ownership is an elegant way to disable it. -- Euler Taveira Timbira - http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte24x7 e Treinamento
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Monday, July 29, 2013 7:15 PM Cédric Villemain wrote: > Le lundi 29 juillet 2013 13:47:57, Amit Kapila a écrit : > > On Sunday, July 28, 2013 11:12 AM Amit kapila wrote: > > > On Friday, July 26, 2013 6:18 PM Tom Lane wrote: > > > > > > Alvaro Herrera <alvherre@2ndquadrant.com> writes: > > > >> The main contention point I see is where conf.d lives; the two > > > >> options are in $PGDATA or together with postgresql.conf. > > > > > > Tom > > > > > > >> and Robert, above, say it should be in $PGDATA; but this goes > > > > > > against > > > > > > >> Debian packaging and the Linux FHS (or whatever that thing is > > > > > > called). > > > > > > > Ordinarily, if postgresql.conf is not in $PGDATA, it will be > > > > > > somewhere > > > > > > > that the postmaster does not (and should not) have write > > > > permissions for. I have no objection to inventiny a conf.d > > > > subdirectory, I just > > > > > > say > > > > > > > that it must be under $PGDATA. The argument that this is against > > > > FHS is utter nonsense, because anything we write there is not > > > > static configuration, it's just data. > > > > > > > > Come to think of it, maybe part of the reason we're having such a > > > > > > hard > > > > > > > time getting to consensus is that people are conflating the > "snippet" > > > > part with the "writable" part? I mean, if you are thinking you > > > > want system-management tools to be able to drop in configuration > > > > fragments > > > > > > as > > > > > > > separate files, there's a case to be made for a conf.d > > > > subdirectory > > > > > > that > > > > > > > lives somewhere that the postmaster can't necessarily write. We > > > > just mustn't confuse that with support for ALTER SYSTEM SET. I > > > > strongly believe that ALTER SYSTEM SET must not be designed to > > > > write anywhere outside $PGDATA. > > > > > > I think if we can design conf.d separately for config files of > > > management tools, then it is better to have postgresql.auto.conf to > > > be in $PGDATA rather than in $PGDATA/conf.d > > > > > > Kindly let me know if you feel otherwise, else I will update and > > > send patch tomorrow. > > > > Modified patch to have postgresql.auto.conf in $PGDATA. Changes are > as > > below: > > > > 1. initdb to create auto file in $PGDATA 2. ProcessConfigFile to open > > auto file from data directory, special case handling for initdb 3. > > AlterSystemSetConfigFile function to consider data directory as > > reference for operating on auto file 4. modified comments in code and > > docs to remove usage of config directory 5. modified function > > write_auto_conf_file() such that even if there is no configuration > > item to write, it should write header message. > > This is to handle case when there is only one parameter value and > > user set it to default, before this modification ,it > > will write empty file. > > I just read the patch, quickly. Thank you for review. > You may split the patch thanks to validate_conf_option(), however it is > not a rule on postgresql-hacker. The review of the core functionality of patch has been done before the introduction of function validate_conf_option() in the patch. It was introduced because there were some common parts incore implementation of AlterSystem and set_config_option(). I am really not sure, after having multiple round of reviews by reviewers, it can add significant value to split it. > Why not harcode in ParseConfigFp() that we should parse the auto.conf > file at the end (and/or if USE_AUTO_CONF is not OFF) instead of > hacking > ProcessConfigFile() with data_directory ? (data_directory should be set > at this > point) ... No data_directory will not be set by that time incase of initdb, when ProcessConfigFile() is called from SelectConfigFiles() > just thinking, a very convenient way to enable/disable that > is just to add/remove the include directive in postgresql.conf. So no > change should be required in ParseConf at all. Except maybe > AbsoluteConfigLocation which should prefix the path to auto.conf.d with > data_directory. What I like with the include directive is that Sysadmin > can define some GUC *after* the auto.conf so he is sure those are not > 'erased' by auto.conf (or by the DBA). I think earlier versions have this implementation, but later based on suggestions, I have changed it to automatic parsing of auto file after postgresql.conf > Also, it looks very interesting to stick to an one-file-for-many-GUC > when we absolutely don't care : this file should (MUST ?) not be edited > by hand. > The thing achieve is that it limits the access to ALTER SYSTEM. One > file per GUC allows to LWlock only this GUC, isn't it ? (and also does > not require machinery for holding old/new auto GUC, or at least more > simple). > > It also prevent usage of ALTER SYSTEM for a cluster (as in replication) > because this is not WAL logged. But it can be easier if trying to > manage only one GUC at a time. > > I agree with Tom comment that this file(s) must be in data_directory. > postgresql.auto.conf is useless, a "data_directory/auto.conf" (.d/ ?) > is enough. There were multiple suggestions for names, but I have kept name postgresql.auto.conf based on more votes for it and consensus. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tuesday, July 30, 2013 1:17 AM Josh Berkus wrote: On Tuesday, July 30, 2013 1:17 AM Amit kapila wrote: > All, > > Based on the ongoing discussion of this patch, I have moved it to > 9.4CF2 > (9-2013). I had already sent the updated patch based on recent suggestions. I am really not sure if something is pending from myside, so bit confused by the status of patch as Waiting On Author? > Mind you, it would be good to commit some version of it before > September. True, I am also hoping for the same. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Cédric Villemain
Date:
Le lundi 29 juillet 2013 18:03:21, Alvaro Herrera a écrit : > > Why not harcode in ParseConfigFp() that we should parse the auto.conf > > file at the end (and/or if USE_AUTO_CONF is not OFF) instead of > > hacking ProcessConfigFile() with data_directory ? (data_directory should > > be set at this point) ... just thinking, a very convenient way to > > enable/disable that is just to add/remove the include directive in > > postgresql.conf. So no change should be required in ParseConf at all. > > Except maybe AbsoluteConfigLocation which should prefix the path to > > auto.conf.d with data_directory. What I like with the include directive > > is that Sysadmin can define some GUC *after* the auto.conf so he is sure > > those are not 'erased' by auto.conf (or by the DBA). > > Why do you think DBAs would like an option to disable this feature? I > see no point in that. And being able to relocate the parsing of > auto.conf to be in the middle of postgresql.conf instead of at the end > ... that seems nightmarish. I mean, things are *already* nontrivial to > follow, I don't see what would can come from a DBA running ALTER SYSTEM > and wondering why their changes don't take. I don't find that hard to do nor to understand, but if that has already reach a consensus then let's do that. > > Also, it looks very interesting to stick to an one-file-for-many-GUC when > > we absolutely don't care : this file should (MUST ?) not be edited by > > hand. The thing achieve is that it limits the access to ALTER SYSTEM. > > One file per GUC allows to LWlock only this GUC, isn't it ? (and also > > does not require machinery for holding old/new auto GUC, or at least > > more simple). > > This has already been debated, and we have already reached consensus > (one file to rule them all). I don't think it's a good idea to go over > all that discussion again. ok, I've only lost track for the consensus based on the technical objective. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Cédric Villemain
Date:
Le mardi 30 juillet 2013 05:27:12, Amit Kapila a écrit : > On Monday, July 29, 2013 7:15 PM Cédric Villemain wrote: > > Le lundi 29 juillet 2013 13:47:57, Amit Kapila a écrit : > > > On Sunday, July 28, 2013 11:12 AM Amit kapila wrote: > > > > On Friday, July 26, 2013 6:18 PM Tom Lane wrote: > > > > > > > > Alvaro Herrera <alvherre@2ndquadrant.com> writes: > > > > >> The main contention point I see is where conf.d lives; the two > > > > >> options are in $PGDATA or together with postgresql.conf. > > > > > > > > Tom > > > > > > > > >> and Robert, above, say it should be in $PGDATA; but this goes > > > > > > > > against > > > > > > > > >> Debian packaging and the Linux FHS (or whatever that thing is > > > > > > > > called). > > > > > > > > > Ordinarily, if postgresql.conf is not in $PGDATA, it will be > > > > > > > > somewhere > > > > > > > > > that the postmaster does not (and should not) have write > > > > > permissions for. I have no objection to inventiny a conf.d > > > > > subdirectory, I just > > > > > > > > say > > > > > > > > > that it must be under $PGDATA. The argument that this is against > > > > > FHS is utter nonsense, because anything we write there is not > > > > > static configuration, it's just data. > > > > > > > > > > Come to think of it, maybe part of the reason we're having such a > > > > > > > > hard > > > > > > > > > time getting to consensus is that people are conflating the > > > > "snippet" > > > > > > > part with the "writable" part? I mean, if you are thinking you > > > > > want system-management tools to be able to drop in configuration > > > > > fragments > > > > > > > > as > > > > > > > > > separate files, there's a case to be made for a conf.d > > > > > subdirectory > > > > > > > > that > > > > > > > > > lives somewhere that the postmaster can't necessarily write. We > > > > > just mustn't confuse that with support for ALTER SYSTEM SET. I > > > > > strongly believe that ALTER SYSTEM SET must not be designed to > > > > > write anywhere outside $PGDATA. > > > > > > > > I think if we can design conf.d separately for config files of > > > > management tools, then it is better to have postgresql.auto.conf to > > > > be in $PGDATA rather than in $PGDATA/conf.d > > > > > > > > Kindly let me know if you feel otherwise, else I will update and > > > > send patch tomorrow. > > > > > > Modified patch to have postgresql.auto.conf in $PGDATA. Changes are > > > > as > > > > > below: > > > > > > 1. initdb to create auto file in $PGDATA 2. ProcessConfigFile to open > > > auto file from data directory, special case handling for initdb 3. > > > AlterSystemSetConfigFile function to consider data directory as > > > reference for operating on auto file 4. modified comments in code and > > > docs to remove usage of config directory 5. modified function > > > write_auto_conf_file() such that even if there is no configuration > > > item to write, it should write header message. > > > > > > This is to handle case when there is only one parameter value and > > > > > > user set it to default, before this modification ,it > > > > > > will write empty file. > > > > I just read the patch, quickly. > > Thank you for review. > > > You may split the patch thanks to validate_conf_option(), however it is > > not a rule on postgresql-hacker. > > The review of the core functionality of patch has been done before the > introduction of function > validate_conf_option() in the patch. It was introduced because there > were some common parts in core implementation of AlterSystem and > set_config_option(). > I am really not sure, after having multiple round of reviews by > reviewers, it can add > significant value to split it. Yes, it just appeared that this part was a significant one in the patch. I understand that it is not interesting to split now. > > > Why not harcode in ParseConfigFp() that we should parse the auto.conf > > file at the end (and/or if USE_AUTO_CONF is not OFF) instead of > > hacking > > ProcessConfigFile() with data_directory ? (data_directory should be set > > at this > > point) ... > > No data_directory will not be set by that time incase of initdb, when > ProcessConfigFile() > is called from SelectConfigFiles() ok. > > just thinking, a very convenient way to enable/disable that > > is just to add/remove the include directive in postgresql.conf. So no > > change should be required in ParseConf at all. Except maybe > > AbsoluteConfigLocation which should prefix the path to auto.conf.d with > > data_directory. What I like with the include directive is that Sysadmin > > can define some GUC *after* the auto.conf so he is sure those are not > > 'erased' by auto.conf (or by the DBA). > > I think earlier versions have this implementation, but later based on > suggestions, I have changed it to automatic parsing of auto file after > postgresql.conf I probably missed those suggestions. > > Also, it looks very interesting to stick to an one-file-for-many-GUC > > when we absolutely don't care : this file should (MUST ?) not be edited > > by hand. > > The thing achieve is that it limits the access to ALTER SYSTEM. One > > file per GUC allows to LWlock only this GUC, isn't it ? (and also does > > not require machinery for holding old/new auto GUC, or at least more > > simple). > > > > It also prevent usage of ALTER SYSTEM for a cluster (as in replication) > > because this is not WAL logged. But it can be easier if trying to > > manage only one GUC at a time. > > > > I agree with Tom comment that this file(s) must be in data_directory. > > postgresql.auto.conf is useless, a "data_directory/auto.conf" (.d/ ?) > > is enough. > > There were multiple suggestions for names, but I have kept name > postgresql.auto.conf based > on more votes for it and consensus. yeah, I vote here for a shorter name, but it changes nothing to the patch. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Smith
Date:
On 7/28/13 10:49 PM, Tom Lane wrote: > Josh Berkus <josh@agliodbs.com> writes: >>> I think if we can design conf.d separately for config files of management tools, then >>> it is better to have postgresql.auto.conf to be in $PGDATA rather than in >>> $PGDATA/conf.d > >> One of the biggest current complaints about recovery.conf from >> Debian/Ubuntu users is the fact that it lives in $PGDATA. Won't we just >> get the same thing here? > > I don't think that's the same case, but ... why exactly don't they like > recovery.conf, and can you show that the location of the file has > anything to do with the underlying complaint? Here is a quick survey of config files: RHEL6: Cluster configuration file at /etc/sysconfig/pgsql, writeable by root. postgresql.conf is in $HOME/8.4/data Debian Squeeze: All configuration files in /etc/postgresql/8.4/main and are owned by the postgres user. That last one shows why there's a complaint, and why it's 75% due to location rather than split role. Debian users expect all of the config files to be relocated to /etc/postgresql or /etc/postgresql-common. Those files are also writeable by the postgres user. recovery.conf doesn't fit that model, and people don't like that. The admin who has to manage servers with puppet would be much happier if recovery.conf was in the /etc/postgresql tree they are already managing. I get this complaint every single time I introduce an experienced Debian admin to PostgreSQL replication setup. They aren't sure whether it's the packagers or PostgreSQL that is to blame for recovery.conf being in $PGDATA instead of /etc, but it's obvious to them someone has screwed up. Given that Debian is the major example of a relocated postgresql.conf in the field, and that work has been done such that a) all config files are in /etc and b) they are all writeable by the postgres user, I don't understand why people are suggesting the auto.conf file or conf.d directory will go anywhere else on that platform anymore. I think the only claims that are suggesting otherwise are thinking about an older, now unsupported version of the Debian packaging. Debian packaging absolutely will want to relocate any new files added here into their /etc directory tree as well, where they will be writeable. They will not go into the $PGDATA directory. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
Amit, > I had already sent the updated patch based on recent suggestions. Yes, but there is no consensus yet on certain fundamental issues, such as: 1. what directory should postgresql.auto.conf live in? 2. should admins be able to turn it "off"? Once we have consensus on these issues, then hopefully we can commit your patch. I don't *think* anyone is arguing with the code at this stage. Note that I count the name of the file as something on which we *have* consensus. While I personally don't like the name, I like arguing about names even less. We had a vote, let's stick with what we have. > I am really not sure if something is pending from myside, > so bit confused by the status of patch as Waiting On Author? It's our policy that patches which are under spec discussion are marked "waiting on author". -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Josh Berkus escribió: > Amit, > > > I had already sent the updated patch based on recent suggestions. > > Yes, but there is no consensus yet on certain fundamental issues, such as: > > 1. what directory should postgresql.auto.conf live in? > > 2. should admins be able to turn it "off"? > > Once we have consensus on these issues, then hopefully we can commit > your patch. I don't *think* anyone is arguing with the code at this stage. There's one other thing, raised by Cédric upthread, about the multiple files vs. single file. We decided on a single file because Greg said many little files scattered in conf.d/ would be a mess; but if we're going to have two new items, one conf.d/ for snippets from external tools (living in the config directory) and one for ALTER SYSTEM (living in PGDATA), then the idea of going back to one setting per file is perhaps not bad. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Stark
Date:
On Tue, Jul 30, 2013 at 6:10 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > one for ALTER SYSTEM (living > in PGDATA), then the idea of going back to one setting per file is > perhaps not bad. Well more to the point, if we conf.d for sysadmins to drop in extra snippets in a different place then we could drop conf.d in PGDATA since there's no need for it any more and just have auto.conf in PGDATA directly. -- greg
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 07/30/2013 10:10 AM, Alvaro Herrera wrote: >> Once we have consensus on these issues, then hopefully we can commit >> your patch. I don't *think* anyone is arguing with the code at this stage. > > There's one other thing, raised by Cédric upthread, about the multiple > files vs. single file. We decided on a single file because Greg said > many little files scattered in conf.d/ would be a mess; but if we're That's not what I get out of the discussion thread. I believed we settled on One File because that's the patch Amit wrote, and a patch in the hand is worth two in WIP. > going to have two new items, one conf.d/ for snippets from external > tools (living in the config directory) and one for ALTER SYSTEM (living > in PGDATA), then the idea of going back to one setting per file is > perhaps not bad. On 07/30/2013 10:28 AM, Greg Stark wrote:> On Tue, Jul 30, 2013 at 6:10 PM, Alvaro Herrera > Well more to the point, if we conf.d for sysadmins to drop in extra > snippets in a different place then we could drop conf.d in PGDATA > since there's no need for it any more and just have auto.conf in > PGDATA directly. That assumes that the only reason we have for a conf.d is to support auto.conf, which I don't agree with; I personally have a need for conf.d which has nothing to do with auto.conf. Also, see gsmith's point about forcing auto.conf to be in PGDATA as not satisfactory for Debian users (and others). -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Stark
Date:
On Tue, Jul 30, 2013 at 6:40 PM, Josh Berkus <josh@agliodbs.com> wrote: > > On 07/30/2013 10:28 AM, Greg Stark wrote:> On Tue, Jul 30, 2013 at 6:10 > PM, Alvaro Herrera >> Well more to the point, if we conf.d for sysadmins to drop in extra >> snippets in a different place then we could drop conf.d in PGDATA >> since there's no need for it any more and just have auto.conf in >> PGDATA directly. > > That assumes that the only reason we have for a conf.d is to support > auto.conf, which I don't agree with; I personally have a need for conf.d > which has nothing to do with auto.conf. No, wait. We had two reasons for conf.d and we thought we could kill two birds with one stone. It seems it's turning out that we can't kill both birds with the same stone because we want to separate the user maintained config from the automatically maintained config even further than just having them in separate files. Now we need them in separate directories. If you have a need for conf.d which has nothing to do with auto.conf then you presumably will want to be editing the system conf.d which might be in PGDATA or might be in /etc or might be someplace else. I totally agree that conf.d is useful for sysadmins as well as for distribution authors. But if we're going to insist that conf.d be in PGDATA then I'm saying we don't need a second conf.d just to contain that one file. And if we let distributions and sysadmins drop files in there then we'll be back to square 0 with wanting to separate them. I guess what I'm saying is that auto.conf shouldn't be in conf.d after all. It should just be in PGDATA directly and trying to kill two birds with one stone led us astray. Putting it in conf.d creates a raft of problems of what if conf.d is moved and what if it's shared between databases and what if the user puts files that come after it, etc. Putting it directly in PGDATA lets us manage those issues however we want and ties the automatic file to a specific database regardless of how we handle conf.d in the future. > Also, see gsmith's point about forcing auto.conf to be in PGDATA as not > satisfactory for Debian users (and others). Greg Smith's argument was about recovery.conf which is a file that users are expected to edit. A file which user's are not expected to edit and is maintained by the software is no more a configuration file than pg_auth or pg_database which are actually being stored in the database itself. Having an automatically maintained file and a manually maintained file is going to raise some tricky problems. In Oracle they have exactly the same issue. In that case the automatically maintained one is actually kept in a binary format. You can dump it out in text format but unless you switch which one you're using it doesn't read the text format file at all. I assume they did this because working out the conflicts between the two was just too complex for users. I think we're fine with allowing users to use both but we should try to keep the two as separate as possible to avoid confusion. Keeping the auto.conf inside conf.d sounds like it will actually confuse users about which files they're supposed to edit and which belong to the system. -- greg
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 07/30/2013 11:12 AM, Greg Stark wrote: > But if we're going to insist that conf.d be in PGDATA then I'm saying > we don't need a second conf.d just to contain that one file. And if we > let distributions and sysadmins drop files in there then we'll be back > to square 0 with wanting to separate them. Ah, misunderstood your point. We certainly don't need *two* conf.d's. > Greg Smith's argument was about recovery.conf which is a file that > users are expected to edit. A file which user's are not expected to > edit and is maintained by the software is no more a configuration file > than pg_auth or pg_database which are actually being stored in the > database itself. I don't think you understood GSmith's argument. For Debian/Ubuntu sysadmins, configuration files live in /etc/, *period*. Even ones which were automatically generated. The packagers and scripters of that OS have taken a significant amount of trouble to make that so, including writing their own wrapper scripts around utilities for Postgres, Apache, etc., in order to support putting all configuration files in /etc/. Now we're proposing to break that *again*, by putting yet another configuration file in PGDATA, and making it impossible to relocate. That's fairly hostile to the concerns of possibly our most popular OSes. Saying "it's not a configuration file because it's automatic" is rather disingenuous. If you can set work_mem via postgresql.auto.conf, it's a configuration file, regardless of the means you used to set it. > Having an automatically maintained file and a manually maintained file > is going to raise some tricky problems. In Oracle they have exactly > the same issue. In that case the automatically maintained one is > actually kept in a binary format. You can dump it out in text format > but unless you switch which one you're using it doesn't read the text > format file at all. I assume they did this because working out the > conflicts between the two was just too complex for users. I'm not sure we won't end up in the same place as Oracle, eventually. We'll see. Re: allowing users to disable ALTER SYSTEM SET, I think we'll have to implement that before 9.4. comes out, for the simple reason that users of Puppet and other large-scale configuration management utilities will demand it. If you're managing 120 PostgreSQL servers using centrally version-controlled Chef recipes, the last thing in the world you want is having your DBAs bypass that via ALTER SYSTEM SET. HOWEVER, I think adding the ability to disable ALTER SYSTEM SET can (and should) be a second, separate patch. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Stark
Date:
<p dir="ltr"><br /> On Jul 30, 2013 7:32 PM, "Josh Berkus" <<a href="mailto:josh@agliodbs.com">josh@agliodbs.com</a>>wrote:<br /> ><br /> ><br /> > I don't think you understoodGSmith's argument. For Debian/Ubuntu<br /> > sysadmins, configuration files live in /etc/, *period*. Evenones which<br /> > were automatically generated. The packagers and scripters of that OS<br /> > have taken asignificant amount of trouble to make that so, including<br /> > writing their own wrapper scripts around utilities forPostgres, Apache,<br /> > etc., in order to support putting all configuration files in /etc/.<br /> ><br /><p dir="ltr">Iused to be a Debian package maintainer. I packaged kerberos and afs and a number of smaller packages. I know whyDebian separates config files.<p dir="ltr">If a file is automatically maintained then it's internal program state justlike the list of users, databases, tables, etc.<p dir="ltr">Programs keep their state in /var. It would be just as wrongto put auto.conf in /etc as it is to put the existing postgresql.conf in PGDATA.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Greg Stark (stark@mit.edu) wrote:
> I used to be a Debian package maintainer. I packaged kerberos and afs and a
> number of smaller packages. I know why Debian separates config files.
Ditto (though I suppose I'm still technically "active").
> If a file is automatically maintained then it's internal program state just
> like the list of users, databases, tables, etc.
Right.
> Programs keep their state in /var. It would be just as wrong to put
> auto.conf in /etc as it is to put the existing postgresql.conf in PGDATA.
Agreed.
Though, I have to say that I think we need "pg_hba.d" and "pg_ident.d"
more than we need a conf.d for postgresql.conf snippets.. And ditto for
auto.conf-style managed-through-PG options for pg_hba and pg_ident.
Apologies for not having been able to keep up with this thread fully, so
feel free to point out if this has already been discussed. :)
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tuesday, July 30, 2013 10:25 PM Josh Berkus wrote:
> Amit,
>
> > I had already sent the updated patch based on recent suggestions.
>
> Yes, but there is no consensus yet on certain fundamental issues, such
> as:
>
> 1. what directory should postgresql.auto.conf live in?
Current situation is that Greg Smith has provided a usecase for all config files to be present in location other than
$PGDATA,but based on further suggestions given by Greg Stark and Stephen Frost it is clear that keeping
postgresql.auto.confwill not harm the users of Debian.
More to that point, I think if we try to address conf.d (for maintaining config files) along with this Patch, it will
notbe possible to get every one on same plane.
I think apart from Greg Stark and Stephen Frost other people (Tom, Robert) are of opinion that auto file should be
placedin $PGDATA, so shouldn't we consider this as consensus and move forward for this point unless Tom or Robert have
changedtheir view based on usecase provided by Greg Smith?
> 2. should admins be able to turn it "off"?
IIRC there are not many people who want this feature to be turned off. Peter E. had asked seeing the complexity,
butin general I don't think This is a blocking point for patch, am I missing something here?
> Once we have consensus on these issues, then hopefully we can commit
> your patch. I don't *think* anyone is arguing with the code at this
> stage.
Sure, I understand that it is important to have consensus and agreement on above points.
With Regards,
Amit Kapila.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Greg Stark <stark@mit.edu> writes: > Greg Smith's argument was about recovery.conf which is a file that > users are expected to edit. A file which user's are not expected to > edit and is maintained by the software is no more a configuration file > than pg_auth or pg_database which are actually being stored in the > database itself. +1 > I think we're fine with allowing users to use both but we should try > to keep the two as separate as possible to avoid confusion. Keeping > the auto.conf inside conf.d sounds like it will actually confuse users > about which files they're supposed to edit and which belong to the > system. +1 I think we need both an ALTER SYSTEM SET implementation using files somewhere in $PGDATA, and a separate conf.d facility that lives alongside wherever the main postgresql.conf is maintained on your OS of choice. Also, now that we have decided to separate away those two fundamentally different aspects of the configuration setup, I join Álvaro and Cédric in thinking that we should review the implementation choice of the ALTER SYSTEM SET facility, and vote for having one-file-per-GUC. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Stark
Date:
On Thu, Aug 1, 2013 at 1:12 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote: > we should review the implementation choice of the ALTER > SYSTEM SET facility, and vote for having one-file-per-GUC. Zombie crazy design idea arise! I think people are going to laugh at us if an open source database software can't manage a simple flat file database of settings, especially one that is purely write-only and can be a simple dump of settings that are set by alter system. -- greg
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Greg Stark <stark@mit.edu> writes: > I think people are going to laugh at us if an open source database > software can't manage a simple flat file database of settings, > especially one that is purely write-only and can be a simple dump of > settings that are set by alter system. So you say it's easier to implement per-GUC locking semantics correctly when using a single file with multiple units of information that all are of the same type? Interesting. Maybe the storage should actually be a shared catalog, in fact. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-01 15:55:25 +0200, Dimitri Fontaine wrote: > Greg Stark <stark@mit.edu> writes: > > I think people are going to laugh at us if an open source database > > software can't manage a simple flat file database of settings, > > especially one that is purely write-only and can be a simple dump of > > settings that are set by alter system. Why would using a single-file solution imply that we cannot do the other? > So you say it's easier to implement per-GUC locking semantics correctly > when using a single file with multiple units of information that all are > of the same type? Interesting. > Maybe the storage should actually be a shared catalog, in fact. We can't read those early enough. Think a) of options that can only be set at postmaster start b) think of crash recovery. We can't read the catalog till we're in a consistent state. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Stark
Date:
<p dir="ltr">We don't need per guc locking. This is the whole objection Tom had about this patch being more complex thanit has to be.<p dir="ltr">-- <br /> greg<div class="gmail_quote">On 1 Aug 2013 14:55, "Dimitri Fontaine" <<a href="mailto:dimitri@2ndquadrant.fr">dimitri@2ndquadrant.fr</a>>wrote:<br type="attribution" /><blockquote class="gmail_quote"style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Greg Stark <<a href="mailto:stark@mit.edu">stark@mit.edu</a>>writes:<br /> > I think people are going to laugh at us if an open sourcedatabase<br /> > software can't manage a simple flat file database of settings,<br /> > especially one that ispurely write-only and can be a simple dump of<br /> > settings that are set by alter system.<br /><br /> So you sayit's easier to implement per-GUC locking semantics correctly<br /> when using a single file with multiple units of informationthat all are<br /> of the same type? Interesting.<br /><br /> Maybe the storage should actually be a shared catalog,in fact.<br /><br /> Regards,<br /> --<br /> Dimitri Fontaine<br /><a href="http://2ndQuadrant.fr" target="_blank">http://2ndQuadrant.fr</a> PostgreSQL : Expertise, Formation et Support<br /><br /><br /> --<br /> Sentvia pgsql-hackers mailing list (<a href="mailto:pgsql-hackers@postgresql.org">pgsql-hackers@postgresql.org</a>)<br />To make changes to your subscription:<br /><a href="http://www.postgresql.org/mailpref/pgsql-hackers" target="_blank">http://www.postgresql.org/mailpref/pgsql-hackers</a><br/></blockquote></div>
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-01 15:17:04 +0100, Greg Stark wrote: > We don't need per guc locking. This is the whole objection Tom had about > this patch being more complex than it has to be. IIRC he objected to using locking *at all* because a simple one-file-per-setting approach should be used. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Stark
Date:
<p dir="ltr">Why isn't it enough to just dump out all variables with a source of alter system to a text file? You can eitherhave a single global lock around that operation or write it to a new file and move it into place. <p dir="ltr">-- <br/> greg<div class="gmail_quote">On 1 Aug 2013 15:19, "Andres Freund" <<a href="mailto:andres@2ndquadrant.com">andres@2ndquadrant.com</a>>wrote:<br type="attribution" /><blockquote class="gmail_quote"style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> On 2013-08-01 15:17:04 +0100, GregStark wrote:<br /> > We don't need per guc locking. This is the whole objection Tom had about<br /> > this patchbeing more complex than it has to be.<br /><br /> IIRC he objected to using locking *at all* because a simple<br />one-file-per-setting approach should be used.<br /><br /> Greetings,<br /><br /> Andres Freund<br /><br /> --<br /> AndresFreund <a href="http://www.2ndQuadrant.com/" target="_blank">http://www.2ndQuadrant.com/</a><br/> PostgreSQL Development, 24x7 Support, Training & Services<br /></blockquote></div>
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
David Johnston
Date:
Minor request: could someone enlighten me as to why making the directory location a compile-time option is undesirable. Packagers then can setup whatever structure they desire when they compile their distributions. In which case the discussion becomes what is a reasonable default and that can be made with respect of other defaults that are in place for people that would self-compile. I get the "supporting users - telling them where to go to find these files" aspect but I believe that ship has already sailed. The goal should be to make it as easy as possible to allow distributions and/or individual users to integrate PostgreSQL into their normal routine as possible. It isn't like we are adding unneeded complexity since it is obvious from the discussion that where files/directories are placed in the file system is a major variable. Enforcing $PGDATA when we know Debian is going to be upset doesn't seem to be that great an idea - it isn't like we are going to suddenly make them realize they have been doing things incorrectly all this time. I am not familiar with all of the configurations but I do recall that the location of postgres.conf and related files is already distribution specific so why shouldn't these extensions be as well? Sorry if this was discussed previously; I'll go look deeper in the thread if someone confirms that indeed it is. David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/Proposal-for-Allow-postgresql-conf-values-to-be-changed-via-SQL-tp5729917p5765892.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
Hi, On 2013-08-01 15:40:22 +0100, Greg Stark wrote: > Why isn't it enough to just dump out all variables with a source of alter > system to a text file? You can either have a single global lock around that > operation or write it to a new file and move it into place. It saves you from several complications: * No need to iterate over all GUCs, figuring out which come from which source, when writing out the file. * Less logic required when writing out a value, since you have an acceptable input ready. * No need to make sure the autogenerated file is written out in the same order when adding/changing a setting (to make sureyou can diff/version control it sensibly) * No locking necessary, without locking concurrent changes can vanish. * Way easier to delete a setting if it ends up stopping postgres from starting. Greetings, Andres Freund PS: .oO(quoting?) -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 08/01/2013 07:47 AM, David Johnston wrote: > Minor request: could someone enlighten me as to why making the directory > location a compile-time option is undesirable. Packagers then can setup > whatever structure they desire when they compile their distributions. In > which case the discussion becomes what is a reasonable default and that can > be made with respect of other defaults that are in place for people that > would self-compile. Hey, that's a good idea. Anyone else? On 08/01/2013 06:32 AM, Greg Stark wrote:> On Thu, Aug 1, 2013 at 1:12 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote: >> we should review the implementation choice of the ALTER >> SYSTEM SET facility, and vote for having one-file-per-GUC. > > Zombie crazy design idea arise! > > I think people are going to laugh at us if an open source database > software can't manage a simple flat file database of settings, > especially one that is purely write-only and can be a simple dump of > settings that are set by alter system. While I find some value in the one-setting-per-file approach, there's also some major issues with it. And we already argued this out months ago, and ended up with the current single-file approach. Let's not rehash the past infinitely, please? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Josh Berkus <josh@agliodbs.com> writes: > While I find some value in the one-setting-per-file approach, there's > also some major issues with it. And we already argued this out months > ago, and ended up with the current single-file approach. Let's not > rehash the past infinitely, please? We rehash because the situation did change *a lot*. We just decided that the ALTER SYSTEM SET setup will live in PGDATA and will not have to be edited by DBA nor sysadmin nor tools ever. We will have a separate facility (conf.d) for that. As a result, I don't think there's any issues left with one-setting-per-file now. So yes, I think it makes sense to review our position and make the thing as easy as possible to code and maintain. See Andres' list about that, earlier today on this same thread. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 07/26/2013 12:19 PM, Stephen Frost wrote: > Agreed. To continue that thought, I find it *very* unlikely that a > given environment would use *both* a tool like puppet to manage the > files in their conf.d *and* have people using ALTER SYSTEM SET. You're > going to do one or the other, almost certainly; not the least of which > is because those are very likely two different teams and only one of > them is going to be responsible for the PG system config. Ideally, yes. And that's the reason why I think that we will need to implement a way to disable ALTER SYSTEM SET in postgresql.conf before 9.4.0 is done. The big-Puppet-management shops will demand it; they do NOT want their DBAs setting unversioned settings in isolation on one database server out of 200. However, I can imagine "hybrid" approaches. For example, my personal main use for conf.d/ is to modify logging settings on a temporary basis.I can imagine DBAs using ALTER SYSTEM SET for thispurpose as well, that is just to turn log_min_duration_statement down to 0 and then back up again. In that case, most of the "live" settings would live in /etc/postgresql, but some of the logging settings would be controlled with ALTER SYSTEM SET> That conceptually seems to work fine with the existing design, I just wanted to bring it up as a likely use case. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
Dimitri, > We rehash because the situation did change *a lot*. We just decided that > the ALTER SYSTEM SET setup will live in PGDATA and will not have to be > edited by DBA nor sysadmin nor tools ever. We will have a separate > facility (conf.d) for that. As a result, I don't think there's any > issues left with one-setting-per-file now. Let's please NOT call it conf.d if it's living in PGDATA and is not meant to be edited by hand. conf.d is for a directory of config files created by users and external utilities, living in CONFIGDIR. If we're going to have a directory of individual file settings in PGDATA, let's be consistent with the existing command and call it "system_set". -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-01 10:13:37 -0700, Josh Berkus wrote: > On 07/26/2013 12:19 PM, Stephen Frost wrote: > > Agreed. To continue that thought, I find it *very* unlikely that a > > given environment would use *both* a tool like puppet to manage the > > files in their conf.d *and* have people using ALTER SYSTEM SET. You're > > going to do one or the other, almost certainly; not the least of which > > is because those are very likely two different teams and only one of > > them is going to be responsible for the PG system config. > > Ideally, yes. And that's the reason why I think that we will need to > implement a way to disable ALTER SYSTEM SET in postgresql.conf before > 9.4.0 is done. The big-Puppet-management shops will demand it; they do > NOT want their DBAs setting unversioned settings in isolation on one > database server out of 200. They would need a setting that disables ALTER (DATABASE|USER) ... SET ... as well though. At least for some settings. I don't think enforcing things on that level makes much sense. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-01 10:16:59 -0700, Josh Berkus wrote: > Dimitri, > > > We rehash because the situation did change *a lot*. We just decided that > > the ALTER SYSTEM SET setup will live in PGDATA and will not have to be > > edited by DBA nor sysadmin nor tools ever. We will have a separate > > facility (conf.d) for that. As a result, I don't think there's any > > issues left with one-setting-per-file now. > > Let's please NOT call it conf.d if it's living in PGDATA and is not > meant to be edited by hand. conf.d is for a directory of config files > created by users and external utilities, living in CONFIGDIR. How nice that that's not what's being discussed here then. conf.d *IS* the thing thats been proposed to be a separate feature from ALTER SYSTEM. For the use case you describe. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Josh Berkus <josh@agliodbs.com> writes: > Let's please NOT call it conf.d if it's living in PGDATA and is not > meant to be edited by hand. conf.d is for a directory of config files > created by users and external utilities, living in CONFIGDIR. +1 Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 08/01/2013 10:24 AM, Andres Freund wrote: >> Let's please NOT call it conf.d if it's living in PGDATA and is not >> meant to be edited by hand. conf.d is for a directory of config files >> created by users and external utilities, living in CONFIGDIR. > > How nice that that's not what's being discussed here then. conf.d *IS* > the thing thats been proposed to be a separate feature from ALTER > SYSTEM. For the use case you describe. Some of the earlier emails sounded like that's exactly what was proposed. Glad to clarify then. *if* we do file-per-setting, what do you think of the directory name system_set or system_conf then? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
Josh Berkus <josh@agliodbs.com> schrieb: >On 08/01/2013 10:24 AM, Andres Freund wrote: >>> Let's please NOT call it conf.d if it's living in PGDATA and is not >>> meant to be edited by hand. conf.d is for a directory of config >files >>> created by users and external utilities, living in CONFIGDIR. >> >> How nice that that's not what's being discussed here then. conf.d >*IS* >> the thing thats been proposed to be a separate feature from ALTER >> SYSTEM. For the use case you describe. > >Some of the earlier emails sounded like that's exactly what was >proposed. Glad to clarify then. > >*if* we do file-per-setting, what do you think of the directory name >system_set or system_conf then? Postgresql.auto.conf.d is what I'd propose, but the decision about that seems to be one of the smaller problems around thisfeature... That naming seems to make it sensible to extend other files (hba, ident) similarly at a later point. Please excuse brevity and formatting - I am writing this on my mobile phone. Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Andres Freund wrote:
>
> > Postgresql.auto.conf.d is what I'd propose, but the decision about
> > that seems to be one of the smaller problems around this feature...
> > That naming seems to make it sensible to extend other files (hba,
> > ident) similarly at a later point.
>
> I don't like this particular naming proposal, but I'm glad this patch
> finally seems to be getting somewhere.
Yeah, also not a fan. We don't have any 'conf.d' directories in PGDATA
and I don't think we should start now. I liked Josh's suggestions of
something like "system_conf". For multiple independent config files, we
could use directories under that (eg: 'pgconf', 'pghba', 'pgident'..).
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Andres Freund <andres@2ndquadrant.com> writes: > They would need a setting that disables ALTER (DATABASE|USER) ... SET > ... as well though. At least for some settings. > > I don't think enforcing things on that level makes much sense. If only we could trigger some actions when a command is about to be executed, in a way that it's easy for the user to implement whatever policy he fancies… Oh, maybe I should finish preparing those patches for Event Triggers to be fully usable in 9.4 then ;) Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-01 14:37:45 -0400, Stephen Frost wrote: > * Alvaro Herrera (alvherre@2ndquadrant.com) wrote: > > Andres Freund wrote: > > > > > Postgresql.auto.conf.d is what I'd propose, but the decision about > > > that seems to be one of the smaller problems around this feature... > > > That naming seems to make it sensible to extend other files (hba, > > > ident) similarly at a later point. > > > > I don't like this particular naming proposal, but I'm glad this patch > > finally seems to be getting somewhere. > > Yeah, also not a fan. We don't have any 'conf.d' directories in PGDATA > and I don't think we should start now. I liked Josh's suggestions of > something like "system_conf". For multiple independent config files, we > could use directories under that (eg: 'pgconf', 'pghba', 'pgident'..). People know what to expect from .d directories, that's why I suggested it, don't feel really strongly about it. I dislike naming the subdirectories pgconf/... et al though, we should reference the original files name, instead of introducing new abbreviations. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Andres Freund (andres@2ndquadrant.com) wrote:
> People know what to expect from .d directories, that's why I suggested
> it, don't feel really strongly about it. I dislike naming the
> subdirectories pgconf/... et al though, we should reference the original
> files name, instead of introducing new abbreviations.
*People* should not be messing around with files under $PGDATA- that's
half the point. conf.d or other names which look like configuration
directories that an admin should modify are a *bad* idea.
Thanks,
Stephen
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-01 20:33:49 -0400, Stephen Frost wrote: > * Andres Freund (andres@2ndquadrant.com) wrote: > > People know what to expect from .d directories, that's why I suggested > > it, don't feel really strongly about it. I dislike naming the > > subdirectories pgconf/... et al though, we should reference the original > > files name, instead of introducing new abbreviations. > > *People* should not be messing around with files under $PGDATA- that's > half the point. conf.d or other names which look like configuration > directories that an admin should modify are a *bad* idea. I personally consider readers of this list persons... And even people not interested in internals will have to look in there if they set something stupid before. Like setting max_connections higher than the currently configured kernel's max number of semaphores. Or a good number of other settings. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Andres Freund (andres@2ndquadrant.com) wrote:
> I personally consider readers of this list persons... And even people
> not interested in internals will have to look in there if they set
> something stupid before. Like setting max_connections higher than the
> currently configured kernel's max number of semaphores. Or a good number
> of other settings.
And that's actually one of the issues that I have with this overall
approach.. If configurations can be set in 'data' areas which prevent
the system from starting, that's *bad*. Very bad. It means people will
not be able to trust PG to manage the configuration sanely and will have
a lot of distrust and fear of the ALTER SYSTEM capability.
Requiring users to go monkey around inside of a system data directory to
clean things up in order to get PG to come up is a situation we should
do our best to prevent from ever happening.
Thanks,
Stephen
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-01 20:45:38 -0400, Stephen Frost wrote: > * Andres Freund (andres@2ndquadrant.com) wrote: > > I personally consider readers of this list persons... And even people > > not interested in internals will have to look in there if they set > > something stupid before. Like setting max_connections higher than the > > currently configured kernel's max number of semaphores. Or a good number > > of other settings. > > And that's actually one of the issues that I have with this overall > approach.. If configurations can be set in 'data' areas which prevent > the system from starting, that's *bad*. Very bad. It means people will > not be able to trust PG to manage the configuration sanely and will have > a lot of distrust and fear of the ALTER SYSTEM capability. I agree that we need to do reasonable checks, like running GUC validators, but we simply can't control the overall system state. And it's not like this are errors that you couldn't get before. And we should (that's something to improve on) report the relevant guc + file in many cases. > Requiring users to go monkey around inside of a system data directory to > clean things up in order to get PG to come up is a situation we should > do our best to prevent from ever happening. Even trying to do this completely will guarantee that this patch will never, ever, suceed. There simply is no way to reliably detect problems that have complex interactions with the rest of the system. We can improve the detection rate of problems after some real world experience. Don't make this unneccesarily complex. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Andres Freund (andres@2ndquadrant.com) wrote:
> I agree that we need to do reasonable checks, like running GUC
> validators, but we simply can't control the overall system state. And
> it's not like this are errors that you couldn't get before. And we
> should (that's something to improve on) report the relevant guc + file
> in many cases.
You could get the errors before, sure, but when you did, you could read
the log output and go modify the *configuration files* (which things in
$PGDATA are *not*) and fix it and get the system back online. If the
files in $PGDATA have to be modified to get the system online then they
are configuration files and should be in /etc.
> Even trying to do this completely will guarantee that this patch will
> never, ever, suceed. There simply is no way to reliably detect problems
> that have complex interactions with the rest of the system.
The patch will never be able to completely remove the need for external
config files, without changes to PG to deal with these conditions
better.
> We can improve the detection rate of problems after some real world
> experience. Don't make this unneccesarily complex.
Actually, putting it out there as "this can be used to modify anything
and means you can trivially make PG unstartable" is actually the wrong
move to make, imv. Consider that, to deal with the issues caused, we'd
have to *remove* things from being modifyable through this function.
That's a whole lot harder to do from a backward-compatibility view than
adding things later as we improve PG to be able to still come up enough
to be useful even with configuration issues.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
David Johnston
Date:
Andres Freund-3 wrote > Even trying to do this completely will guarantee that this patch will > never, ever, suceed. There simply is no way to reliably detect problems > that have complex interactions with the rest of the system. > > We can improve the detection rate of problems after some real world > experience. Don't make this unneccesarily complex. Instead of prevention some thought to recovery should be considered then. How about some form of persistence mechanism so that, before making these kinds of changes, the admin can "save" the current configuration. Then, in a worse case-scenario, they could run something like "pg_ctl --restore-persisted-configuration ..." to reset everything back the last known good configuration. A single-version save-restore routine for the configuration. When restoring you would want to keep the "current/non-working" configuration and associated logging information - maybe archived somewhere along with the a copy of the last known working version. This would provide some level of audit capability as well as a convenient way for someone to take that archive and send it off to someone more knowledgeable for assistance. Having it auto-run at boot time - possibly to a different archive area than when run manually - would be possible as well; so you'd have both the last good boot configuration as well as whatever point-in-time configurations you wish to save. -- View this message in context: http://postgresql.1045698.n5.nabble.com/Proposal-for-Allow-postgresql-conf-values-to-be-changed-via-SQL-tp5729917p5765968.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* David Johnston (polobo@yahoo.com) wrote:
> How about some form of persistence mechanism so that, before making these
> kinds of changes, the admin can "save" the current configuration. Then, in
> a worse case-scenario, they could run something like "pg_ctl
> --restore-persisted-configuration ..." to reset everything back the last
> known good configuration.
Yeah, I had considered the possibility that we would provide a tool to
manage the config in $PGDATA but I'm not really thrilled with that idea
either.
> A single-version save-restore routine for the configuration. When restoring
> you would want to keep the "current/non-working" configuration and
> associated logging information - maybe archived somewhere along with the a
> copy of the last known working version. This would provide some level of
> audit capability as well as a convenient way for someone to take that
> archive and send it off to someone more knowledgeable for assistance.
> Having it auto-run at boot time - possibly to a different archive area than
> when run manually - would be possible as well; so you'd have both the last
> good boot configuration as well as whatever point-in-time configurations you
> wish to save.
Yeah, there's a lot of work involved in writing a whole system for
managing multiple configurations with history, diffs, etc.. Problems
which, really, our existing text-based config w/ a tool like puppet have
already solved.
Thanks,
Stephen
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
Hi, FWIW, I think you've just put the final nail in the coffin of this patch by raising the barriers unreasonably high. > * Andres Freund (andres@2ndquadrant.com) wrote: > > I agree that we need to do reasonable checks, like running GUC > > validators, but we simply can't control the overall system state. And > > it's not like this are errors that you couldn't get before. And we > > should (that's something to improve on) report the relevant guc + file > > in many cases. > > You could get the errors before, sure, but when you did, you could read > the log output and go modify the *configuration files* (which things in > $PGDATA are *not*) and fix it and get the system back online. If the > files in $PGDATA have to be modified to get the system online then they > are configuration files and should be in /etc. That doesn't seem to be a logical consequence to me. On 2013-08-01 21:06:49 -0400, Stephen Frost wrote: > > Even trying to do this completely will guarantee that this patch will > > never, ever, suceed. There simply is no way to reliably detect problems > > that have complex interactions with the rest of the system. > > The patch will never be able to completely remove the need for external > config files, without changes to PG to deal with these conditions > better. That's not the goal of the patch as far as I understand it. > > We can improve the detection rate of problems after some real world > > experience. Don't make this unneccesarily complex. > > Actually, putting it out there as "this can be used to modify anything > and means you can trivially make PG unstartable" is actually the wrong > move to make, imv. Consider that, to deal with the issues caused, we'd > have to *remove* things from being modifyable through this function. > That's a whole lot harder to do from a backward-compatibility view than > adding things later as we improve PG to be able to still come up enough > to be useful even with configuration issues. I think this chain of argument doesn't have much for it. There are litteraly dozens of ways to break postgres from SQL which we don't even try to defend against. Starting from DELETE FROM pg_class, ending with COPYing files into the datadir. This is a database, not a children's toy, and the feature *is* superuser only. Anyway, I don't see much point arguing this further. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Andres Freund (andres@2ndquadrant.com) wrote:
> FWIW, I think you've just put the final nail in the coffin of this
> patch by raising the barriers unreasonably high.
For my 2c, I don't think it's an unreasonable idea to actually
*consider* what options are available through this mechanism rather than
just presuming that it's a good idea to be able to modify anything,
including things that you wouldn't be able to fix after a restart w/o
hacking around in $PGDATA.
I also don't believe that limiting the set of options which can be
modified through this system is a particularly difficult thing to
implement.
> > * Andres Freund (andres@2ndquadrant.com) wrote:
> On 2013-08-01 21:06:49 -0400, Stephen Frost wrote:
> > > Even trying to do this completely will guarantee that this patch will
> > > never, ever, suceed. There simply is no way to reliably detect problems
> > > that have complex interactions with the rest of the system.
> >
> > The patch will never be able to completely remove the need for external
> > config files, without changes to PG to deal with these conditions
> > better.
>
> That's not the goal of the patch as far as I understand it.
The point above is that we will always need some amount of external
config file and, as such, we should probably consider which items should
really only be set in the *config* files and which can be set in either
place.
> I think this chain of argument doesn't have much for it. There are
> litteraly dozens of ways to break postgres from SQL which we don't even
> try to defend against.
This is a strawman. An admin doing "DELETE FROM pg_class;" or using
COPY to overwrite files in PG's data dir and doing "ALTER SYSTEM SET
shared_buffers = '2GB';", "ALTER SYSTEM SET port = 123;" or even "ALTER
SYSTEM SET data_directory = '/new/path/for/db';" (how would doing that
even make sense..?) are not nearly the same. On the flip side, there's
not nearly as much risk around allowing log_line_prefix and friends to
be set through ALTER SYSTEM SET because it's pretty unlikely that such
a misconfiguration would cause PG to not start.
Thanks,
Stephen
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Peter Geoghegan
Date:
On Thu, Aug 1, 2013 at 8:27 PM, Stephen Frost <sfrost@snowman.net> wrote: > The point above is that we will always need some amount of external > config file and, as such, we should probably consider which items should > really only be set in the *config* files and which can be set in either > place. What settings did you have in mind? Which ones ought to be only be settable in config files in your estimation? -- Peter Geoghegan
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
> On Thu, Aug 1, 2013 at 8:27 PM, Stephen Frost <sfrost@snowman.net> wrote: > > The point above is that we will always need some amount of external > > config file and, as such, we should probably consider which items should > > really only be set in the *config* files and which can be set in either > > place. I think this idea might make some sense: mark the settings (add a flag bit in the guc.c tables) that can be changed via alter setting. Or perhaps add the bit to ones that can *not* be changed. I don't think we need to promise exact compatibility on the set of settings that can be changed via ALTER SYSTEM. If you can change setting XYZ in 9.4, and we find that it wasn't such a good idea and have to disallow it in 9.5, well, too bad. (Or perhaps even a minor version. Am I sacrilegious?) -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Friday, August 02, 2013 4:34 AM Dimitri Fontaine wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > They would need a setting that disables ALTER (DATABASE|USER) ... SET > > ... as well though. At least for some settings. > > > > I don't think enforcing things on that level makes much sense. > > If only we could trigger some actions when a command is about to be > executed, in a way that it's easy for the user to implement whatever > policy he fancies… > > Oh, maybe I should finish preparing those patches for Event Triggers to > be fully usable in 9.4 then ;) I think this can be one useful way to disable. With Regards, Amit Kapila.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Friday, August 02, 2013 8:57 AM Stephen Frost wrote: > * Andres Freund (andres@2ndquadrant.com) wrote: > > FWIW, I think you've just put the final nail in the coffin of this > > patch by raising the barriers unreasonably high. > > For my 2c, I don't think it's an unreasonable idea to actually > *consider* what options are available through this mechanism rather > than just presuming that it's a good idea to be able to modify > anything, including things that you wouldn't be able to fix after a > restart w/o hacking around in $PGDATA. I think if user has set any value wrong such that it doesn't allow server to start, we can provide an option similar to pg_resetxlog to reset the auto file. How about with such an option user might loose some settings? 1. We can think of providing reset for an particular parameter. 2. Suggestions in docs that in case of such a scenario, he can remember values from auto file and reset the settings. As this can happen only in rare scenario's, I think it can make sense to provide such option. With Regards, Amit Kapila.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Cédric Villemain
Date:
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Levendredi 2 août 2013 09:23:17, Amit Kapila a écrit :<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> On Friday, August 02, 20138:57 AM Stephen Frost wrote:<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">> > * Andres Freund (andres@2ndquadrant.com) wrote:<p style="margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>> > FWIW, I think you've just put the final nail in the coffin of this<p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>> > patch by raising the barriers unreasonably high.<p style=" margin-top:0px; margin-bottom:0px;margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > <pstyle=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">>> For my 2c, I don't think it's an unreasonable idea to actually<p style=" margin-top:0px; margin-bottom:0px;margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > *consider*what options are available through this mechanism rather<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > than just presuming thatit's a good idea to be able to modify<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px;-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > anything, including things that you wouldn'tbe able to fix after a<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">> > restart w/o hacking around in $PGDATA.<p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">><p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">> I think if user has set any value wrong such that it doesn'tallow server<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;-qt-user-state:0;">> to start,<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px;-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> we can provide an option similar to pg_resetxlogto reset the auto file.<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; "> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">While guessing what changes aresafe is hard, it is easy to discover the GUCs preventing PostgreSQL from restarting correctly.<p style=" margin-top:0px;margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">pg_ctlmight be able to expose a clear message like : <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">MSG: Params X and Y set by ALTERSYSTEM prevent PostgreSQL from starting<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px;-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">HINT: Issue pg_ctl --ignore-bad-GUC start<p style="-qt-paragraph-type:empty;margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; "> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">Note that the same may also allow postgresql to start with bad GUCvalue in postgresql.conf ....<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">So this is another topic (IMHO).<p style="-qt-paragraph-type:empty;margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; "> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">With the current idea of one-GUC-one-file in a PGDATA subdirectoryit is *easy* to fix for any DBA or admin. However, note that I do prefer a simple 'include_auto_conf=here|start|end|off'in postgresql.conf (by default at end of file, with value 'end' set). <p style="-qt-paragraph-type:empty;margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; "> <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0;text-indent:0px; -qt-user-state:0;">-- <p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Cédric Villemain +33 (0)6 20 3022 52<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;-qt-user-state:0;">http://2ndQuadrant.fr/<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px;-qt-block-indent:0; text-indent:0px; -qt-user-state:0;">PostgreSQL: Support 24x7 - Développement, Expertiseet Formation<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px;-qt-block-indent:0; text-indent:0px; ">
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Thursday, August 01, 2013 8:37 PM Andres Freund wrote: > Hi, > > On 2013-08-01 15:40:22 +0100, Greg Stark wrote: > > Why isn't it enough to just dump out all variables with a source of > alter > > system to a text file? You can either have a single global lock > around that > > operation or write it to a new file and move it into place. > > It saves you from several complications: > * No need to iterate over all GUCs, figuring out which come from which > source, when writing out the file. Not all GUC's, only which are in auto file. > * Less logic required when writing out a value, since you have an > acceptable input ready. > * No need to make sure the autogenerated file is written out in the > same > order when adding/changing a setting (to make sure you can > diff/version control it sensibly) > * No locking necessary, without locking concurrent changes can vanish. > * Way easier to delete a setting if it ends up stopping postgres from > starting. The logic needed in current patch for above points is not all that complex that it needs to be thought of redesign until the basic idea of one-file-per-setting scores high over one-file-all-setting. Below are some points in my mind due to which I have supported/implemented one-file-all-setting approach: a. I had heard quite a few times that Postgres has lot of files (each relation has separate file) as compare to Oracle. Users feel that Oracle's table space approach is better. b. While server start/Sighup, we needs to open/read/close each file separately which in-itself seems to be overhead. I believe what Greg Stark has said in his below mail link is the more appropriate way and the current patch has done that way. http://www.postgresql.org/message-id/CAM-w4HP7=a2VowyJUD0CAZL5b8FsaHymdQeouL udSOhdnCw_zg@mail.gmail.com Also when other commercial database (Oracle) can do it with single file, users will try to compare with it. I understand that our views are not matching on this point and I totally respect your views, but not able to convince myself for one-file-per-setting approach. Can you please once again think and see if there is a viable way for moving forward. I had modified the patch for many suggestions which had made it simpler and if you have any idea to make one-file-all-settings implementation better, then I can address it. With Regards, Amit Kapila.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Friday, August 02, 2013 4:17 PM Cédric Villemain wrote: Le vendredi 2 août 2013 09:23:17, Amit Kapila a écrit : > On Friday, August 02, 2013 8:57 AM Stephen Frost wrote: > > * Andres Freund (andres@2ndquadrant.com) wrote: >> > > FWIW, I think you've just put the final nail in the coffin of this >> > > patch by raising the barriers unreasonably high. >> > >> > For my 2c, I don't think it's an unreasonable idea to actually >> > *consider* what options are available through this mechanism rather >> > than just presuming that it's a good idea to be able to modify >> > anything, including things that you wouldn't be able to fix after a >> > restart w/o hacking around in $PGDATA. > >> I think if user has set any value wrong such that it doesn't allow server >> to start, >> we can provide an option similar to pg_resetxlog to reset the auto file. > While guessing what changes are safe is hard, it is easy to discover the GUCs preventing PostgreSQL from restarting correctly. > pg_ctl might be able to expose a clear message like : > MSG: Params X and Y set by ALTER SYSTEM prevent PostgreSQL from starting > HINT: Issue pg_ctl --ignore-bad-GUC start > Note that the same may also allow postgresql to start with bad GUC value in postgresql.conf .... > So this is another topic (IMHO). Yes, this can be viable option to ignore variable values that don't allow server to start, also I agree with you that this can be a separate patch. > With the current idea of one-GUC-one-file in a PGDATA subdirectory it is *easy* to fix for any DBA or admin. > However, note that I do prefer a simple 'include_auto_conf=here|start|end|off' in postgresql.conf (by default at end of file, with value 'end' set). Earlier patch has this implementation, but later based on suggestions, I made it default. With Regards, Amit Kapila.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila@huawei.com) wrote: > Below are some points in my mind due to which I have supported/implemented > one-file-all-setting approach: > a. I had heard quite a few times that Postgres has lot of files (each > relation has separate file) as compare to Oracle. > Users feel that Oracle's table space approach is better. This is completely unrelated to this discussion, imv. > b. While server start/Sighup, we needs to open/read/close each file > separately which in-itself seems to be overhead. I also don't think performance of this particular operation should be a high priority. > I believe what Greg Stark has said in his below mail link is the more > appropriate way and the current patch has done that way. > http://www.postgresql.org/message-id/CAM-w4HP7=a2VowyJUD0CAZL5b8FsaHymdQeouL > udSOhdnCw_zg@mail.gmail.com He doesn't actually provide any reasoning for it. That said, I've not heard any particularly good reason for having a setting per file either. This is an internal-to-PG data file and we should really implement it in whichever way makes the most sense for us. My general feeling is that one file is simpler and sufficient for the postgresql.conf-like parameters, but I wonder what we're going to do for pg_hba/pg_ident. Those would be good to have multiple files for because (as we saw with pg_authid) they could get quite large because they can have per-user entries in them and rewriting a large file for every change would be quite painful. > Also when other commercial database (Oracle) can do it with single file, > users will try to compare with it. To make it clear- this isn't justification for this design. Also, the notion that Oracle's *configuration* is all done with a *single-file* is.. laughable. Thanks, Stephen
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila@huawei.com) wrote:
> Yes, this can be viable option to ignore variable values that don't allow
> server to start, also I agree with you that
> this can be a separate patch.
I disagree that this can be a separate patch. Adding an option to not
allow certain GUCs from being modified through this mechanism should be
trivial. Then, *after* we have such utility that will allow users to
fix a bad situation, we can consider relaxing those restrictions to
allow more values to be set. I still contend that there will be some
GUCs that simply don't make any sense to have in the auto-conf area of
PGDATA.
Thanks,
Stephen
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Friday, August 02, 2013 5:19 PM Stephen Frost wrote: > * Amit Kapila (amit.kapila@huawei.com) wrote: > > Below are some points in my mind due to which I have > > supported/implemented one-file-all-setting approach: > > a. I had heard quite a few times that Postgres has lot of files (each > > relation has separate file) as compare to Oracle. > > Users feel that Oracle's table space approach is better. > > This is completely unrelated to this discussion, imv. The point I wanted to convey is that having more files for databasein general is not a great idea. > > b. While server start/Sighup, we needs to open/read/close each file > > separately which in-itself seems to be overhead. > > I also don't think performance of this particular operation should be a > high priority. If it makes startup taking more time, then isn't it a performance critical path? > > I believe what Greg Stark has said in his below mail link is the more > > appropriate way and the current patch has done that way. > > http://www.postgresql.org/message-id/CAM- > w4HP7=a2VowyJUD0CAZL5b8FsaHym > > dQeouL > > udSOhdnCw_zg@mail.gmail.com > > He doesn't actually provide any reasoning for it. That said, I've not > heard any particularly good reason for having a setting per file > either. > This is an internal-to-PG data file and we should really implement it > in whichever way makes the most sense for us. My general feeling is > that one file is simpler and sufficient for the postgresql.conf-like > parameters, Sure, I also feel the same that if it can be addressed with single file, then lets do that way only. > but I wonder what we're going to do for pg_hba/pg_ident. > Those would be good to have multiple files for because (as we saw with > pg_authid) they could get quite large because they can have per-user > entries in them and rewriting a large file for every change would be > quite painful. > > > Also when other commercial database (Oracle) can do it with single > > file, users will try to compare with it. > > To make it clear- this isn't justification for this design. > Also, the > notion that Oracle's *configuration* is all done with a *single-file* > is.. laughable. Not all Oracle's configuration, but Change of configuration parameters. IIRC, before starting this feature I had checked Oracle's specs and it seems to be not doing with multiple files for Alter System. If you have doubt, I can once again Verify it. With Regards, Amit Kapila.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila@huawei.com) wrote:
> > This is an internal-to-PG data file and we should really implement it
> > in whichever way makes the most sense for us. My general feeling is
> > that one file is simpler and sufficient for the postgresql.conf-like
> > parameters,
>
> Sure, I also feel the same that if it can be addressed with single file,
> then lets do that way only.
We need to settle on one choice and then implement it, yes. It
certainly doesn't make any sense to have two different ways to deal with
an internal-to-PG data structure. Of course, that might argue for
making this file actually *be* like pg_authid is today; has that been
considered? I'm guessing it's not practical because the point where we
need to read the config is before certain things have been set up to
allow reading from heap files, but it seems like something which should
at least be considered. If we can make it work, then that may also
solve the pg_hba/pg_ident issue, which is about a bazillion times more
interesting than the mostly set-and-forget postgresql.conf settings.
Perhaps having the file be a heap file instead of anything a sysadmin
can be asked to go hack would also make it more clear that this is an
internal PG file which is to be managed only through PG and stop all
this arguing about how "oh, they can just fix it by twiddling things in
$PGDATA" is considered by some to be an acceptable solution. Heck, it'd
also keep the number of files down while allowing more fine-grained
modifications and writes (iow, we wouldn't have to rewrite the whole
file every time..).
Thanks,
Stephen
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Stephen Frost <sfrost@snowman.net> writes:
> Perhaps having the file be a heap file instead of anything a sysadmin
> can be asked to go hack would also make it more clear that this is an
> internal PG file which is to be managed only through PG and stop all
> this arguing about how "oh, they can just fix it by twiddling things in
> $PGDATA" is considered by some to be an acceptable solution.
Whether you think it's acceptable or not, sometimes it's *necessary*.
What if you set a combination of parameters that prevents Postgres from
starting?
Even if we could make config-in-a-catalog work, which I'm very dubious
of, I would think it far too unsafe to use.
We could of course fix that problem by only storing "safe" parameters
in a catalog, but then you lose the supposed appeal of a single-file
solution.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Smith
Date:
On 7/29/13 3:46 PM, Josh Berkus wrote: > Based on the ongoing discussion of this patch, I have moved it to 9.4CF2 > (9-2013). > > Mind you, it would be good to commit some version of it before September. Quite a bit of the patch adds some refactored GUC parameter validation code that seems necessary for this feature, and that's now where the controversial parts are either. That's the part I thought should be looked at for commit now. I'd like to get the change size Amit is carrying around and (and other people are reviewing) reduced here. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-02 08:41:09 -0400, Stephen Frost wrote: > Perhaps having the file be a heap file instead of anything a sysadmin > can be asked to go hack would also make it more clear that this is an > internal PG file which is to be managed only through PG and stop all > this arguing about how "oh, they can just fix it by twiddling things in > $PGDATA" is considered by some to be an acceptable solution. Heck, it'd > also keep the number of files down while allowing more fine-grained > modifications and writes (iow, we wouldn't have to rewrite the whole > file every time..). Crash recovery & PGC_SIGHUP|PGC_POSTMASTER… Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Stark
Date:
Writing out each guc in a separate file is a singularly bad idea. It's going out of our way to confuse users about what's going on and how they're expected to interact with the settings files and it actively makes it harder or nearly impossible to protect against simple failures. 1) The whole reason for conf.d directories for things like Apache or cron or whatever is so that other software can drop in snippets without having to parse and edit a file in place. We *do not* want users doing that inside PGDATA. I'm not even clear we do want this in /etc since none of our GUC options are repeatable things like Apache virtual servers. It actually makes *more* sense for pg_hba than it does for gucs. I think we can assume that in the future we'll have something like it however. 2) Directories are notoriously hard to version control, most version control systems either don't do it at all or do a weak form of version control on directories. Even if users tried to track the changes in these files they'll likely find it difficult to tell when two settings were changed together or in separate changes. On the other hand if all the settings are in a single file then even the simplest form of version control -- backup files -- would suffice. If we just keep a backup copy of the settings file for each change then it would be easy for people to diff from one version to another and see all the changes together and easy for them to restore an old copy if the current one isn't starting up. If they're in a million tiny files then users would have to keep a backup copy of the whole directory and dig thorugh a recursive diff of the whole directory. Or they would have tons of backup files for different settings at different times and need to figure out which ones were in effect at a given time.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Smith
Date:
On 8/1/13 10:47 AM, David Johnston wrote: > Minor request: could someone enlighten me as to why making the directory > location a compile-time option is undesirable. The ongoing argument here is whether to allow moving the directory at all, or to make it fixed to $PGDATA the way recovery.conf is. If you accept that it should float, then it actually needs to be a start time option. Software like Debian moves around the postgresql.conf like this: pg_ctl -c config_file=/etc/postgresql/9.3/main/postgresql.conf ... The way this argument is going the last few days, I'm starting to think that it's worth breaking this style of config directory setup out into its own feature now. Whether or not ALTER SYSTEM SET takes advantage of the config directory or not seems a still raging question. I've been coupling the two together because I think the design of ALTER SYSTEM SET should consider a config directory based approach. But from the perspective of what can get committed first, the config directory really should go first. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> What if you set a combination of parameters that prevents Postgres from
> starting?
This was what I was trying to get at up-thread. Things that prevent PG
from being able to start (or, really, which cause PG to be started in a
completely different mode, ala recovery.conf) need to be able to be
modified outside of PG and therefore should, imv, be considered
configuration parameters and therefore live outside of $PGDATA (when
installed from a distro, blah, blah).
> We could of course fix that problem by only storing "safe" parameters
> in a catalog, but then you lose the supposed appeal of a single-file
> solution.
I don't see "having them all in one file is more convenient for the
admin" as being a justification for the single-file approach, simply
because I don't consider the 'auto' file to be something that the admin
would be modifying.
Curiously, I've not heard any argument about what parameters are "safe"
and what aren't, though I was asked which ones I thought were safe and
which weren't. Perhaps looking at the specific options that would
likely cause PG to not start would be useful to this discussion.
Off-hand, I see:
data_directory- Clearly we need to know this before starting, so it has to be defined somewhere and then passed to PG
insome way. Currently this is done in Ubuntu by having the init script read the directory out of the postgresql.conf,
butit could also be passed through Ubuntu's "pg_ctl.conf", which takes a set of parameters to pass. I will note here,
thoughit may apply in other places also, that this part of the configuration is necessairly the distro's
responsibility.
hba_file- Won't start without one.
ident_file- We *will* start without one. We'll even start if it's got garbage in it. I have to admit, I was a bit
surprisedthat this behaves so differently from hba_file wrt dealing with existance / errors.
listen_addresses- Won't start if it's invalid, but if it's not there we'll just try to listen on localhost:5432, but if
thatfails we won't start.
port- Similar to listen_addresses
unix_socket_directories, unix_socket_group- Won't start if it's invalid, will start w/ a default if not set.
ssl_cert_file, ssl_key_file, ssl_ca_file, ssl_crl_file,
krb_server_keyfile, shared_preload_libraries- Won't start if it's invalid, not used if not set.
shared_buffers- Older versions won't start if it's set above the SHM limits on the system. Even in 9.3+ though, if set
toohigh, will either cause it to not start or to possible crash very quickly (eg: user set it to however much real
memorythey have in the system).
log_directory, log_filename- Won't start if set such that PG can't create the necessary directories or open its log
file.
log_timezone, timezone, lc_messages, lc_monetary, etc- Won't start if set invalid- can we check validity of this when
setthrough ALTER SYSTEM?
local_preload_libraries- Will start if it's set to something invalid, but then you can't connect because new backend
startswill fail.
Now, there's certainly a whole slew of things which *can* be modified
w/o too much risk to being able to get PG started again. Also, many of
the things above could probably be changed to deal with error cases and
keep starting up (eg: ssl_*). Andrew pointed out that we can use
command-line arguments to override badly configured parameters, but I'm
worried that'd simply end up making those items be configured through
the distro scripts, and probably in a way that isn't as nice as
postgresql.conf, eg:
Ubuntu's pg_ctl.conf might become:
pg_ctl_options = '-o "-c listen_addresses=127.0.01.1" -o "-c port=5433"'
etc, or, more likely, a *new* config file being created to manage these
things which is completely distribution-specific...
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
Greg,
* Greg Stark (stark@mit.edu) wrote:
> Writing out each guc in a separate file is a singularly bad idea. It's
> going out of our way to confuse users about what's going on and how
> they're expected to interact with the settings files and it actively
> makes it harder or nearly impossible to protect against simple
> failures.
I agree that having a separate file for each GUC is a bad idea. That
said, I still contend that there's a difference between files in $PGDATA
and files found in /etc.
> 1) The whole reason for conf.d directories for things like Apache or
> cron or whatever is so that other software can drop in snippets
> without having to parse and edit a file in place. We *do not* want
> users doing that inside PGDATA.
Agreed. That said, if users *want* a separate file per GUC in their
/etc, we can let them do that with the conf.d structure.
> I'm not even clear we do want this in /etc since none of our GUC
> options are repeatable things like Apache virtual servers. It actually
> makes *more* sense for pg_hba than it does for gucs. I think we can
> assume that in the future we'll have something like it however.
I tend to agree with this also, though I can imagine wanting to separate
things in a conf.d directory ala exim's conf.d directories, to allow
tools like puppet to manage certain things environment-wide (perhaps
krb_server_keyfile) while other configuration options are managed
locally.
> If we just keep a backup copy of the settings file for each change
> then it would be easy for people to diff from one version to another
> and see all the changes together and easy for them to restore an old
> copy if the current one isn't starting up. If they're in a million
> tiny files then users would have to keep a backup copy of the whole
> directory and dig thorugh a recursive diff of the whole directory. Or
> they would have tons of backup files for different settings at
> different times and need to figure out which ones were in effect at a
> given time.
This I don't entirely follow though. Above, you don't want users
monkeying around in PGDATA dropping files, but it's ok for them to be
diffing them and, presumably, rewritting them with older version when
they feel the need to?
I agree that we should provide a way for users to get old versions of
their config and know when things changed, but, to be honest, I'd like
to see that kind of audit log information for various catalog tables
too- eg: pg_database. If we had event triggers and they could be fired
for ALTER SYSTEM along with ALTER DATABASE, then we could have an audit
system with an audit table which takes who did what when, old value, new
value, etc..
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Dimitri Fontaine escribió: > Andres Freund <andres@2ndquadrant.com> writes: > > They would need a setting that disables ALTER (DATABASE|USER) ... SET > > ... as well though. At least for some settings. > > > > I don't think enforcing things on that level makes much sense. > > If only we could trigger some actions when a command is about to be > executed, in a way that it's easy for the user to implement whatever > policy he fancies… > > Oh, maybe I should finish preparing those patches for Event Triggers to > be fully usable in 9.4 then ;) I remind you that event triggers are not fired for global objects such as databases and roles. Do you intend to lift that restriction? -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Stephen Frost escribió: > * Tom Lane (tgl@sss.pgh.pa.us) wrote: > > What if you set a combination of parameters that prevents Postgres from > > starting? > > This was what I was trying to get at up-thread. Things that prevent PG > from being able to start (or, really, which cause PG to be started in a > completely different mode, ala recovery.conf) need to be able to be > modified outside of PG and therefore should, imv, be considered > configuration parameters and therefore live outside of $PGDATA (when > installed from a distro, blah, blah). I think the way out of this situation is to have a postmaster and/or pg_ctl switch that disables reading of ALTER SYSTEM settings. Then the DBA can do ALTER SYSTEM RESET until it works again. No need to mess with "data" files. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 08/02/2013 07:54 AM, Stephen Frost wrote: > Curiously, I've not heard any argument about what parameters are "safe" > and what aren't, though I was asked which ones I thought were safe and > which weren't. Perhaps looking at the specific options that would > likely cause PG to not start would be useful to this discussion. I really think this is the wrong approach. If we start removing "unsafe" parameters from ALTER SYSTEM SET, we basically hobble the feature to the point of uselessness. Out of the 15 or so parameters 80% of our users touch, half of them are on your "unsafe" list. A much simpler solution to the issue Stephen proposes is to have a way to start up the server with all settings from ALTER SYSTEM SET disabled, just like some software allows you to start it up in "safe mode". Of course, automatically disabling the *individual* parameters would be even better from a usability perspective. This would be equally useful for a manually-written postgresql.conf, i.e.: "PostgreSQL is unable to start because we couldn't allocate all of the memory you asked for. Starting up with shared_buffers set to 32MB.". ... However, I'm not confident that we'll always be able to do that. We'd also need to have a way to "kick and scream" so that sysadmins would actually SEE it when the system disables a parameter. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Josh Berkus (josh@agliodbs.com) wrote:
> On 08/02/2013 07:54 AM, Stephen Frost wrote:
> > Curiously, I've not heard any argument about what parameters are "safe"
> > and what aren't, though I was asked which ones I thought were safe and
> > which weren't. Perhaps looking at the specific options that would
> > likely cause PG to not start would be useful to this discussion.
>
> I really think this is the wrong approach. If we start removing
> "unsafe" parameters from ALTER SYSTEM SET, we basically hobble the
> feature to the point of uselessness. Out of the 15 or so parameters 80%
> of our users touch, half of them are on your "unsafe" list.
They're on the 'unsafe' list because they're likely to *break* things.
I'm not at *all* surprised that the list comprises 80% of the parameters
that people actually modify/use, but that doesn't mean it's smart to let
them hack at those parameters remotely w/ no access to the host system
to fix things if they screw it up, which is really the case that I'm
thinking about here because it's what we're enabling through this
mechanism. If the users already had access to the host system to go
modify the parameters in the config on the filesystem, they'd be likely
to just go *do* that, no?
> A much simpler solution to the issue Stephen proposes is to have a way
> to start up the server with all settings from ALTER SYSTEM SET disabled,
> just like some software allows you to start it up in "safe mode".
See above for why I'm not thrilled wih this approach, unless it was set
up to happen automatically, but you couldn't simply ignore *all* the
ALTER SYSTEM SET parameters because then you might not be able to
connect in due to some ALTER SYSTEM SET parameter being necessary for
remote connectivity or authentication.
> Of course, automatically disabling the *individual* parameters would be
> even better from a usability perspective. This would be equally useful
> for a manually-written postgresql.conf, i.e.:
>
> "PostgreSQL is unable to start because we couldn't allocate all of the
> memory you asked for. Starting up with shared_buffers set to 32MB.".
Right; this is part of what I was getting at with the list- there are
definitely items on that list that we could start up without, were they
misconfigured. The one question remaining from *that* however is if
there would be such a security impact from the config change that it
wouldn't be *safe* for us to do so (consider if we used our 'default'
pg_hba.conf, with 'trust' all over the place, in the event that we
couldn't load the system pg_hba.conf..).
> ... However, I'm not confident that we'll always be able to do that.
> We'd also need to have a way to "kick and scream" so that sysadmins
> would actually SEE it when the system disables a parameter.
Yes, this is also true. When it comes to a 'safe mode', my original
inclination was to only allow connections over a unix socket from a user
running as the same Unix UID as the database is running as... That's
not a solution for the Windows folks though, I'm afraid. :/
Thanks,
Stephen
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Stephen Frost <sfrost@snowman.net> writes:
> * Josh Berkus (josh@agliodbs.com) wrote:
>> A much simpler solution to the issue Stephen proposes is to have a way
>> to start up the server with all settings from ALTER SYSTEM SET disabled,
>> just like some software allows you to start it up in "safe mode".
> See above for why I'm not thrilled wih this approach, unless it was set
> up to happen automatically, but you couldn't simply ignore *all* the
> ALTER SYSTEM SET parameters because then you might not be able to
> connect in due to some ALTER SYSTEM SET parameter being necessary for
> remote connectivity or authentication.
Yeah, this approach is a nonstarter because there's no reason to assume
that a postmaster started with default parameters will start successfully,
or will be connectable-to if it does start. Maybe there's another
postmaster hogging the default port, for instance.
regards, tom lane
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Josh Berkus (josh@agliodbs.com) wrote:
> I really think this is the wrong approach. If we start removing
> "unsafe" parameters from ALTER SYSTEM SET, we basically hobble the
> feature to the point of uselessness. Out of the 15 or so parameters 80%
> of our users touch, half of them are on your "unsafe" list.
Reflecting on this a bit more, I'm curious what your list-of-15 is.
Many of the items on my list were file locations or other things which
generally require coordination with other groups (like the unix admins
or network admins) to change, eg: listen_addresses, port, SSL or
Kerberos file locations, etc.
There's quite a few parameters which I've changed that are "safe" and
internal-to-PG which weren't on my list- work_mem, maint_work_mem,
vacuum / autovacuum settings, effective_io_concurrency, wal_level,
sync_commit, checkpoint settings, max_wal_senders, planner costs,
logging settings (though this might have to be coordinated w/ the unix
admins unless splunk or similar is being used), etc.
Thanks,
Stephen
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Smith
Date:
On 8/2/13 4:48 PM, Stephen Frost wrote: > * Josh Berkus (josh@agliodbs.com) wrote: >> I really think this is the wrong approach. If we start removing >> "unsafe" parameters from ALTER SYSTEM SET, we basically hobble the >> feature to the point of uselessness. Out of the 15 or so parameters 80% >> of our users touch, half of them are on your "unsafe" list. > > Reflecting on this a bit more, I'm curious what your list-of-15 is. You can get a top 15-ish list from https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server The list of things I see changed regularly that are on your unsafe list are: listen_addresses, port, shared_buffers, log_directory, log_filename Obvious missing thing from your unsafe list that is also changed regularly is max_connection. I count 6 parameters that are both unsafe and changed regularly. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Greg Stark <stark@mit.edu> writes: > Writing out each guc in a separate file is a singularly bad idea. It's I'm not buying into any of your arguments here, and have something to add to that part: > I'm not even clear we do want this in /etc since none of our GUC > options are repeatable things like Apache virtual servers. It actually > makes *more* sense for pg_hba than it does for gucs. I think we can > assume that in the future we'll have something like it however. Given a conf.d option in /etc it's then quite easy to add per-extension configuration files in the packaging system, so that users don't have to edit postgresql.conf for default values. We still need some kind of repeatable settings that we don't have yet for that to happen: cumulative setting of a "list" GUC such as local_preload_libraries and suchlike. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Alvaro Herrera <alvherre@2ndquadrant.com> writes: > I remind you that event triggers are not fired for global objects > such as databases and roles. Do you intend to lift that restriction? That's not on my TODO list for 9.4. My understanding about implementing that is: - we agree that it would be nice to have, - it requires a separate *shared* catalog for event triggers. What I'm yet unsure about is that there's a consensus that the use cases are worthy of a new shared catalog in the system. Also I didn't look how hard it is to actually provide for it. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Saturday, August 03, 2013 12:53 AM Tom Lane wrote: > Stephen Frost <sfrost@snowman.net> writes: > > * Josh Berkus (josh@agliodbs.com) wrote: > >> A much simpler solution to the issue Stephen proposes is to have a > way > >> to start up the server with all settings from ALTER SYSTEM SET > disabled, > >> just like some software allows you to start it up in "safe mode". > > > See above for why I'm not thrilled wih this approach, unless it was > set > > up to happen automatically, but you couldn't simply ignore *all* the > > ALTER SYSTEM SET parameters because then you might not be able to > > connect in due to some ALTER SYSTEM SET parameter being necessary for > > remote connectivity or authentication. > > Yeah, this approach is a nonstarter because there's no reason to assume > that a postmaster started with default parameters will start > successfully, > or will be connectable-to if it does start. Maybe there's another > postmaster hogging the default port, for instance. Okay, but user will always have option to start server with different value of parameter (pg_ctl -o "-p 5434"). Now as a summarization we have below ways to move forward: 1. Provide a way for user to start server if not able to start due to in-appropriate value of unsafe parameter a. already user has an option that he can mention value of any particular parameter with which sever can start b. keep one backup copy of parameters, so that user can option to start with that copy, else if that also doesn't work he can use point 'a'. 2. Don't allow unsafe parameters to be modified by ALTER SYSTEM a. List of un-safe parameters b. mechanism so that ALTERSYSTEM throws error for non-modifiable parameters c. user can view non-modifiable parameters (may be in pg_settings) d. some way such that if user wants to takerisk of server not getting started, he should allow to modify such parameters. may be server is started with some specific option. This can reduce the fear Josh had regarding this command to be not of much use. I think if we choose Option-2, then one of the initial difficulty will be to get an agreement on list of un-safe parameters. I believe even if we want to go with Option-2, then in first cut the work should be minimized. With Regards, Amit Kapila.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Amit Kapila <amit.kapila@huawei.com> writes:
> On Saturday, August 03, 2013 12:53 AM Tom Lane wrote:
>> Yeah, this approach is a nonstarter because there's no reason to assume
>> that a postmaster started with default parameters will start
>> successfully,
>> or will be connectable-to if it does start. Maybe there's another
>> postmaster hogging the default port, for instance.
> Okay, but user will always have option to start server with different value
> of parameter (pg_ctl -o "-p 5434").
You're assuming that the user starts the postmaster manually. On most
modern installations there's a few layers of scripting in there, which
might not be that easy to hack to add some command-line parameters,
even assuming that the DBA has sufficient wits about him to think of
this solution. (When your postmaster unexpectedly fails to restart
at four in the morning, having to think of such an approach isn't what
you want to be doing.)
My point here is just that we should keep the parameter values in plain
text files, so that one possible solution is reverting a bogus change with
vi/emacs/your-weapon-of-choice. If we "improve" matters so that the only
possible way to fix the parameter setting is via a running postmaster,
we've narrowed the number of escape routes that a frantic DBA will have.
And what would we have bought for that? Not much.
regards, tom lane
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Monday, August 05, 2013 11:57 AM Tom Lane wrote: > Amit Kapila <amit.kapila@huawei.com> writes: > > On Saturday, August 03, 2013 12:53 AM Tom Lane wrote: > >> Yeah, this approach is a nonstarter because there's no reason to > assume > >> that a postmaster started with default parameters will start > >> successfully, > >> or will be connectable-to if it does start. Maybe there's another > >> postmaster hogging the default port, for instance. > > > Okay, but user will always have option to start server with different > value > > of parameter (pg_ctl -o "-p 5434"). > > You're assuming that the user starts the postmaster manually. On most > modern installations there's a few layers of scripting in there, which > might not be that easy to hack to add some command-line parameters, > even assuming that the DBA has sufficient wits about him to think of > this solution. (When your postmaster unexpectedly fails to restart > at four in the morning, having to think of such an approach isn't what > you want to be doing.) > > My point here is just that we should keep the parameter values in plain > text files, Here by text files, do you mean to say you are expecting file-per-guc-setting? > so that one possible solution is reverting a bogus change > with > vi/emacs/your-weapon-of-choice. If we "improve" matters so that the > only > possible way to fix the parameter setting is via a running postmaster, > we've narrowed the number of escape routes that a frantic DBA will > have. > And what would we have bought for that? Not much. Although it is not advisable to edit this file manually, but I think in such situations (postmaster doesn't start up due to inappropriate parameter value) it can help user to come out of situation much easily. My only point was to address the concerns regarding un-safe parameter values. With Regards, Amit Kapila.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Amit Kapila <amit.kapila@huawei.com> writes:
> On Monday, August 05, 2013 11:57 AM Tom Lane wrote:
>> My point here is just that we should keep the parameter values in plain
>> text files,
> Here by text files, do you mean to say you are expecting
> file-per-guc-setting?
No, I don't think this argument really says much one way or the other
about one-file vs one-file-per-GUC. I'm just lobbying against the
"put them in a catalog" idea.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Stark
Date:
On Fri, Aug 2, 2013 at 4:05 PM, Stephen Frost <sfrost@snowman.net> wrote: >> I'm not even clear we do want this in /etc since none of our GUC >> options are repeatable things like Apache virtual servers. It actually >> makes *more* sense for pg_hba than it does for gucs. I think we can >> assume that in the future we'll have something like it however. > > I tend to agree with this also, though I can imagine wanting to separate > things in a conf.d directory ala exim's conf.d directories, to allow > tools like puppet to manage certain things environment-wide (perhaps > krb_server_keyfile) while other configuration options are managed > locally. Extensions are actually a pretty good argument for why conf.d in /etc (or wherever the non-auto-config is) is pretty important useful. That's the kind of thing conf.d directories are meant for. A user can install a package containing an extension and the extension would automatically drop in the config entries needed in that directory. -- greg
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Amit Kapila <amit.kapila@huawei.com> writes:
> > On Monday, August 05, 2013 11:57 AM Tom Lane wrote:
> >> My point here is just that we should keep the parameter values in plain
> >> text files,
>
> > Here by text files, do you mean to say you are expecting
> > file-per-guc-setting?
>
> No, I don't think this argument really says much one way or the other
> about one-file vs one-file-per-GUC. I'm just lobbying against the
> "put them in a catalog" idea.
imv there are things which should be in a catalog and things which
shouldn't. I'm not sure that "can the postmaster start up if its wrong"
is necessairly the right test to use but the results tend to line up
with my expectations. postgresql.conf seems to have a number of items
which don't even make a lot of sense to set server-side: enable_* being
a good example. I know it *works* to set those in postgresql.conf, but
I dislike using postgresql.conf as documentation for what options exist.
I keep thinking that postgresql.conf should really be those items that
define how PG interacts with the OS (which tends to also line up with
"things the postmaster needs correct to start") and the rest of the
options should go somewhere else, perhaps as things which can be
configured through this ALTER SYSTEM option, which starts to sound more
like "default GUC values" for those items which take effect on
individual backend startup.
I've some more thoughts on this and will try to formulate them later
today.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Greg Stark (stark@mit.edu) wrote:
> On Fri, Aug 2, 2013 at 4:05 PM, Stephen Frost <sfrost@snowman.net> wrote:
> >> I'm not even clear we do want this in /etc since none of our GUC
> >> options are repeatable things like Apache virtual servers. It actually
> >> makes *more* sense for pg_hba than it does for gucs. I think we can
> >> assume that in the future we'll have something like it however.
> >
> > I tend to agree with this also, though I can imagine wanting to separate
> > things in a conf.d directory ala exim's conf.d directories, to allow
> > tools like puppet to manage certain things environment-wide (perhaps
> > krb_server_keyfile) while other configuration options are managed
> > locally.
>
> Extensions are actually a pretty good argument for why conf.d in /etc
> (or wherever the non-auto-config is) is pretty important useful.
> That's the kind of thing conf.d directories are meant for. A user can
> install a package containing an extension and the extension would
> automatically drop in the config entries needed in that directory.
Agreed, though I think there should be a difference between "shared
library load" being added-to for extensions, and "random
extension-specific GUC"..
Thanks,
Stephen
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Thu, Aug 1, 2013 at 03:40:22PM +0100, Greg Stark wrote: > Why isn't it enough to just dump out all variables with a source of alter > system to a text file? You can either have a single global lock around that > operation or write it to a new file and move it into place. > > -- > greg > > On 1 Aug 2013 15:19, "Andres Freund" <andres@2ndquadrant.com> wrote: > > On 2013-08-01 15:17:04 +0100, Greg Stark wrote: > > We don't need per guc locking. This is the whole objection Tom had about > > this patch being more complex than it has to be. > > IIRC he objected to using locking *at all* because a simple > one-file-per-setting approach should be used. I am unclear why we don't need a lock around _each_ GUC, i.e. if two sessions try to modify the same GUC at the same time. And if we need a lock, seems we can have just one and write all the settings to one file --- it is not like we have trouble doing locking, though this is cluster-wide locking. How would users handle renamed GUC variables, as we have done in the past? Would pg_dumpall dump the settings out? Would unrecognized settings throw an error, causing pg_upgrade to fail? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
All, To limit the argument here, let's please not argue about things which people already agree on. So: We seem to have consensus around: A) the inadvisability of storing GUCs in a system catalog. B) the utility of a conf.d in /etc/ which may have nothing to do with ALTER SYSTEM SET C) that any file for ALTER SYSTEM SET go in $PGDATA somewhere. What we are still arguing about: D) one-big-file vs. file-per-setting E) whether "unsafe" settings or "restart" settings should be allowed in ALTER SYSTEM SET. F) whether admins need the ability to disable ALTER SYSTEM SET. Since each of D, E and F issues are completely orthagonal to each other, I suggest that maybe we argue them each out on their own threads? I'll start. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-05 12:18:25 -0400, Bruce Momjian wrote: > On Thu, Aug 1, 2013 at 03:40:22PM +0100, Greg Stark wrote: > > Why isn't it enough to just dump out all variables with a source of alter > > system to a text file? You can either have a single global lock around that > > operation or write it to a new file and move it into place. > > > > -- > > greg > > > > On 1 Aug 2013 15:19, "Andres Freund" <andres@2ndquadrant.com> wrote: > > > > On 2013-08-01 15:17:04 +0100, Greg Stark wrote: > > > We don't need per guc locking. This is the whole objection Tom had about > > > this patch being more complex than it has to be. > > > > IIRC he objected to using locking *at all* because a simple > > one-file-per-setting approach should be used. > > I am unclear why we don't need a lock around _each_ GUC, i.e. if two > sessions try to modify the same GUC at the same time. And if we need a > lock, seems we can have just one and write all the settings to one file > --- it is not like we have trouble doing locking, though this is > cluster-wide locking. If you have two sessions modifying the same variable, one is going to win and overwrite the other's setting with or without locking around GUCs unless you error out if somebody else holds the lock. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Andres Freund <andres@2ndquadrant.com> writes:
> On 2013-08-05 12:18:25 -0400, Bruce Momjian wrote:
>> I am unclear why we don't need a lock around _each_ GUC, i.e. if two
>> sessions try to modify the same GUC at the same time. And if we need a
>> lock, seems we can have just one and write all the settings to one file
>> --- it is not like we have trouble doing locking, though this is
>> cluster-wide locking.
> If you have two sessions modifying the same variable, one is going to
> win and overwrite the other's setting with or without locking around
> GUCs unless you error out if somebody else holds the lock.
The point of a lock is just to ensure that the end result is one valid
state or the other, and not something corrupt. We would certainly need a
lock if we write to a single file. With file-per-GUC, we could possibly
dispense with a lock if we depend on atomic file rename(); though whether
it's wise to assume that for Windows is unclear. (Note that we ought to
write a temp file and rename it into place anyway, to avoid possibly
corrupting the existing file on out-of-disk-space. The only thing that
needs discussion is whether to add an explicit lock around that.)
regards, tom lane
Stephen, all: (forked from main ALTER SYSTEM discussion. this thread is meant to discuss only this question: E) whether "unsafe" settings or "restart" settings should be allowed in ALTER SYSTEM SET.) On 08/02/2013 01:48 PM, Stephen Frost wrote: > Reflecting on this a bit more, I'm curious what your list-of-15 is. Based on serving literally hundreds of clients, the below are the settings we change on client servers 50% or more of the time. Other settings I touch maybe 10% of the time. THese are also, in general, the settings which I modify when we create Puppet/Chef/Salt scripts for clients. listen_addresses*@ shared_buffers*@ work_mem maintenance_work_mem effective_cache_size synchronous_commit (because of amazon) wal_buffers*@ checkpoint_segments*@ checkpoint_completion_target* (because of ext3) autovacuum* (turn off for data warehouses, turn back on for very mistaken users) stats_file_directory*@ replication/archiving settings as a set*@ wal_level, max_wal_senders, wal_keep_segments, hot_standby, archive_mode and archive_command logging settings as a set logging_collector* everything else * = requires a cold start to change @ potentially can cause failure to restart Note that two of the settings, shared_buffers and wal_buffers, become much less of an issue for restarting the system in 9.3. Also, it's possible that Heikki's automated WAL log management might deal with out-of-disk-space better than currently, which makes that less of a risk. However, you'll see that among the 11 core settings, 7 require a full restart, and 5 could "potentially cause failure to restart". That means that from my perspective, ALTER SYSTEM SET is at least 45% useless if it can't touch unsafe settngs, and 63% useless if it can't touch any setting which requires a restart. Adding the replication settings into things makes stuff significantly worse that way, although ALTER SYSTEM SET would be very useful for logging options provided that logging_collector was turned on. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Mon, Aug 5, 2013 at 12:53:24PM -0400, Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On 2013-08-05 12:18:25 -0400, Bruce Momjian wrote: > >> I am unclear why we don't need a lock around _each_ GUC, i.e. if two > >> sessions try to modify the same GUC at the same time. And if we need a > >> lock, seems we can have just one and write all the settings to one file > >> --- it is not like we have trouble doing locking, though this is > >> cluster-wide locking. > > > If you have two sessions modifying the same variable, one is going to > > win and overwrite the other's setting with or without locking around > > GUCs unless you error out if somebody else holds the lock. > > The point of a lock is just to ensure that the end result is one valid > state or the other, and not something corrupt. We would certainly need a > lock if we write to a single file. With file-per-GUC, we could possibly > dispense with a lock if we depend on atomic file rename(); though whether > it's wise to assume that for Windows is unclear. (Note that we ought to > write a temp file and rename it into place anyway, to avoid possibly > corrupting the existing file on out-of-disk-space. The only thing that > needs discussion is whether to add an explicit lock around that.) So my larger question is why a single-guc-per-file avoids corruption while having all the gucs in a single file does not. It seems the later reduces the probability of lost updates, but does not eliminate it. Also, should I be concerned that everyone removed my pg_upgrade question in their replies. :-O -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-05 12:53:24 -0400, Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On 2013-08-05 12:18:25 -0400, Bruce Momjian wrote: > >> I am unclear why we don't need a lock around _each_ GUC, i.e. if two > >> sessions try to modify the same GUC at the same time. And if we need a > >> lock, seems we can have just one and write all the settings to one file > >> --- it is not like we have trouble doing locking, though this is > >> cluster-wide locking. > > > If you have two sessions modifying the same variable, one is going to > > win and overwrite the other's setting with or without locking around > > GUCs unless you error out if somebody else holds the lock. > > The point of a lock is just to ensure that the end result is one valid > state or the other, and not something corrupt. Sure. That's what I tried to explain. I've completely missed to mention in this mail why the safe rename dance is sufficient to guarantee a correct file with one-file-per-guc... > We would certainly need a > lock if we write to a single file. With file-per-GUC, we could possibly > dispense with a lock if we depend on atomic file rename(); though whether > it's wise to assume that for Windows is unclear. Afaik it should be safe on anything NT based via Replacefile. Anything else/earlier isn't supported anyways... I think there's already a custom rename() implementation on windows? Don't we already rely on atomic renames working for the control file? > (Note that we ought to > write a temp file and rename it into place anyway, to avoid possibly > corrupting the existing file on out-of-disk-space. The only thing that > needs discussion is whether to add an explicit lock around that.) Yes, agreed. All versions of the patch that I've read have done so luckily. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Bruce Momjian <bruce@momjian.us> writes:
> So my larger question is why a single-guc-per-file avoids corruption
> while having all the gucs in a single file does not.
If it's file-per-GUC, then when two sessions try to write different GUCs,
there is no conflict. When they try to write the same GUC, the end result
will be one value or the other (assuming use of atomic rename). Which
seems fine.
If it's single-file, and we don't lock, then when two sessions try to
write different GUCs, one's update can be lost altogether, because
whichever one renames second didn't see the first one's update. That
doesn't seem acceptable.
regards, tom lane
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
(this discussion concerns issue (D), file-per-setting vs. one-big-file) On 08/05/2013 10:16 AM, Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: >> So my larger question is why a single-guc-per-file avoids corruption >> while having all the gucs in a single file does not. > > If it's file-per-GUC, then when two sessions try to write different GUCs, > there is no conflict. When they try to write the same GUC, the end result > will be one value or the other (assuming use of atomic rename). Which > seems fine. > > If it's single-file, and we don't lock, then when two sessions try to > write different GUCs, one's update can be lost altogether, because > whichever one renames second didn't see the first one's update. That > doesn't seem acceptable. I'll also point out that some of our settings only really "work" in combinations of two or more settings. For example, one doesn't want to set archive_mode = on unless one is setting archive_command as well. And generally if one sets sequential_page_cost, one is changing the other cost parameters as well. And logging parameters are generally managed as a set. So the case of two sessions both modifying ALTER SYSTEM SET, and one succeeding for some-but-all-GUCS, and the other succeeding for some-but-not-all-GUCs, would not be user-friendly or pretty, even if each setting change succeeded or failed atomically. Also, one of the reasons Amit went to one-big-file was the question of: if each setting is changed independantly, how do we know when to send the backend a reload()? IIRC, anyway. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Mon, Aug 5, 2013 at 01:16:10PM -0400, Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > So my larger question is why a single-guc-per-file avoids corruption > > while having all the gucs in a single file does not. > > If it's file-per-GUC, then when two sessions try to write different GUCs, > there is no conflict. When they try to write the same GUC, the end result > will be one value or the other (assuming use of atomic rename). Which > seems fine. > > If it's single-file, and we don't lock, then when two sessions try to > write different GUCs, one's update can be lost altogether, because > whichever one renames second didn't see the first one's update. That > doesn't seem acceptable. Yes, I understand now --- with file-per-GUC, you can say one was later than the other, but when changing two different GUCs, a single file implementation doesn't have that logical clarity. FYI, we will need to use create-rename even without the problem of creating corrupted files because we need it to avoid backends reading partially-written files. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Mon, Aug 5, 2013 at 10:21:56AM -0700, Josh Berkus wrote: > (this discussion concerns issue (D), file-per-setting vs. one-big-file) > > On 08/05/2013 10:16 AM, Tom Lane wrote: > > Bruce Momjian <bruce@momjian.us> writes: > >> So my larger question is why a single-guc-per-file avoids corruption > >> while having all the gucs in a single file does not. > > > > If it's file-per-GUC, then when two sessions try to write different GUCs, > > there is no conflict. When they try to write the same GUC, the end result > > will be one value or the other (assuming use of atomic rename). Which > > seems fine. > > > > If it's single-file, and we don't lock, then when two sessions try to > > write different GUCs, one's update can be lost altogether, because > > whichever one renames second didn't see the first one's update. That > > doesn't seem acceptable. > > I'll also point out that some of our settings only really "work" in > combinations of two or more settings. For example, one doesn't want to > set archive_mode = on unless one is setting archive_command as well. > And generally if one sets sequential_page_cost, one is changing the > other cost parameters as well. And logging parameters are generally > managed as a set. > > So the case of two sessions both modifying ALTER SYSTEM SET, and one > succeeding for some-but-all-GUCS, and the other succeeding for > some-but-not-all-GUCs, would not be user-friendly or pretty, even if > each setting change succeeded or failed atomically. Wow, that is a good point. I just mentioned in a previous email that sessions are going to be created while this is going on and they can't see partially-written files, so we need the create/rename dance. One new twist is that certain settings have to be _all_ set, or the backend is going to throw an error. > Also, one of the reasons Amit went to one-big-file was the question of: > if each setting is changed independantly, how do we know when to send > the backend a reload()? IIRC, anyway. Remember the backends starting up during this too. A global lock is looking unavoidable. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Josh,
* Josh Berkus (josh@agliodbs.com) wrote:
> Based on serving literally hundreds of clients, the below are the
> settings we change on client servers 50% or more of the time. Other
> settings I touch maybe 10% of the time. THese are also, in general, the
> settings which I modify when we create Puppet/Chef/Salt scripts for clients.
And that is actually half my point with all of this- these things are
often set by change management systems like puppet or chef and you can
be almost guaranteed that the folks running those pieces of the
infrastructure will be *very* unhappy if the DBA tries to adjust those
parameters independently because it'll almost certainly *not work*. In
particular, that goes for things like listen_addresses and port, but
also shared_buffers, checkpoint_segments, wal_keep_segments, etc.
When it comes to things like the stats_file_directory, the admin will
need to be doing things on the host server anyway to set that up
(directories, permissions, etc), so it's unclear to me why they would
care to have the ability to modify these things through ALTER SYSTEM.
Also, to reiterate, what this patch is primairly adding, from my pov
anyway, is a way for DBAs to modify settings where they don't have
access to do so directly. Today, you can prevent that from happening
while giving the DBA "superuser" (which is required generally because
we don't have sufficiently fine-grained control to avoid it) by making
postgresql.conf root-owned. To be clear, I'm not suggesting that this
is some malicious act on the part of the DBA.
In the end, I'd prefer that we shrink up the config file to just those
items which really need to be in a config file, have a conf.d structure
which allows tools like puppet/chef to more easily break up and have
different components (eg: logging) configured specially for certain
classes of systems, and then move all of the internal-to-PG type of
config (enable_*, etc) into a catalog which is modified through an
'ALTER CLUSTER SET' type command, allowing DBAs to set parameters
across the entire cluster instead of just for a given database or role.
Josh, I really have to ask- are these people who are implementing puppet
to control these configs really clamoring to have an 'ALTER SYSTEM' PG
command to have to code against instead of dealing with text files? I
feel like you're arguing for these parameters to be modifiable through
ALTER SYSTEM on the grounds that these parameters need to be set at some
point and in some way and not because having them set through ALTER
SYSTEM actually makes any *sense*.
Thanks,
Stephen
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
David Johnston
Date:
Josh Berkus wrote > (this discussion concerns issue (D), file-per-setting vs. one-big-file) > > So the case of two sessions both modifying ALTER SYSTEM SET, and one > succeeding for some-but-all-GUCS, and the other succeeding for > some-but-not-all-GUCs, would not be user-friendly or pretty, even if > each setting change succeeded or failed atomically. Can the final file write occur only at "COMMIT;" with anything inside a transaction simply staged up for later saving (or rollback). The file write phase as a whole then needs to be atomic and not just a single GUC-file. Could the system read the last update timestamp of each GUC-file when the original statement is executed and then re-read all of them at commit and fail with some kind of serialization error if the last-update timestamp on any of the files has changed? I dislike the idea of any kind of automatic reload. That said some kind of "have their been any configuration changes since last reload?" query/function makes sense. In can be plugged into Nagios or similar to warn if these changes are occurring but made live. David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/Proposal-for-Allow-postgresql-conf-values-to-be-changed-via-SQL-tp5729917p5766338.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Josh Berkus <josh@agliodbs.com> writes:
> I'll also point out that some of our settings only really "work" in
> combinations of two or more settings. For example, one doesn't want to
> set archive_mode = on unless one is setting archive_command as well.
> And generally if one sets sequential_page_cost, one is changing the
> other cost parameters as well. And logging parameters are generally
> managed as a set.
> So the case of two sessions both modifying ALTER SYSTEM SET, and one
> succeeding for some-but-all-GUCS, and the other succeeding for
> some-but-not-all-GUCs, would not be user-friendly or pretty, even if
> each setting change succeeded or failed atomically.
That is a killer point. So really the value of the global lock is to
ensure serializability when transactions are updating multiple GUCs.
regards, tom lane
Re: Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Mon, Aug 5, 2013 at 10:52:52AM -0700, David Johnston wrote: > Josh Berkus wrote > > (this discussion concerns issue (D), file-per-setting vs. one-big-file) > > > > So the case of two sessions both modifying ALTER SYSTEM SET, and one > > succeeding for some-but-all-GUCS, and the other succeeding for > > some-but-not-all-GUCs, would not be user-friendly or pretty, even if > > each setting change succeeded or failed atomically. > > Can the final file write occur only at "COMMIT;" with anything inside a > transaction simply staged up for later saving (or rollback). The file write > phase as a whole then needs to be atomic and not just a single GUC-file. Yes, the writes would only happen at commit, but I see of no way to have the renames happen atomically. > Could the system read the last update timestamp of each GUC-file when the > original statement is executed and then re-read all of them at commit and > fail with some kind of serialization error if the last-update timestamp on > any of the files has changed? That sounds like a mess --- clearly a global lock and a single file for all GUCs would be perferrable. > I dislike the idea of any kind of automatic reload. That said some kind of > "have their been any configuration changes since last reload?" > query/function makes sense. In can be plugged into Nagios or similar to > warn if these changes are occurring but made live. Who talked about automatic reload? The bigger problem is new backends starting and seeing partial state during the renames. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Josh Berkus (josh@agliodbs.com) wrote:
> I'll also point out that some of our settings only really "work" in
> combinations of two or more settings. For example, one doesn't want to
> set archive_mode = on unless one is setting archive_command as well.
And one doesn't want to set archive_command w/o doing some kind of setup
work on the server in question, perhaps setting up SSH keys or putting a
decent shell script in place to be called from archive_command. That's
part of my issue w/ this- the parameters which can't be set up correctly
through sole use of "ALTER SYSTEM" should be excluded from it and
instead set up through a config file.
Thanks,
Stephen
On 08/05/2013 10:49 AM, Stephen Frost wrote: > Josh, I really have to ask- are these people who are implementing puppet > to control these configs really clamoring to have an 'ALTER SYSTEM' PG > command to have to code against instead of dealing with text files? I > feel like you're arguing for these parameters to be modifiable through > ALTER SYSTEM on the grounds that these parameters need to be set at some > point and in some way and not because having them set through ALTER > SYSTEM actually makes any *sense*. Nope. ALTER SYSTEM, from my POV, is mainly for folks who *don't* use Puppet/Chef/whatever. Here's where I see ALTER SYSTEM being useful: * invididually managed servers with out centralized management (i.e. one DBA, one server). * developer machines (i.e. laptops and vms) * automated testing of tweaking performance parameters * setting logging parameters temporarily on systems under centralized management For that reason, the only way in which I think it makes sense to try to make ALTER SYSTEM set work together with Puppet/Chef is in the rather limited context of modifying the logging settings for limited-time data collection. Mostly, ALTER SYSTEM SET is for systems were people *aren't* using Puppet/Chef. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Mon, Aug 5, 2013 at 02:02:50PM -0400, Stephen Frost wrote: > * Josh Berkus (josh@agliodbs.com) wrote: > > I'll also point out that some of our settings only really "work" in > > combinations of two or more settings. For example, one doesn't want to > > set archive_mode = on unless one is setting archive_command as well. > > And one doesn't want to set archive_command w/o doing some kind of setup > work on the server in question, perhaps setting up SSH keys or putting a > decent shell script in place to be called from archive_command. That's > part of my issue w/ this- the parameters which can't be set up correctly > through sole use of "ALTER SYSTEM" should be excluded from it and > instead set up through a config file. Yes, but many setting changes don't require file system changes. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 08/05/2013 11:02 AM, Stephen Frost wrote: > * Josh Berkus (josh@agliodbs.com) wrote: >> I'll also point out that some of our settings only really "work" in >> combinations of two or more settings. For example, one doesn't want to >> set archive_mode = on unless one is setting archive_command as well. > > And one doesn't want to set archive_command w/o doing some kind of setup > work on the server in question, perhaps setting up SSH keys or putting a > decent shell script in place to be called from archive_command. That's > part of my issue w/ this- the parameters which can't be set up correctly > through sole use of "ALTER SYSTEM" should be excluded from it and > instead set up through a config file. I'm really trying to make progress on this discussion by separating it into the distinct, orthagonal, arguments which need to be resolved individually. You are NOT helping by then intermixing the threads. We are already discussing that on "unsafe GUCs", please keep it there. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Aug 5, 2013 at 02:02:50PM -0400, Stephen Frost wrote:
> > * Josh Berkus (josh@agliodbs.com) wrote:
> > > I'll also point out that some of our settings only really "work" in
> > > combinations of two or more settings. For example, one doesn't want to
> > > set archive_mode = on unless one is setting archive_command as well.
> >
> > And one doesn't want to set archive_command w/o doing some kind of setup
> > work on the server in question, perhaps setting up SSH keys or putting a
> > decent shell script in place to be called from archive_command. That's
> > part of my issue w/ this- the parameters which can't be set up correctly
> > through sole use of "ALTER SYSTEM" should be excluded from it and
> > instead set up through a config file.
>
> Yes, but many setting changes don't require file system changes.
Agreed, and those are the changes which would make sense to allow
through ALTER SYSTEM (or ALTER CLUSTER, or whatever)- settings which
don't require file system, network, etc, changes.
Thanks,
Stephen
On 08/05/2013 08:02 PM, Josh Berkus wrote: > On 08/05/2013 10:49 AM, Stephen Frost wrote: >> Josh, I really have to ask- are these people who are implementing puppet >> to control these configs really clamoring to have an 'ALTER SYSTEM' PG >> command to have to code against instead of dealing with text files? I >> feel like you're arguing for these parameters to be modifiable through >> ALTER SYSTEM on the grounds that these parameters need to be set at some >> point and in some way and not because having them set through ALTER >> SYSTEM actually makes any *sense*. > > Nope. ALTER SYSTEM, from my POV, is mainly for folks who *don't* use > Puppet/Chef/whatever. Here's where I see ALTER SYSTEM being useful: > > * invididually managed servers with out centralized management (i.e. one > DBA, one server). > * developer machines (i.e. laptops and vms) > * automated testing of tweaking performance parameters > * setting logging parameters temporarily on systems under centralized > management overridding the configuration system, that will just lead to very confused sysadmins why something that was configurated now behaves differently and I cause operational hazards because people _WILL_ forget changing those "temporary only" settings back? > > For that reason, the only way in which I think it makes sense to try to > make ALTER SYSTEM set work together with Puppet/Chef is in the rather > limited context of modifying the logging settings for limited-time data > collection. Mostly, ALTER SYSTEM SET is for systems were people > *aren't* using Puppet/Chef. I tend to disagree, the current approach of ALTER SYSTEM requiring superuser basically means: * in a few years from now people will just use superuser over the network for almost all stuff "because its easy and I can click around in $gui", having potential "unsafe" operations available over the network will in turn cause a lot of actual downtime (in a lot of cases the reason why people want remote management is because the don't have physical/shell access - so if they break stuff they cannot fix) * for classic IaaS/SaaS/DBaaS the ALTER SYSTEM seems to be mostly useless in the current form - because most of them will not or cannot hand out flat out superuser (like if you run a managed service you might want customers to be able to tweak some stuff but say not archive/pitr/replication stuff because the responsibility for backups is with the hosting company) Stefan
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Josh Berkus
Date:
All: (this thread concerns argument (F) whether or not admins need a way to disable ALTER SYSTEM SET) On 08/01/2013 04:03 PM, Dimitri Fontaine wrote:> If only we could trigger some actions when a command is about to be > executed, in a way that it's easy for the user to implement whatever > policy he fancies… > > Oh, maybe I should finish preparing those patches for Event Triggers to > be fully usable in 9.4 then ;) I don't see this as a solution at all. "Mr. Sysadmin, we've given the DBAs a new tool which allows them to override your version-controlled database parameter settings. You can turn it off, though, by using this incredibly complicated, brand-new Event Trigger tool which requires writing lots of SQL code to make work." Per Stephen Frost's arguments, some system owners are going to be opposed to allowing ALTER SYSTEM SET at all because it can mess systems up and cause downtime. Yes, that's already true of ALTER ROLE and ALTER DATABASE, but ALTER SYSTEM SET expands this the ability of the DBA to change setting substantially. That's obviously its benefit, but it's also clearly a misfeature for some system owners. Also, to be blunt, most DBAs/DEVs don't *know* about ALTER ROLE/DATABASE. I don't think this is a small thing. I really think we'll get a LOT of blowback from sysadmins -- and maybe even refusals to upgrade -- if we add ALTER SYSTEM SET in 9.4.0 with no easy way to disable it. Having an easy way to disable ALTER SYSTEM SET is *also* a good way to get out of the whole situation of "I set shared_buffers to 100GB using ALTER SYSTEM SET and now PostgreSQL won't start" problem. As I've said before, I think the disable switch can be a follow-up patch to the main ALTER SYSTEM SET patch. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
On 08/05/2013 07:01 PM, Josh Berkus wrote: > Stephen, all: > > (forked from main ALTER SYSTEM discussion. this thread is meant to > discuss only this question: > > E) whether "unsafe" settings or "restart" settings should be allowed in > ALTER SYSTEM SET.) > > On 08/02/2013 01:48 PM, Stephen Frost wrote: >> Reflecting on this a bit more, I'm curious what your list-of-15 is. > > Based on serving literally hundreds of clients, the below are the > settings we change on client servers 50% or more of the time. Other > settings I touch maybe 10% of the time. THese are also, in general, the > settings which I modify when we create Puppet/Chef/Salt scripts for clients. > > listen_addresses*@ > shared_buffers*@ > work_mem > maintenance_work_mem > effective_cache_size > synchronous_commit (because of amazon) > wal_buffers*@ > checkpoint_segments*@ > checkpoint_completion_target* (because of ext3) > autovacuum* (turn off for data warehouses, > turn back on for very mistaken users) > stats_file_directory*@ > > replication/archiving settings as a set*@ > wal_level, max_wal_senders, wal_keep_segments, hot_standby, archive_mode > and archive_command > > logging settings as a set > logging_collector* > everything else > > * = requires a cold start to change > @ potentially can cause failure to restart > > Note that two of the settings, shared_buffers and wal_buffers, become > much less of an issue for restarting the system in 9.3. Also, it's > possible that Heikki's automated WAL log management might deal with > out-of-disk-space better than currently, which makes that less of a risk. > > However, you'll see that among the 11 core settings, 7 require a full > restart, and 5 could "potentially cause failure to restart". That means > that from my perspective, ALTER SYSTEM SET is at least 45% useless if it > can't touch unsafe settngs, and 63% useless if it can't touch any > setting which requires a restart. Adding the replication settings into > things makes stuff significantly worse that way, although ALTER SYSTEM > SET would be very useful for logging options provided that > logging_collector was turned on. not sure at all I agree with our "% useless" measure but we need to consider that having all of those available over remote means they will suddenly become "action at a distance" thingies, people will play with them more and randomly change stuff, and a lot of those can break the entire system because of say overrunning system resources. The same thing can happen now just as well, but having them available from remote will also result in tools doing this and people that have less information about the hardware and system or what else is going on on that box. Also we have to keep in mind that in most scenarios the logfile and potentially reported errors/warnings there will be useless because people will never see them... Stefan
On 08/05/2013 11:14 AM, Stefan Kaltenbrunner wrote: > * in a few years from now people will just use superuser over the > network for almost all stuff "because its easy and I can click around in > $gui", having potential "unsafe" operations available over the network > will in turn cause a lot of actual downtime (in a lot of cases the > reason why people want remote management is because the don't have > physical/shell access - so if they break stuff they cannot fix) See thread "Disabling ALTER SYSTEM SET". > * for classic IaaS/SaaS/DBaaS the ALTER SYSTEM seems to be mostly > useless in the current form - because most of them will not or cannot > hand out flat out superuser (like if you run a managed service you might > want customers to be able to tweak some stuff but say not > archive/pitr/replication stuff because the responsibility for backups is > with the hosting company) 100% in agreement. If someone thought we were serving DBAAS with this, they haven't paid attention to the patch. However, there are other places where ALTER SYSTEM SET will be valuable.For example, for anyone who wants to implement anautotuning utility. For example, I'm writing a network utility which checks bgwriter stats and tries adjusting settings over the network to improve checkpoint issues. Not having to SSH configuration files into place (and make sure they're not overridden by other configuration files) would make writing that script a *lot* easier. Same thing with automated performance testing. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Bruce Momjian
Date:
On Mon, Aug 5, 2013 at 11:16:24AM -0700, Josh Berkus wrote: > I don't see this as a solution at all. "Mr. Sysadmin, we've given the > DBAs a new tool which allows them to override your version-controlled > database parameter settings. You can turn it off, though, by using this > incredibly complicated, brand-new Event Trigger tool which requires > writing lots of SQL code to make work." > > Per Stephen Frost's arguments, some system owners are going to be > opposed to allowing ALTER SYSTEM SET at all because it can mess systems > up and cause downtime. Yes, that's already true of ALTER ROLE and ALTER > DATABASE, but ALTER SYSTEM SET expands this the ability of the DBA to > change setting substantially. That's obviously its benefit, but it's > also clearly a misfeature for some system owners. Also, to be blunt, > most DBAs/DEVs don't *know* about ALTER ROLE/DATABASE. > > I don't think this is a small thing. I really think we'll get a LOT of > blowback from sysadmins -- and maybe even refusals to upgrade -- if we ---------------------------------- Really? Is that a reasonable statement? > add ALTER SYSTEM SET in 9.4.0 with no easy way to disable it. Having an > easy way to disable ALTER SYSTEM SET is *also* a good way to get out of > the whole situation of "I set shared_buffers to 100GB using ALTER SYSTEM > SET and now PostgreSQL won't start" problem. > > As I've said before, I think the disable switch can be a follow-up patch > to the main ALTER SYSTEM SET patch. You are mixing the behaviors of disabling the ALTER SYSTEM SET command with recognizing ALTER SYSTEM SET settings already made --- the later would be to fix a problem with the server not starting. However, frankly, these are flat files, so I don't see a problem with having the administrator modify the flat file. Would disabling the ALTER SYSTEM SET command also disable recognizing any ALTER SYSTEM SET commands already performed? Maybe that was already understood, but I missed that point. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
* Josh Berkus (josh@agliodbs.com) wrote:
> Nope. ALTER SYSTEM, from my POV, is mainly for folks who *don't* use
> Puppet/Chef/whatever.
Ok, that's fine, but let's try to avoid making life difficult for those
who *do* use puppet/chef/whatever. This capability runs a very high
risk of that by allowing a DBA who *isn't* a sysadmin to go modifying
things that depend on external-to-PG factors.
> Here's where I see ALTER SYSTEM being useful:
>
> * invididually managed servers with out centralized management (i.e. one
> DBA, one server).
> * developer machines (i.e. laptops and vms)
The above strikes me as being already dealt with through pgAdmin and the
'admin pack', if the user wants a GUI to use for modifying these
parameters (which seems like what they'd primairly get out of ALTER
SYSTEM SET- pgAdmin, or whatever $gui wouldn't have to depend on the
admin pack).
> * automated testing of tweaking performance parameters
This sounds like you'd need tooling around it to make it work anyway,
which could probably handle modifying a text file, but even if not,
these paremeters may be on the 'safe' list.
> * setting logging parameters temporarily on systems under centralized
> management
This is the kind of argument that I could get behind- in an environment
where logs are shipped to a server where DBAs can view and analyze them,
being able to modify the logging parameters on the fly could be useful
(eg: log_min_duration_statement or similar). As I tried to get at
up-thread, my concern is primairly around those parameters which can't
be set sensibly through ALTER SYSTEM because they depend on other
activities happening.
Thanks,
Stephen
On 08/05/2013 08:21 PM, Josh Berkus wrote: > On 08/05/2013 11:14 AM, Stefan Kaltenbrunner wrote: >> * in a few years from now people will just use superuser over the >> network for almost all stuff "because its easy and I can click around in >> $gui", having potential "unsafe" operations available over the network >> will in turn cause a lot of actual downtime (in a lot of cases the >> reason why people want remote management is because the don't have >> physical/shell access - so if they break stuff they cannot fix) > > See thread "Disabling ALTER SYSTEM SET". > >> * for classic IaaS/SaaS/DBaaS the ALTER SYSTEM seems to be mostly >> useless in the current form - because most of them will not or cannot >> hand out flat out superuser (like if you run a managed service you might >> want customers to be able to tweak some stuff but say not >> archive/pitr/replication stuff because the responsibility for backups is >> with the hosting company) > > 100% in agreement. If someone thought we were serving DBAAS with this, > they haven't paid attention to the patch. > > However, there are other places where ALTER SYSTEM SET will be valuable. > For example, for anyone who wants to implement an autotuning utility. > For example, I'm writing a network utility which checks bgwriter stats > and tries adjusting settings over the network to improve checkpoint > issues. Not having to SSH configuration files into place (and make sure > they're not overridden by other configuration files) would make writing > that script a *lot* easier. Same thing with automated performance testing. seems like an excessively narrow usecase to me - people doing that kind of specific testing can easily do automation over ssh, and those are very few vs. having to maintain a fairly complex piece of code in postgresql core. Nevertheless my main point is that people _WILL_ use this as a simple convinience tool not fully understanding all the complex implications, and in a few years from now running people with superuser by default (because people will create "cool little tools say to change stuff from my tray or using $IOS app" that have a little small comment "make sure to create the user "WITH SUPERUSER" and people will follow like lemmings. Stefan
On 08/05/2013 11:28 AM, Stephen Frost wrote: > * Josh Berkus (josh@agliodbs.com) wrote: >> Nope. ALTER SYSTEM, from my POV, is mainly for folks who *don't* use >> Puppet/Chef/whatever. > > Ok, that's fine, but let's try to avoid making life difficult for those > who *do* use puppet/chef/whatever. This capability runs a very high > risk of that by allowing a DBA who *isn't* a sysadmin to go modifying > things that depend on external-to-PG factors. See thread "Disabling ALTER SYSTEM SET". In short, I agree with you. > >> Here's where I see ALTER SYSTEM being useful: >> >> * invididually managed servers with out centralized management (i.e. one >> DBA, one server). >> * developer machines (i.e. laptops and vms) > > The above strikes me as being already dealt with through pgAdmin and the > 'admin pack', if the user wants a GUI to use for modifying these > parameters (which seems like what they'd primairly get out of ALTER > SYSTEM SET- pgAdmin, or whatever $gui wouldn't have to depend on the > admin pack). Except that forcing developers to install the admin pack and pgadmin to get this functionality is a high barrier to entry exactly where we don't want one. > >> * automated testing of tweaking performance parameters > > This sounds like you'd need tooling around it to make it work anyway, > which could probably handle modifying a text file, but even if not, > these paremeters may be on the 'safe' list. Well, frankly, it's the main reason why *I* want ALTER SYSTEM SET. It makes my job writing automated testing scripts easier. Certainly it was possible before, but there's value in "easier". And that's the reason I don't want you to take away the ability to modify shared_buffers et. al. ;-) On 08/05/2013 11:30 AM, Stefan Kaltenbrunner wrote:> Nevertheless my main point is that people _WILL_ use this as a simple > convinience tool not fully understanding all the complex implications, > and in a few years from now running people with superuser by default > (because people will create "cool little tools say to change stuff from > my tray or using $IOS app" that have a little small comment "make sure > to create the user "WITH SUPERUSER" and people will follow like lemmings. Most of our users not on Heroku are running with superuser as the app user now. Like, 95% of them based on my personal experience (because our object permissions management sucks). In that this feature will further discourage people from having a separate application user, there's some argument. However, it's really an argument for not having ALTER SYSTEM SET *at all* rather than restricting it to "safe" GUCs, no? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
* Josh Berkus (josh@agliodbs.com) wrote:
> On 08/05/2013 11:14 AM, Stefan Kaltenbrunner wrote:
> > * in a few years from now people will just use superuser over the
> > network for almost all stuff "because its easy and I can click around in
> > $gui", having potential "unsafe" operations available over the network
> > will in turn cause a lot of actual downtime (in a lot of cases the
> > reason why people want remote management is because the don't have
> > physical/shell access - so if they break stuff they cannot fix)
>
> See thread "Disabling ALTER SYSTEM SET".
I'm really not thrilled with this "solution".
> However, there are other places where ALTER SYSTEM SET will be valuable.
> For example, for anyone who wants to implement an autotuning utility.
> For example, I'm writing a network utility which checks bgwriter stats
> and tries adjusting settings over the network to improve checkpoint
> issues. Not having to SSH configuration files into place (and make sure
> they're not overridden by other configuration files) would make writing
> that script a *lot* easier. Same thing with automated performance testing.
I've done a fair bit of this myself and find templating postgresql.conf
isn't really all that hard and comes in handy for a lot things, not to
mention that it can then be integrated into a configuration management
system more easily.. Still, I'm fine w/ parameters which don't depend
on external things happening, which I think covers a lot of this
use-case.
Thanks,
Stephen
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Stephen Frost
Date:
* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Aug 5, 2013 at 11:16:24AM -0700, Josh Berkus wrote:
> > I don't think this is a small thing. I really think we'll get a LOT of
> > blowback from sysadmins -- and maybe even refusals to upgrade -- if we
> ----------------------------------
>
> Really? Is that a reasonable statement?
I don't believe Josh was saying that in jest.. I certainly believe that
it could happen.
> > As I've said before, I think the disable switch can be a follow-up patch
> > to the main ALTER SYSTEM SET patch.
Even if it's a different patch, it should go in at the same time, imv..
> You are mixing the behaviors of disabling the ALTER SYSTEM SET command
> with recognizing ALTER SYSTEM SET settings already made --- the later
> would be to fix a problem with the server not starting.
That's a good point.
> However,
> frankly, these are flat files, so I don't see a problem with having the
> administrator modify the flat file.
Admins on Ubuntu or Debian or a host of their derivatives aren't going
to be looking in $PGDATA for config files that they have to hand-modify
to fix something the DBA did. When they eventually figure it out,
they're going to be *very* unhappy.
> Would disabling the ALTER SYSTEM SET command also disable recognizing
> any ALTER SYSTEM SET commands already performed? Maybe that was already
> understood, but I missed that point.
I don't have an answer to that one, though I do like the idea of a
switch that says "only read the settings from my postgresql.conf file".
Thanks, Stephen
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Bruce Momjian
Date:
On Mon, Aug 5, 2013 at 02:52:40PM -0400, Stephen Frost wrote: > > However, > > frankly, these are flat files, so I don't see a problem with having the > > administrator modify the flat file. > > Admins on Ubuntu or Debian or a host of their derivatives aren't going > to be looking in $PGDATA for config files that they have to hand-modify > to fix something the DBA did. When they eventually figure it out, > they're going to be *very* unhappy. Well, can you assume that if you have a problem with one of your ALTER SYSTEM SET commands, that disabling _all_ of them is going to get you a running system? I question that, e.g. port. With postgresql.conf, you can modify the bad entry, but how would that happen with ALTER SYSTEM SET? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Tom Lane
Date:
Bruce Momjian <bruce@momjian.us> writes:
> Well, can you assume that if you have a problem with one of your ALTER
> SYSTEM SET commands, that disabling _all_ of them is going to get you a
> running system? I question that, e.g. port. With postgresql.conf, you
> can modify the bad entry, but how would that happen with ALTER SYSTEM
> SET?
I think we already have consensus that the settings will be in text files,
so that system breakage can be handled by editing the files.
What Josh seems to be concerned with in this thread is the question of
whether we should support an installation *policy decision* not to allow
ALTER SYSTEM SET. Not because a particular set of parameters is broken,
but just because somebody is afraid the DBA might break things. TBH
I'm not sure I buy that, at least not as long as ALTER SYSTEM is a
superuser feature. There is nothing in Postgres that denies permissions
to superusers, and this doesn't seem like a very good place to start.
regards, tom lane
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Alvaro Herrera
Date:
Tom Lane escribió: > What Josh seems to be concerned with in this thread is the question of > whether we should support an installation *policy decision* not to allow > ALTER SYSTEM SET. Not because a particular set of parameters is broken, > but just because somebody is afraid the DBA might break things. TBH > I'm not sure I buy that, at least not as long as ALTER SYSTEM is a > superuser feature. There is nothing in Postgres that denies permissions > to superusers, and this doesn't seem like a very good place to start. Someone made an argument about this on IRC: GUI tool users are going to want to use ALTER SYSTEM through point-and-click, and if all we offer is superuser-level access to the feature, we're going to end up with a lot of people running with superuser privileges just so that they are able to tweak inconsequential settings. This seems dangerous. The other issue is that currently you can only edit a server's config if you are logged in to it. If we permit SQL-level access to that, and somebody who doesn't have access to edit the files blocks themselves out, there is no way for them to get a working system *at all*. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Dimitri Fontaine
Date:
Hi, I'm now completely lost in the current threads. Is there a single valid use case for the feature we're trying to design? Who is the target audience of this patch? Josh Berkus <josh@agliodbs.com> writes: > I don't see this as a solution at all. "Mr. Sysadmin, we've given the > DBAs a new tool which allows them to override your version-controlled > database parameter settings. You can turn it off, though, by using this > incredibly complicated, brand-new Event Trigger tool which requires > writing lots of SQL code to make work." Well, given what has been said already and very recently again by Tom, it's superuser only and installing a precedent wherein superuser is not allowed to use a feature looks like a dead-end. You would have to make a case that it's comparable to allow_system_table_mods. If you believe that revoking ALTER SYSTEM SET privileges to superusers isn't going to be accepted, I know of only two other paths to allow you to implementing your own policy, including per-GUC policy and non-superuser granting of ALTER SYSTEM SET in a controled fashion: - add per GUC GRANT/REVOKE capabilities to SETTINGs, - implement the same thing within an Event Trigger. The former has been talked about lots of time already in the past and I'm yet to see any kind of progress made about it despite plenty of user support for the feature, the latter requires a shared catalog for global object Event Triggers and maybe a flexible Extension that you can manage by just dropping a configuration file into the PostgreSQL conf.d. So when trying to be realistic the answer is "incredibly complicated" because it involves a stored procedure to implement the local policy and a command to enable the policy, really, I wonder who you're addressing there. Certainly not DBA, so that must be sysadmins, who would be better off without the feature in the first place if I'm understanding you. Again, what are we trying to achieve?! Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Tom Lane
Date:
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Tom Lane escribi�:
>> What Josh seems to be concerned with in this thread is the question of
>> whether we should support an installation *policy decision* not to allow
>> ALTER SYSTEM SET. Not because a particular set of parameters is broken,
>> but just because somebody is afraid the DBA might break things. TBH
>> I'm not sure I buy that, at least not as long as ALTER SYSTEM is a
>> superuser feature. There is nothing in Postgres that denies permissions
>> to superusers, and this doesn't seem like a very good place to start.
> Someone made an argument about this on IRC: GUI tool users are going to
> want to use ALTER SYSTEM through point-and-click, and if all we offer is
> superuser-level access to the feature, we're going to end up with a lot
> of people running with superuser privileges just so that they are able
> to tweak inconsequential settings. This seems dangerous.
Agreed, but what else are you going to do? You can't have random
unprivileged users changing settings that affect other users, even
if those settings are somehow "safe". And what is more to the point
in this thread, having an additional shutoff that prevents even superusers
from doing it doesn't reduce the temptation for everyone to make
themselves superuser all the time.
> The other issue is that currently you can only edit a server's config if
> you are logged in to it. If we permit SQL-level access to that, and
> somebody who doesn't have access to edit the files blocks themselves
> out, there is no way for them to get a working system *at all*.
True. So, if they were smart enough to prevent themselves from changing
any settings remotely, they have no feature. And if they weren't, having
such a blocking capability didn't really help them either.
These are both valid worries, but what design is going to make them
better? Other than forgetting about ALTER SYSTEM entirely?
regards, tom lane
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Rodrigo Gonzalez
Date:
On Mon, 5 Aug 2013 15:53:01 -0400 Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > The other issue is that currently you can only edit a server's config > if you are logged in to it. If we permit SQL-level access to that, > and somebody who doesn't have access to edit the files blocks > themselves out, there is no way for them to get a working system *at > all*. > This is not a valid point, at least for me....basically all OSs should disable any change to firewall if you are connected remotely cause you can block yourself....giving power to users does not mean being their babysitter....we should smart enough to use the power we receive...if we are not smart...we should start thinking about other profession And excuse my English, I hope the point is clear... Best regards Rodrigo Gonzalez
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Stefan Kaltenbrunner
Date:
On 08/05/2013 10:18 PM, Dimitri Fontaine wrote: > Hi, > > I'm now completely lost in the current threads. Is there a single valid > use case for the feature we're trying to design? Who is the target > audience of this patch? wonder about that myself... > > Josh Berkus <josh@agliodbs.com> writes: >> I don't see this as a solution at all. "Mr. Sysadmin, we've given the >> DBAs a new tool which allows them to override your version-controlled >> database parameter settings. You can turn it off, though, by using this >> incredibly complicated, brand-new Event Trigger tool which requires >> writing lots of SQL code to make work." > > Well, given what has been said already and very recently again by Tom, > it's superuser only and installing a precedent wherein superuser is not > allowed to use a feature looks like a dead-end. You would have to make a > case that it's comparable to allow_system_table_mods. > > If you believe that revoking ALTER SYSTEM SET privileges to superusers > isn't going to be accepted, I know of only two other paths to allow you > to implementing your own policy, including per-GUC policy and > non-superuser granting of ALTER SYSTEM SET in a controled fashion: > > - add per GUC GRANT/REVOKE capabilities to SETTINGs, realistically I think this is what we "want"(fsvo) for this feature as a prerequisite, however that also will make it fairly complex to use for both humans and tools so not sure we would really gain anything... > - implement the same thing within an Event Trigger. > > The former has been talked about lots of time already in the past and > I'm yet to see any kind of progress made about it despite plenty of user > support for the feature, the latter requires a shared catalog for global > object Event Triggers and maybe a flexible Extension that you can manage > by just dropping a configuration file into the PostgreSQL conf.d. > > So when trying to be realistic the answer is "incredibly complicated" > because it involves a stored procedure to implement the local policy and > a command to enable the policy, really, I wonder who you're addressing > there. Certainly not DBA, so that must be sysadmins, who would be better > off without the feature in the first place if I'm understanding you. > > Again, what are we trying to achieve?! no idea - wondering about that myself... Stefan
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Stefan Kaltenbrunner
Date:
On 08/05/2013 09:53 PM, Alvaro Herrera wrote: > Tom Lane escribió: > >> What Josh seems to be concerned with in this thread is the question of >> whether we should support an installation *policy decision* not to allow >> ALTER SYSTEM SET. Not because a particular set of parameters is broken, >> but just because somebody is afraid the DBA might break things. TBH >> I'm not sure I buy that, at least not as long as ALTER SYSTEM is a >> superuser feature. There is nothing in Postgres that denies permissions >> to superusers, and this doesn't seem like a very good place to start. > > Someone made an argument about this on IRC: GUI tool users are going to > want to use ALTER SYSTEM through point-and-click, and if all we offer is > superuser-level access to the feature, we're going to end up with a lot > of people running with superuser privileges just so that they are able > to tweak inconsequential settings. This seems dangerous. indeed it is > > The other issue is that currently you can only edit a server's config if > you are logged in to it. If we permit SQL-level access to that, and > somebody who doesn't have access to edit the files blocks themselves > out, there is no way for them to get a working system *at all*. thinking more about that - is there _ANY_ prerequisite of an application that can be completely reconfigured over a remote access protocol and solved the reliability and security challenges of that to a reasonable degree? Stefan
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Josh Berkus
Date:
Stefan, > thinking more about that - is there _ANY_ prerequisite of an application > that can be completely reconfigured over a remote access protocol and > solved the reliability and security challenges of that to a reasonable > degree? Good question! I also think that, given the issues raised in discussion of ALTER SYSTEM SET, we should poll a bit of the wider community. I'll put something up on my blog, unless someone can think of a better way. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Mon, Aug 5, 2013 at 01:56:40PM -0400, Tom Lane wrote: > Josh Berkus <josh@agliodbs.com> writes: > > I'll also point out that some of our settings only really "work" in > > combinations of two or more settings. For example, one doesn't want to > > set archive_mode = on unless one is setting archive_command as well. > > And generally if one sets sequential_page_cost, one is changing the > > other cost parameters as well. And logging parameters are generally > > managed as a set. > > > So the case of two sessions both modifying ALTER SYSTEM SET, and one > > succeeding for some-but-all-GUCS, and the other succeeding for > > some-but-not-all-GUCs, would not be user-friendly or pretty, even if > > each setting change succeeded or failed atomically. > > That is a killer point. So really the value of the global lock is to > ensure serializability when transactions are updating multiple GUCs. Well, I think it is more than that. The global lock will allow multiple GUCs to be modified in a serializable fashion, but unless you have new backends wait until that lock is clear, new backends are going to see the directory in an inconsistent state if we keep a single-guc-per-file. So, unless we want new backends to hang during ALTER SYSTEM SET operations, I think we are going to need a global lock and all gucs in a single file. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Bruce Momjian
Date:
On Mon, Aug 5, 2013 at 03:53:01PM -0400, Alvaro Herrera wrote: > The other issue is that currently you can only edit a server's config if > you are logged in to it. If we permit SQL-level access to that, and > somebody who doesn't have access to edit the files blocks themselves > out, there is no way for them to get a working system *at all*. Well, if we want to give the administrator a way to disable honoring any previously-defined ALTER SYSTEM SET commands, how would they do that without OS access? By definition, they can't connect via SQL, so what would the API be? Also, even if they could do it remotely, if they previously set listen_addresses via ALTER SYSTEM SET, and we then disable all ALTER SYSTEM SET settings, they still can't access the system because by default Postgres will only listen on local sockets. In summary, the SQL interface to configuration parameters is a convenience, but I don't think it is ever going to be something that can replace full file system access --- that is not a limitation of the implemention of ALTER SYSTEM SET, but just something that is impossible. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On Mon, Aug 5, 2013 at 7:02 PM, Josh Berkus <josh@agliodbs.com> wrote: > Nope. ALTER SYSTEM, from my POV, is mainly for folks who *don't* use > Puppet/Chef/whatever. Here's where I see ALTER SYSTEM being useful: > > * invididually managed servers with out centralized management (i.e. one > DBA, one server). Well any server that is treated as a one-off rather than part of a cluster of identical servers. That will actually be true for a lot of users even if they have a slave or two and a development machine or three. It's only when you start managing multiple production servers for multiple databasees that you start to really see the benefits of Puppet/Chef in solving your problems once for every server. Fwiw It's not hard to imagine Puppet/Chef modules that *do* work through ALTER SYSTEM and there would even be advantages. They could easily check what the current value is and even adjust the values differently for each system based on the environment. This is a bit trickier to manage when rolling out files than it is when connected to an instance. I would not be surprised to see both styles be quite common, possibly even in the same environment for different settings (I'm not saying I recommend that though). -- greg
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Greg Stark
Date:
On Mon, Aug 5, 2013 at 6:16 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > If it's file-per-GUC, then when two sessions try to write different GUCs, > there is no conflict Well it's not that there's no conflict. It's just that we don't detect the conflict. If you raise shared_buffers and I calculate the optimal work_mem based on the old shared_buffers and set work_mem I may well set it too large. Or if I set multiple settings together and you set one of them you'll undo my change and lose just part of my changes but not stop me from setting the others inconsistently. I think it's actually better to lose a whole set of changes than to intermix two sets of changes in unpredictable ways. And I would suggest instead of doing a rename we just create a second link leaving behind the original link as well. That means you wouldn't be losing the other changes entirely you would have the history of the settings as they were briefly at least. -- greg
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-05 13:56:40 -0400, Tom Lane wrote: > Josh Berkus <josh@agliodbs.com> writes: > > I'll also point out that some of our settings only really "work" in > > combinations of two or more settings. For example, one doesn't want to > > set archive_mode = on unless one is setting archive_command as well. > > And generally if one sets sequential_page_cost, one is changing the > > other cost parameters as well. And logging parameters are generally > > managed as a set. > > > So the case of two sessions both modifying ALTER SYSTEM SET, and one > > succeeding for some-but-all-GUCS, and the other succeeding for > > some-but-not-all-GUCs, would not be user-friendly or pretty, even if > > each setting change succeeded or failed atomically. > > That is a killer point. So really the value of the global lock is to > ensure serializability when transactions are updating multiple GUCs. I don't think a global lock helps very much WRT this. Likely the lock will be released after a single SET finishes, in that case it doesn't guarantee much for a set of SETs. Even if we were to make it be released only at session end, the session that wants to set something conflicting will only be blocked by wanting to set the lock, it won't be prevented from doing so. The SET will then succeed after the other session finished. I don't think we're designing a feature that's supposed to be used under heavy concurrency here. If you have users/tools doing conflicting actions as superusers you need to solve that by social means, not by technical ones. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Josh Berkus
Date:
On 08/05/2013 05:12 PM, Andres Freund wrote: > I don't think we're designing a feature that's supposed to be used under > heavy concurrency here. If you have users/tools doing conflicting > actions as superusers you need to solve that by social means, not by > technical ones. +1 -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Greg Stark <stark@mit.edu> writes: > set it too large. Or if I set multiple settings together and you set > one of them you'll undo my change and lose just part of my changes but > not stop me from setting the others inconsistently. So we need to be able to change more than one setting in a single command and have a lock around the whole command, or maybe have ALTER SYSTEM SET run in a transaction. -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Cédric Villemain
Date:
> > Again, what are we trying to achieve?! > > no idea - wondering about that myself... It seems we are trying to add grammar for modifying postgresql.conf. Something we can already do easily in a standard extension, but without grammar changes. Maybe better to provide a contrib/ to modify config, then design what we can achieve more with an ALTER SYSTEM command. -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Tom Lane
Date:
Cédric Villemain <cedric@2ndquadrant.com> writes:
>>> Again, what are we trying to achieve?!
>> no idea - wondering about that myself...
> It seems we are trying to add grammar for modifying postgresql.conf.
> Something we can already do easily in a standard extension, but without
> grammar changes.
> Maybe better to provide a contrib/ to modify config, then design what we can
> achieve more with an ALTER SYSTEM command.
Hmm ... putting the UI into a contrib module would neatly solve the
problem for people who don't want the functionality. They just don't
install the module. Another thought here is that it seems like we've
been discussing multiple scenarios that don't necessarily all have the
same best solution. Maybe we should think in terms of multiple contrib
modules.
regards, tom lane
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Mon, Aug 5, 2013 at 05:29:48PM -0400, Bruce Momjian wrote: > > That is a killer point. So really the value of the global lock is to > > ensure serializability when transactions are updating multiple GUCs. > > Well, I think it is more than that. The global lock will allow multiple > GUCs to be modified in a serializable fashion, but unless you have new > backends wait until that lock is clear, new backends are going to see > the directory in an inconsistent state if we keep a single-guc-per-file. > > So, unless we want new backends to hang during ALTER SYSTEM SET > operations, I think we are going to need a global lock and all gucs in a > single file. To clarify, I talked about new backends above, but on Unix and Windows, new backends inherit the postmaster GUC settings, so it would only be config file reload that would possibly see the single-guc-per-file in an inconsistent state. And we already have this possibility if someone edits postgresql.conf, and while the editor is writing the file, a backend is reading the file via a reload. Now, I assume that ALTER SYSTEM SET would automatically issue a pg_reload_conf(), so we would need to make sure that people modifying multiple parameters that are related do it in a single transaction, and that we issue a pg_reload_conf() only after all values have been changed. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Bruce Momjian
Date:
On Tue, Aug 6, 2013 at 12:40:12PM +0200, Cédric Villemain wrote: > > > Again, what are we trying to achieve?! > > > > no idea - wondering about that myself... > > It seems we are trying to add grammar for modifying postgresql.conf. > Something we can already do easily in a standard extension, but without > grammar changes. > > Maybe better to provide a contrib/ to modify config, then design what we can > achieve more with an ALTER SYSTEM command. Let's look at the problems: * remote users can lock themselves out of the server * interconnected GUC variables are complex to change * need a way to disable this feature Given the above, I am not sure I see a way forward for ALTER SYSTEM SET. One compromise that avoids the problems above would be to have a limited feature called ALTER LOG SET that only controls logging parameters, but I am not sure there is enough use-case there. This is not a zero-cost feature as we would be giving people _two_ ways to configure Postgres, and two ways to find a fix if it is broken, so we have to have a clear win to implement this. Also, if you have broken the system via ALTER SYSTEM SET, you might need to edit flat files to fix it, adding further complexity and limitations if someone only knows the SQL method of configuration. Given that, and the problems above, I reluctantly just don't see how this features moves us forward. There seemed to be agreement on having a config.d, though. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Cédric Villemain
Date:
> There seemed to be agreement on having a config.d, though. Yes. Also, the validate_conf_parameter() (or some similar name) Amit added in his patch sounds useful if an extension can use it (to check a GUC someone want to change, to check a configuration file, ...) -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tuesday, August 06, 2013 5:39 PM Bruce Momjian wrote: > On Mon, Aug 5, 2013 at 05:29:48PM -0400, Bruce Momjian wrote: > > > That is a killer point. So really the value of the global lock is > to > > > ensure serializability when transactions are updating multiple > GUCs. > > > > Well, I think it is more than that. The global lock will allow > multiple > > GUCs to be modified in a serializable fashion, but unless you have > new > > backends wait until that lock is clear, new backends are going to see > > the directory in an inconsistent state if we keep a single-guc-per- > file. > > > > So, unless we want new backends to hang during ALTER SYSTEM SET > > operations, I think we are going to need a global lock and all gucs > in a > > single file. > > To clarify, I talked about new backends above, but on Unix and Windows, > new backends inherit the postmaster GUC settings, so it would only be > config file reload that would possibly see the single-guc-per-file in > an > inconsistent state. And we already have this possibility if someone > edits postgresql.conf, and while the editor is writing the file, a > backend is reading the file via a reload. > > Now, I assume that ALTER SYSTEM SET would automatically issue a > pg_reload_conf(), so we would need to make sure that people modifying > multiple parameters that are related do it in a single transaction, and > that we issue a pg_reload_conf() only after all values have been > changed. Can't we leave this onus of conflict of changing dependent parameters on user, such that he should do it carefully? Other databases SQL Server, Oracle also has similar feature, so there user's also must be handling by themselves. With Regards, Amit Kapila.
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
'Bruce Momjian'
Date:
On Tue, Aug 6, 2013 at 06:30:18PM +0530, Amit Kapila wrote: > > Now, I assume that ALTER SYSTEM SET would automatically issue a > > pg_reload_conf(), so we would need to make sure that people modifying > > multiple parameters that are related do it in a single transaction, and > > that we issue a pg_reload_conf() only after all values have been > > changed. > > Can't we leave this onus of conflict of changing dependent parameters on > user, such that he should do it carefully? > Other databases SQL Server, Oracle also has similar feature, so there user's > also must be handling by themselves. Yes, we _can_ have the user deal with it, but we have to consider the final user experience. The initial experience will be, "Oh, great, no more need to edit a configuration file,", but it might end with: * Oh, my server doesn't start and I don't have shell access * How do I undo an ALTER SYSTEM SET if the server can't be started * Why doesn't my server start? postgresql.conf looks fine (see ALTER SYSTEM SET) * What things did I change via ALTER SYSTEM SET * How do I disable ALTER SYSTEM SET so only postgresql.conf works Consider how long it took the hackers to realize all the interactions involved; we would have to effectively communicate that to users. I realize other database systems have this, but those systems are not known to be easy to use. ALTER SYSTEM SET has the promise of making Postgres easier to use, _and_ harder to use. The question is what percentage of users are going to have a positive experience with this feature, and what percentage are going to have a negative or disastrous experience with it, or disable it? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Andres Freund (andres@2ndquadrant.com) wrote:
> I don't think we're designing a feature that's supposed to be used under
> heavy concurrency here. If you have users/tools doing conflicting
> actions as superusers you need to solve that by social means, not by
> technical ones.
If this actually gets used by puppet or another CMS, the chances of the
'social means' being successful drop drastically.
I agree that it doesn't need to work under heavy concurrency, but it
should do something sensible if it happens- perhaps even just throwing
an error if it can't acquire the lock immediately, warning the user that
some other process is trying to modify the config concurrently.
Thanks,
Stephen
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Stephen Frost
Date:
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Cédric Villemain <cedric@2ndquadrant.com> writes:
> > Maybe better to provide a contrib/ to modify config, then design what we can
> > achieve more with an ALTER SYSTEM command.
>
> Hmm ... putting the UI into a contrib module would neatly solve the
> problem for people who don't want the functionality. They just don't
> install the module. Another thought here is that it seems like we've
> been discussing multiple scenarios that don't necessarily all have the
> same best solution. Maybe we should think in terms of multiple contrib
> modules.
I'd certainly be happier with that- but it already exists, to some
extent, in the form of the pgAdmin "adminpack" contrib module. Perhaps
we should be working on improving what we've already got there rather
than adding something completely new..
Thanks,
Stephen
Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Tue, Aug 6, 2013 at 10:54:22AM -0400, Stephen Frost wrote: > * Andres Freund (andres@2ndquadrant.com) wrote: > > I don't think we're designing a feature that's supposed to be used under > > heavy concurrency here. If you have users/tools doing conflicting > > actions as superusers you need to solve that by social means, not by > > technical ones. > > If this actually gets used by puppet or another CMS, the chances of the > 'social means' being successful drop drastically. > > I agree that it doesn't need to work under heavy concurrency, but it > should do something sensible if it happens- perhaps even just throwing > an error if it can't acquire the lock immediately, warning the user that > some other process is trying to modify the config concurrently. My guess is that most users are going to do: SHOW log_min_duration_statement;SET log_min_duration_statement = 3; i.e. there isn't going to be any way to _lock_ that value between viewing it and setting it, and hence no way to warn users. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Bruce Momjian
Date:
On Tue, Aug 6, 2013 at 10:58:52AM -0400, Stephen Frost wrote: > * Tom Lane (tgl@sss.pgh.pa.us) wrote: > > Cédric Villemain <cedric@2ndquadrant.com> writes: > > > Maybe better to provide a contrib/ to modify config, then design what we can > > > achieve more with an ALTER SYSTEM command. > > > > Hmm ... putting the UI into a contrib module would neatly solve the > > problem for people who don't want the functionality. They just don't > > install the module. Another thought here is that it seems like we've > > been discussing multiple scenarios that don't necessarily all have the > > same best solution. Maybe we should think in terms of multiple contrib > > modules. > > I'd certainly be happier with that- but it already exists, to some > extent, in the form of the pgAdmin "adminpack" contrib module. Perhaps > we should be working on improving what we've already got there rather > than adding something completely new.. Yes, "adminpack" has mostly read/write file I/O operations --- it would need a higher-level API. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Boszormenyi Zoltan
Date:
2013-08-05 16:01 keltezéssel, Stephen Frost írta: > * Greg Stark (stark@mit.edu) wrote: >> On Fri, Aug 2, 2013 at 4:05 PM, Stephen Frost <sfrost@snowman.net> wrote: >>>> I'm not even clear we do want this in /etc since none of our GUC >>>> options are repeatable things like Apache virtual servers. It actually >>>> makes *more* sense for pg_hba than it does for gucs. I think we can >>>> assume that in the future we'll have something like it however. >>> I tend to agree with this also, though I can imagine wanting to separate >>> things in a conf.d directory ala exim's conf.d directories, to allow >>> tools like puppet to manage certain things environment-wide (perhaps >>> krb_server_keyfile) while other configuration options are managed >>> locally. >> Extensions are actually a pretty good argument for why conf.d in /etc >> (or wherever the non-auto-config is) is pretty important useful. >> That's the kind of thing conf.d directories are meant for. A user can >> install a package containing an extension and the extension would >> automatically drop in the config entries needed in that directory. > Agreed, though I think there should be a difference between "shared > library load" being added-to for extensions, and "random > extension-specific GUC".. Now that you mention "shared library load", it may be a good idea to add an "append-to-this-GUC" flag instead of overwriting the previous value. Two GUCs may make use of it: shared_preload_libraries and local_preload_libraries. It would make packagers' of extensions and DBA's lives easier. Best regards, Zoltán Böszörményi > > Thanks, > > Stephen -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Josh Berkus
Date:
On 08/06/2013 05:29 AM, Bruce Momjian wrote: > Let's look at the problems: > > * remote users can lock themselves out of the server > * interconnected GUC variables are complex to change > * need a way to disable this feature > > Given the above, I am not sure I see a way forward for ALTER SYSTEM SET. > One compromise that avoids the problems above would be to have a limited > feature called ALTER LOG SET that only controls logging parameters, but > I am not sure there is enough use-case there. > > This is not a zero-cost feature as we would be giving people _two_ ways > to configure Postgres, and two ways to find a fix if it is broken, so we > have to have a clear win to implement this. Also, if you have broken > the system via ALTER SYSTEM SET, you might need to edit flat files to > fix it, adding further complexity and limitations if someone only knows > the SQL method of configuration. Given that, and the problems above, I > reluctantly just don't see how this features moves us forward. Well said. I'd like to look at use cases, and let's see how ALTER SYSTEM SET addresses or doesn't address these use cases. I'd really like it if some other folks also posted use cases they know of. (1) Making is easier for GUIs to manage PostgreSQL settings. (2) Enabling DBAAS services to give users limited control over settings. (3) Making the "initial developer experience" better by not requiring hand-editing of configuration files to play with settings. (4) Making it easier to write automated testing scripts which test changes to settings. (5) Enabling new ways of writing Puppet/Chef/etc. scripts, which can check a setting before changing it. For the current patch, it fullfills these use cases as follows: (1) works, but is ALSO one of the critical problems with the feature. (2) does not work for this purpose. (3) works for this purpose. (4) works for this purpose. (5) works for this purpose, although it's unclear if it's actually wanted. Others? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Tue, Aug 6, 2013 at 06:34:35PM +0200, Boszormenyi Zoltan wrote: > 2013-08-05 16:01 keltezéssel, Stephen Frost írta: > >* Greg Stark (stark@mit.edu) wrote: > >>On Fri, Aug 2, 2013 at 4:05 PM, Stephen Frost <sfrost@snowman.net> wrote: > >>>>I'm not even clear we do want this in /etc since none of our GUC > >>>>options are repeatable things like Apache virtual servers. It actually > >>>>makes *more* sense for pg_hba than it does for gucs. I think we can > >>>>assume that in the future we'll have something like it however. > >>>I tend to agree with this also, though I can imagine wanting to separate > >>>things in a conf.d directory ala exim's conf.d directories, to allow > >>>tools like puppet to manage certain things environment-wide (perhaps > >>>krb_server_keyfile) while other configuration options are managed > >>>locally. > >>Extensions are actually a pretty good argument for why conf.d in /etc > >>(or wherever the non-auto-config is) is pretty important useful. > >>That's the kind of thing conf.d directories are meant for. A user can > >>install a package containing an extension and the extension would > >>automatically drop in the config entries needed in that directory. > >Agreed, though I think there should be a difference between "shared > >library load" being added-to for extensions, and "random > >extension-specific GUC".. > > Now that you mention "shared library load", it may be a good idea > to add an "append-to-this-GUC" flag instead of overwriting the > previous value. Two GUCs may make use of it: shared_preload_libraries > and local_preload_libraries. It would make packagers' of extensions > and DBA's lives easier. 'search_path' might also use it, though we might need append and prepend. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Bruce Momjian
Date:
On Tue, Aug 6, 2013 at 10:33:20AM -0700, Josh Berkus wrote: > On 08/06/2013 05:29 AM, Bruce Momjian wrote: > > Let's look at the problems: > > > > * remote users can lock themselves out of the server > > * interconnected GUC variables are complex to change > > * need a way to disable this feature > > > > Given the above, I am not sure I see a way forward for ALTER SYSTEM SET. > > One compromise that avoids the problems above would be to have a limited > > feature called ALTER LOG SET that only controls logging parameters, but > > I am not sure there is enough use-case there. > > > > This is not a zero-cost feature as we would be giving people _two_ ways > > to configure Postgres, and two ways to find a fix if it is broken, so we > > have to have a clear win to implement this. Also, if you have broken > > the system via ALTER SYSTEM SET, you might need to edit flat files to > > fix it, adding further complexity and limitations if someone only knows > > the SQL method of configuration. Given that, and the problems above, I > > reluctantly just don't see how this features moves us forward. > > Well said. I wish I had good news to "say well". ;-) > I'd like to look at use cases, and let's see how ALTER SYSTEM SET > addresses or doesn't address these use cases. I'd really like it if > some other folks also posted use cases they know of. > > (1) Making is easier for GUIs to manage PostgreSQL settings. One of the traps here is that while it makes it easier, it also could trap the user if they don't have the knowledge to fix a problem because would need to acquire the knowledge while they are trying to fix the problem, rather then while they are making the initial change. > (2) Enabling DBAAS services to give users limited control over settings. Right. This could be accomplished by giving users a function API for certain features, and then marking the function as SECURITY DEFINER. However, I am unclear how to do this in a generic way. Once neat idea would be to have a lookup table define which setting the administrator wishes to allow for non-superusers. If adminpack has an API to change postgresql.conf, it would be easy to create a function with SECURITY DEFINER permissions that SET lookup-allowed values in postgresql.conf. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Alvaro Herrera
Date:
Josh Berkus escribió: > (2) Enabling DBAAS services to give users limited control over settings. > (5) Enabling new ways of writing Puppet/Chef/etc. scripts, which can > check a setting before changing it. Surely these two cases are better covered by conf.d. For (2), DBaaS providers could offer pre-cooked snippets that are dropped in conf.d at the click of a button. For (5) it should be obvious that conf.d is better than ALTER SYSTEM. Not clear on (4); I think conf.d would also solve that one. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters
From
Claudio Freire
Date:
On Tue, Aug 6, 2013 at 3:31 PM, Bruce Momjian <bruce@momjian.us> wrote: >> I'd like to look at use cases, and let's see how ALTER SYSTEM SET >> addresses or doesn't address these use cases. I'd really like it if >> some other folks also posted use cases they know of. >> >> (1) Making is easier for GUIs to manage PostgreSQL settings. > > One of the traps here is that while it makes it easier, it also could > trap the user if they don't have the knowledge to fix a problem because > would need to acquire the knowledge while they are trying to fix the > problem, rather then while they are making the initial change. I think the more serious problem here is not about knowledge acquisition, but access to problem-fixing means. As was said some posts ago, if a DBA has access to a superuser account but not to server configuration files, he can lock himself out for good. No amount of knowledge will avail him/her. That's bad.
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Boszormenyi Zoltan
Date:
2013-08-06 19:41 keltezéssel, Bruce Momjian írta: > On Tue, Aug 6, 2013 at 06:34:35PM +0200, Boszormenyi Zoltan wrote: >> 2013-08-05 16:01 keltezéssel, Stephen Frost írta: >>> * Greg Stark (stark@mit.edu) wrote: >>>> On Fri, Aug 2, 2013 at 4:05 PM, Stephen Frost <sfrost@snowman.net> wrote: >>>>>> I'm not even clear we do want this in /etc since none of our GUC >>>>>> options are repeatable things like Apache virtual servers. It actually >>>>>> makes *more* sense for pg_hba than it does for gucs. I think we can >>>>>> assume that in the future we'll have something like it however. >>>>> I tend to agree with this also, though I can imagine wanting to separate >>>>> things in a conf.d directory ala exim's conf.d directories, to allow >>>>> tools like puppet to manage certain things environment-wide (perhaps >>>>> krb_server_keyfile) while other configuration options are managed >>>>> locally. >>>> Extensions are actually a pretty good argument for why conf.d in /etc >>>> (or wherever the non-auto-config is) is pretty important useful. >>>> That's the kind of thing conf.d directories are meant for. A user can >>>> install a package containing an extension and the extension would >>>> automatically drop in the config entries needed in that directory. >>> Agreed, though I think there should be a difference between "shared >>> library load" being added-to for extensions, and "random >>> extension-specific GUC".. >> Now that you mention "shared library load", it may be a good idea >> to add an "append-to-this-GUC" flag instead of overwriting the >> previous value. Two GUCs may make use of it: shared_preload_libraries >> and local_preload_libraries. It would make packagers' of extensions >> and DBA's lives easier. > 'search_path' might also use it, though we might need append and > prepend. Indeed. Although I was thinking along the lines of the GUC parser, so: shared_preload_library += 'some_new_lib' would append to the old value. Maybe "<+=" is intuitive enough for prepending, or someone may come up with a better idea. But the extra flag would still be needed to indicate the GUC is a list, so these new operators are usable on them and not on regular GUCs. Best regards, Zoltán Böszörményi -- ---------------------------------- Zoltán Böszörményi Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/
On 8/5/13 2:36 PM, Josh Berkus wrote:
> Most of our users not on Heroku are running with superuser as the app
> user now. Like, 95% of them based on my personal experience (because
> our object permissions management sucks).
My percentage wouldn't be nearly that high. 95% of database installs
done by developers? That I could believe. But setups done by
ex-{Oracle,DB2,Sybase...} DBAs avoid superuser rights for the
application, and that's around 1/3 of the systems I see. In general I
agree that there certainly are a lot of superuser only installs out
there, I just don't think it's quite as bad as you're painting it.
The important thing to realize here is that ALTER SYSTEM SET is a
convenience function. It’s useful to the extent that it makes people
feel more comfortable with changing things in the database. We will
never stop people from shooting themselves in the foot. And if the
barriers added for that purpose are too high, like disabling changes to
shared_buffers altogether, all you’ll do is make this so restricted that
it doesn’t satisfy anyone.
The original spec I outlined for how to implement this feature allowed
disabling the whole thing. Then it was just commenting out the
includedir directive from the postgresql.conf. Allowing people to
disable use of this feature in a managed configuration environment is
important. Beyond that, I don’t expect that we’ll ever make this foolproof.
After arguing out this topic in person with Stephen last night, I’ve
lumped ideas here into three major approaches that could be followed:
1) Don’t allow changing unsafe GUCs.
2) Provide a way for the server to start with bad setting and force the
administrator to fix the problem they introduced. Some sort of
maintenance mode that only allows superuser connections would force
someone to clean up this class of problem at next restart, while still
giving them enough access to run a new ALTER SYSTEM SET command.
3) Require extra syntax to modify an unsafe GUC.
As far as fixing the bad settings goes, there’s already code in initdb
to detect how high shared_buffers and max_connections can go. Any bogus
shared_buffers/max_connections value above the kernel limits could be
worked around with that approach.
Here’s how I would try and communicate that a change is unsafe, only
allowing it after proving user is paying some attention to the danger:
# ALTER SYSTEM SET shared_buffers = ‘8GB’;
NOTICE: Changing shared_buffers only takes effect after a server restart.
ERROR: Changing shared_buffers incorrectly can prevent the server from
starting normally. Use the FORCE option to modify the value anyway.
# ALTER SYSTEM SET shared_buffers = ‘8GB’ FORCE;
NOTICE: Changing shared_buffers only takes effect after a server restart.
ALTER SYSTEM
Will bad examples pop up in the Internet that just use FORCE all the
time? Sure they will, and people will cut and paste them without paying
attention. I don't see why that possibility has to block this feature
from being adopted though. That line of thinking leads toward removing
trust authentication, because that's similarly abused with cut and paste
tutorials.
--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
On Tue, Aug 6, 2013 at 09:24:47PM -0400, Greg Smith wrote: > # ALTER SYSTEM SET shared_buffers = ‘8GB’ FORCE; > NOTICE: Changing shared_buffers only takes effect after a server restart. > ALTER SYSTEM > > Will bad examples pop up in the Internet that just use FORCE all the > time? Sure they will, and people will cut and paste them without > paying attention. I don't see why that possibility has to block > this feature from being adopted though. That line of thinking leads > toward removing trust authentication, because that's similarly > abused with cut and paste tutorials. We already have six levels of GUC settings: postgresql.confuserdatabasesessionfunctionsubtransaction If we add ALTER SYSTEM SET and config.d, we would then have eight. ALTER SYSTEM SET seems to add an entirely new set of behaviors and complexity. Is that really what we want? If we do this, perhaps we should unconditionally just print the file name they have to delete to undo the operation in case the server doesn't start; I am unclear we can clearly identify all the GUC settings that could cause a server not to start. Also, I think we need a SHOW SYSTEM command so users can see their settings via SQL. FYI, ALTER SYSTEM SET is hitting the same problems we would have if pg_hba.conf were set in SQL and in flat files. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
> From: Greg Smith [mailto:greg@2ndQuadrant.com]
> Sent: Wednesday, August 07, 2013 6:55 AM
> To: Josh Berkus
> Cc: Stephen Frost; Bruce Momjian; Greg Stark; Andres Freund; Alvaro
> Herrera; Fujii Masao; Robert Haas; Amit Kapila; Dimitri Fontaine;
> pgsql-hackers@postgresql.org; Tom Lane
> Subject: Re: [HACKERS] Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER
> SYSTEM SET
>
> On 8/5/13 2:36 PM, Josh Berkus wrote:
> > Most of our users not on Heroku are running with superuser as the app
> > user now. Like, 95% of them based on my personal experience (because
> > our object permissions management sucks).
>
> My percentage wouldn't be nearly that high. 95% of database installs
> done by developers? That I could believe. But setups done by
> ex-{Oracle,DB2,Sybase...} DBAs avoid superuser rights for the
> application, and that's around 1/3 of the systems I see. In general I
> agree that there certainly are a lot of superuser only installs out
> there, I just don't think it's quite as bad as you're painting it.
>
> The important thing to realize here is that ALTER SYSTEM SET is a
> convenience function. It’s useful to the extent that it makes people
> feel more comfortable with changing things in the database. We will
> never stop people from shooting themselves in the foot. And if the
> barriers added for that purpose are too high, like disabling changes to
> shared_buffers altogether, all you’ll do is make this so restricted
> that
> it doesn’t satisfy anyone.
>
> The original spec I outlined for how to implement this feature allowed
> disabling the whole thing. Then it was just commenting out the
> includedir directive from the postgresql.conf. Allowing people to
> disable use of this feature in a managed configuration environment is
> important. Beyond that, I don’t expect that we’ll ever make this
> foolproof.
>
> After arguing out this topic in person with Stephen last night, I’ve
> lumped ideas here into three major approaches that could be followed:
> 1) Don’t allow changing unsafe GUCs.
>
> 2) Provide a way for the server to start with bad setting and force the
> administrator to fix the problem they introduced. Some sort of
> maintenance mode that only allows superuser connections would force
> someone to clean up this class of problem at next restart, while still
> giving them enough access to run a new ALTER SYSTEM SET command.
>
> 3) Require extra syntax to modify an unsafe GUC.
I also believe this is the right way to move forward and similar thoughts of mine are shared in mail below:
http://www.postgresql.org/message-id/00a301ce91a3$09226970$1b673c50$@kapila@huawei.com
I think for unsafe parameter's, for initial patch may be we can choose limited number of parameters.
> As far as fixing the bad settings goes, there’s already code in initdb
> to detect how high shared_buffers and max_connections can go. Any
> bogus
> shared_buffers/max_connections value above the kernel limits could be
> worked around with that approach.
>
> Here’s how I would try and communicate that a change is unsafe, only
> allowing it after proving user is paying some attention to the danger:
>
> # ALTER SYSTEM SET shared_buffers = ‘8GB’;
> NOTICE: Changing shared_buffers only takes effect after a server
> restart.
> ERROR: Changing shared_buffers incorrectly can prevent the server from
> starting normally. Use the FORCE option to modify the value anyway.
>
> # ALTER SYSTEM SET shared_buffers = ‘8GB’ FORCE;
> NOTICE: Changing shared_buffers only takes effect after a server
> restart.
> ALTER SYSTEM
>
> Will bad examples pop up in the Internet that just use FORCE all the
> time? Sure they will, and people will cut and paste them without
> paying
> attention. I don't see why that possibility has to block this feature
> from being adopted though. That line of thinking leads toward removing
> trust authentication, because that's similarly abused with cut and
> paste
> tutorials.
With Regards,
Amit Kapila.
On Wednesday, August 07, 2013 8:01 AM Bruce Momjian wrote: > On Tue, Aug 6, 2013 at 09:24:47PM -0400, Greg Smith wrote: > > # ALTER SYSTEM SET shared_buffers = ‘8GB’ FORCE; > > NOTICE: Changing shared_buffers only takes effect after a server > restart. > > ALTER SYSTEM > > > > Will bad examples pop up in the Internet that just use FORCE all the > > time? Sure they will, and people will cut and paste them without > > paying attention. I don't see why that possibility has to block > > this feature from being adopted though. That line of thinking leads > > toward removing trust authentication, because that's similarly > > abused with cut and paste tutorials. > > We already have six levels of GUC settings: > > postgresql.conf > user > database > session > function > subtransaction > > If we add ALTER SYSTEM SET and config.d, we would then have eight. > ALTER SYSTEM SET seems to add an entirely new set of behaviors and > complexity. Is that really what we want? > > If we do this, perhaps we should unconditionally just print the file > name they have to delete to undo the operation in case the server > doesn't start; I am unclear we can clearly identify all the GUC > settings that could cause a server not to start. > Also, I think we need > a SHOW SYSTEM command so users can see their settings via SQL. Although users can see the settings in pg_settings as it has sourcefile, but such a command can be useful. > FYI, ALTER SYSTEM SET is hitting the same problems we would have if > pg_hba.conf were set in SQL and in flat files. With Regards, Amit Kapila.
Bruce Momjian escribió: > We already have six levels of GUC settings: > > postgresql.conf > user > database > session > function > subtransaction > > If we add ALTER SYSTEM SET and config.d, we would then have eight. Actually, conf.d (not config.d) would just be reported as a file, and the "file" column in the pg_settings view would show which file it is. That's how it's done for postgresql.conf and any other file it includes, so that seems a pretty reasonable thing to me. > ALTER SYSTEM SET seems to add an entirely new set of behaviors and > complexity. Is that really what we want? I had imagined that ALTER SYSTEM SET would be represented in the same way as above, i.e. just be a different source file. But thinking more about it now, that doesn't make sense, because those files might be in a completely different base directory, and the file names shouldn't even be exposed to the user in the first place; so clearly ALTER SYSTEM should show up differently in pg_settings. Displaying each option's full path seems useful for troubleshooting, as you say: > If we do this, perhaps we should unconditionally just print the file > name they have to delete to undo the operation in case the server > doesn't start; However, bear in mind that if the DBA is administering a server through ALTER SYSTEM and they don't have shell access, they might just be screwed if they bollix the system and they lose access. Knowing what file you have to delete does you no good if you can't actually delete it. > I am unclear we can clearly identify all the GUC > settings that could cause a server not to start. Also, I think we need > a SHOW SYSTEM command so users can see their settings via SQL. Not sure about this. SHOW normally just displays the current value, nothing more. If you want more details, there's the pg_settings view with complete information. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Wed, Aug 7, 2013 at 10:59:05AM -0400, Alvaro Herrera wrote: > > If we do this, perhaps we should unconditionally just print the file > > name they have to delete to undo the operation in case the server > > doesn't start; > > However, bear in mind that if the DBA is administering a server through > ALTER SYSTEM and they don't have shell access, they might just be > screwed if they bollix the system and they lose access. Knowing what > file you have to delete does you no good if you can't actually delete it. My point was that if we tell them what file they have to edit to undo a problem, then at least then know that there is chance this will require shell access, and they can undo the change right there (with SET var = DEFAULT?) e.g.: WARNING: If this change prevents the server from starting, you will need to edit postgresql.conf.auto to undo thechange > > I am unclear we can clearly identify all the GUC > > settings that could cause a server not to start. Also, I think we need > > a SHOW SYSTEM command so users can see their settings via SQL. > > Not sure about this. SHOW normally just displays the current value, > nothing more. If you want more details, there's the pg_settings view > with complete information. The issue is that if we give users the ability to set values via SQL, we should at least document how they can view the things they set via SQL, and pg_settings doesn't do that because it just shows the _active_ value, which might have overridden the SQL-set value. Right now we can see settings in postgresql.conf by viewing the file, and ALTER USER/DATABASE via system tables. Having no API to view SQL-set values except viewing a flat file seems counter-productive. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Monday, August 05, 2013 10:12 PM Josh Berkus wrote: > All, > > To limit the argument here, let's please not argue about things which > people already agree on. So: > > We seem to have consensus around: > > A) the inadvisability of storing GUCs in a system catalog. > > B) the utility of a conf.d in /etc/ which may have nothing to do with > ALTER SYSTEM SET > > C) that any file for ALTER SYSTEM SET go in $PGDATA somewhere. > > What we are still arguing about: Summarization of discussion, please add if I have missed or misunderstood any conclusive point: > D) one-big-file vs. file-per-setting Here I think there are no points against one-big-file and not much in favor offile-per-setting. I believe there is enough support for one-big-file that there is no need to change it to file-per-setting. > E) whether "unsafe" settings or "restart" settings should be allowed in > ALTER SYSTEM SET. a. Settings should continue to be in flat file. This will allow user to change settings incase anythinggoes wrong. b. Few unsafe settings shouldn't be allowed to change, but there can be an option (new keyword FORCE)to change if user wants it. Initial set of parameters suggested by Stephen Frost: http://www.postgresql.org/message-id/20130802145442.GB2706@tamriel.snowman.net > F) whether admins need the ability to disable ALTER SYSTEM SET. a. UI into contrib module, people who don't want doesn'tinclude it b. Have an include file mechanism, so that user can comment the include in postgresql.conf and disableit. Some new requirements 1. Show System command to separately show the values set by Alter System 2. append-to-this-GUC flag instead of overwriting the previous value. With Regards, Amit Kapila.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Sun, Aug 4, 2013 at 4:26 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote: > Alvaro Herrera <alvherre@2ndquadrant.com> writes: >> I remind you that event triggers are not fired for global objects >> such as databases and roles. Do you intend to lift that restriction? > > That's not on my TODO list for 9.4. My understanding about implementing > that is: > > - we agree that it would be nice to have, > - it requires a separate *shared* catalog for event triggers. > > What I'm yet unsure about is that there's a consensus that the use cases > are worthy of a new shared catalog in the system. Also I didn't look how > hard it is to actually provide for it. A new shared catalog wouldn't actually help, because the actual procedure to be run has to live in pg_proc, which is not shared. And that has references to all sorts of other things (like pg_language) that aren't shared either. I think the question isn't really a technical one so much as one of policy. We could quite easily allow event triggers on shared objects.The reason I suggested that we NOT allow that is becausethen operations on those objects would behave differently depending on which database you've attached. If you've attached a database with an event trigger, you get the special behavior; otherwise, you don't. I feared creating user confusion, there, and the use cases seemed marginal anyway. But if there's a sufficient consensus that such a thing is useful and non-confusing, I'll give way. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Robert Haas (robertmhaas@gmail.com) wrote:
> On Sun, Aug 4, 2013 at 4:26 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
> > What I'm yet unsure about is that there's a consensus that the use cases
> > are worthy of a new shared catalog in the system. Also I didn't look how
> > hard it is to actually provide for it.
>
> A new shared catalog wouldn't actually help, because the actual
> procedure to be run has to live in pg_proc, which is not shared. And
> that has references to all sorts of other things (like pg_language)
> that aren't shared either.
A shared catalog which defined which *database* to run the trigger in,
with a way to fire off a new backend worker in that database and tell it
to run the trigger, might be interesting and would deal with the issue
that the trigger would behave differently depending on the database
connected to. That would bring along other issues, of course, but it
seemed an interesting enough idea to mention.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Stephen Frost <sfrost@snowman.net> writes: > A shared catalog which defined which *database* to run the trigger in, > with a way to fire off a new backend worker in that database and tell it > to run the trigger, might be interesting and would deal with the issue > that the trigger would behave differently depending on the database > connected to. That would bring along other issues, of course, but it > seemed an interesting enough idea to mention. Yeah, I like that approach. The only drawback is that it requires having PLproxy in core first, or something like Foreign Functions or something. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
David Fetter
Date:
On Fri, Aug 09, 2013 at 03:08:45PM +0200, Dimitri Fontaine wrote: > Stephen Frost <sfrost@snowman.net> writes: > > A shared catalog which defined which *database* to run the trigger > > in, with a way to fire off a new backend worker in that database > > and tell it to run the trigger, might be interesting and would > > deal with the issue that the trigger would behave differently > > depending on the database connected to. That would bring along > > other issues, of course, but it seemed an interesting enough idea > > to mention. > > Yeah, I like that approach. The only drawback is that it requires > having PLproxy in core first, or something like Foreign Functions or > something. SQL/MED does define such an API. Whether we find it useful enough to make it the default way of doing things is a separate matter. I'll do some research. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Fri, Aug 9, 2013 at 8:44 AM, Stephen Frost <sfrost@snowman.net> wrote: > * Robert Haas (robertmhaas@gmail.com) wrote: >> On Sun, Aug 4, 2013 at 4:26 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote: >> > What I'm yet unsure about is that there's a consensus that the use cases >> > are worthy of a new shared catalog in the system. Also I didn't look how >> > hard it is to actually provide for it. >> >> A new shared catalog wouldn't actually help, because the actual >> procedure to be run has to live in pg_proc, which is not shared. And >> that has references to all sorts of other things (like pg_language) >> that aren't shared either. > > A shared catalog which defined which *database* to run the trigger in, > with a way to fire off a new backend worker in that database and tell it > to run the trigger, might be interesting and would deal with the issue > that the trigger would behave differently depending on the database > connected to. That would bring along other issues, of course, but it > seemed an interesting enough idea to mention. Eh, maybe. I'm not sure there's enough use case for that to justify the amount of infrastructure it would require. I'm happy with the recent enhancements to background workers, but there's an awful lot of ground to cover between that and what you're proposing. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Robert Haas escribió: > On Fri, Aug 9, 2013 at 8:44 AM, Stephen Frost <sfrost@snowman.net> wrote: > > A shared catalog which defined which *database* to run the trigger in, > > with a way to fire off a new backend worker in that database and tell it > > to run the trigger, might be interesting and would deal with the issue > > that the trigger would behave differently depending on the database > > connected to. That would bring along other issues, of course, but it > > seemed an interesting enough idea to mention. > > Eh, maybe. I'm not sure there's enough use case for that to justify > the amount of infrastructure it would require. I'm happy with the > recent enhancements to background workers, but there's an awful lot of > ground to cover between that and what you're proposing. Yeah, agreed. There's a lot of infrastructure required for this; it seems hard to justify it only to support disabling ALTER SYSTEM. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Fri, Aug 16, 2013 at 11:12 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > Robert Haas escribió: >> On Fri, Aug 9, 2013 at 8:44 AM, Stephen Frost <sfrost@snowman.net> wrote: >> > A shared catalog which defined which *database* to run the trigger in, >> > with a way to fire off a new backend worker in that database and tell it >> > to run the trigger, might be interesting and would deal with the issue >> > that the trigger would behave differently depending on the database >> > connected to. That would bring along other issues, of course, but it >> > seemed an interesting enough idea to mention. >> >> Eh, maybe. I'm not sure there's enough use case for that to justify >> the amount of infrastructure it would require. I'm happy with the >> recent enhancements to background workers, but there's an awful lot of >> ground to cover between that and what you're proposing. > > Yeah, agreed. There's a lot of infrastructure required for this; it > seems hard to justify it only to support disabling ALTER SYSTEM. I think disabling ALTER SYSTEM can be more easily supported if we choose one of below options: a. UI into contrib module, people who don't want doesn't include it b. Have an include file mechanism, so that user can comment the include in postgresql.conf and disable it. If I remember correctly, earlier you mentioned that by default auto file should be parsed after postgresql.conf, but how about reverting to previous mechanism of "include" such that if the file is mentioned in include then it will be parsed, else will be ignored. I think this can be reasonable way to disable. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Robert Haas escribió:
> > On Fri, Aug 9, 2013 at 8:44 AM, Stephen Frost <sfrost@snowman.net> wrote:
> > > A shared catalog which defined which *database* to run the trigger in,
> > > with a way to fire off a new backend worker in that database and tell it
> > > to run the trigger, might be interesting and would deal with the issue
> > > that the trigger would behave differently depending on the database
> > > connected to. That would bring along other issues, of course, but it
> > > seemed an interesting enough idea to mention.
> >
> > Eh, maybe. I'm not sure there's enough use case for that to justify
> > the amount of infrastructure it would require. I'm happy with the
> > recent enhancements to background workers, but there's an awful lot of
> > ground to cover between that and what you're proposing.
>
> Yeah, agreed. There's a lot of infrastructure required for this; it
> seems hard to justify it only to support disabling ALTER SYSTEM.
This would be support for generic triggers on shared catalog updates,
which would certainly extend beyond just ALTER SYSTEM and into ALTER
DATABASE and ALTER ROLE..
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila16@gmail.com) wrote:
> I think disabling ALTER SYSTEM can be more easily supported if we
> choose one of below options:
> a. UI into contrib module, people who don't want doesn't include it
> b. Have an include file mechanism, so that user can comment the
> include in postgresql.conf and disable it.
Either of these would be good options, imv. In my new world order, it'd
be a '#include' in the postgresql.conf- which is commented out by
default.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tue, Aug 20, 2013 at 5:49 AM, Stephen Frost <sfrost@snowman.net> wrote: > * Amit Kapila (amit.kapila16@gmail.com) wrote: >> I think disabling ALTER SYSTEM can be more easily supported if we >> choose one of below options: >> a. UI into contrib module, people who don't want doesn't include it >> b. Have an include file mechanism, so that user can comment the >> include in postgresql.conf and disable it. > > Either of these would be good options, imv. In my new world order, it'd > be a '#include' in the postgresql.conf- which is commented out by > default. If both are okay, then I would like to go with second option (include file mechanism). I think by default, we should allow auto file to be included and if user faces any problem or otherwise, then he can decide to disable it. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila16@gmail.com) wrote:
> If both are okay, then I would like to go with second option (include
> file mechanism).
> I think by default, we should allow auto file to be included and if
> user faces any problem or otherwise,
> then he can decide to disable it.
If it's enabled by default, then we need to ship an 'auto' file which is
empty by default... I don't particularly like that (implies that an
empty file is acceptable and not an error condition, which it otherwise
probably would be...). I'm still not a big fan of enabling it by default
tho.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tue, Aug 20, 2013 at 8:27 AM, Stephen Frost <sfrost@snowman.net> wrote: > * Amit Kapila (amit.kapila16@gmail.com) wrote: >> If both are okay, then I would like to go with second option (include >> file mechanism). >> I think by default, we should allow auto file to be included and if >> user faces any problem or otherwise, >> then he can decide to disable it. > > If it's enabled by default, then we need to ship an 'auto' file which is > empty by default... initdb will create the empty auto file. If we don't enable by default, then if user uses ALTER SYSTEM and do sighup/restart, changed config parameter values will not come into affect until he manually enables it by removing '#' from '#include'. > I don't particularly like that (implies that an > empty file is acceptable and not an error condition, which it otherwise > probably would be...). I'm still not a big fan of enabling it by default > tho. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila16@gmail.com) wrote:
> initdb will create the empty auto file. If we don't enable by default,
> then if user uses
> ALTER SYSTEM and do sighup/restart, changed config parameter values
> will not come into affect
> until he manually enables it by removing '#' from '#include'.
That's a pretty simple thing to detect and return an ERROR case on
though.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tue, Aug 20, 2013 at 8:43 AM, Stephen Frost <sfrost@snowman.net> wrote: > * Amit Kapila (amit.kapila16@gmail.com) wrote: >> initdb will create the empty auto file. If we don't enable by default, >> then if user uses >> ALTER SYSTEM and do sighup/restart, changed config parameter values >> will not come into affect >> until he manually enables it by removing '#' from '#include'. > > That's a pretty simple thing to detect and return an ERROR case on > though. Do you mean to say, that when user uses ALTER SYSTEM and include is disabled, then we can give ERROR in ALTER SYSTEM, that to reflect the values, he need to enable it? Actually in previousversion of patch, there was WARNING on such a condition. I agree that it can be detected and we issue Warning orError, but I think it will be an un-necessary inconvenience for user. However if you still vote for disable by default, I will modify the patch that way only. The other part which needs suggestions is about un-safe parameters: I think for first version of feature, we can decide on some minimum list of parameters that will be considere un-safe and user will not be allowed to modify them with ALTER SYSTEM. To start with we can consider below 4 parameters in that category and later on keep adding more parameters: data_directory, listen_addresses, port, shared_buffers. These are from the list you provided at link: http://www.postgresql.org/message-id/20130802145442.GB2706@tamriel.snowman.net I have no strong reason why only these 4 parameters, we can choose other parameters as well. The only thing is I want to make a base for this feature and then later we can enhance it as per below points: a. add more parameters to un-safe category b. user can view non-modifiable parameters (may be in pg_settings) c. some way such that if user wants to take risk of server not getting started, he should allow to modify such parameters (may be with additional syntax in ALTER SYSTEM .. FORCE as suggested by Greg Smith). With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila16@gmail.com) wrote:
> Do you mean to say, that when user uses ALTER SYSTEM and include is
> disabled, then we can give ERROR in
> ALTER SYSTEM, that to reflect the values, he need to enable it?
> Actually in previous version of patch, there was WARNING on such a condition.
> I agree that it can be detected and we issue Warning or Error, but
> I think it will be an un-necessary inconvenience
> for user. However if you still vote for disable by default, I will
> modify the patch that way only.
It hardly seems unnecessary.. Otherwise we accept a command that won't
actually do anything, yes? We usually notice or warn, at least, on such
cases (see similar things around GRANT/REVOKE...).
> The other part which needs suggestions is about un-safe parameters:
> I think for first version of feature, we can decide on some minimum
> list of parameters that will be considere un-safe and user will not be
> allowed to modify them with ALTER SYSTEM.
I do feel like it's easier to add things to the 'allowed' set later,
should we see some good justification for it, than to disallow things
already allowed.
> To start with we can consider below 4 parameters in that category and
> later on keep adding more parameters:
> data_directory, listen_addresses, port, shared_buffers.
What about include directives?
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tue, Aug 20, 2013 at 9:33 AM, Stephen Frost <sfrost@snowman.net> wrote: > * Amit Kapila (amit.kapila16@gmail.com) wrote: >> Do you mean to say, that when user uses ALTER SYSTEM and include is >> disabled, then we can give ERROR in >> ALTER SYSTEM, that to reflect the values, he need to enable it? >> Actually in previous version of patch, there was WARNING on such a condition. >> I agree that it can be detected and we issue Warning or Error, but >> I think it will be an un-necessary inconvenience >> for user. However if you still vote for disable by default, I will >> modify the patch that way only. > > It hardly seems unnecessary.. Otherwise we accept a command that won't > actually do anything, yes? Okay, let us decide how things will be if we disable by default: 1. initdb won't create any empty auto file, rather thefile will be created first time user uses ALTER SYSTEM. 2. Alter system should issue warning, if it detects that feature is disabled (for this we need to error detection code during parsing of postgresql.conf as it was previously) 3. postgresql.confwill contain include directive in below form: #include = 'postgresql.auto.conf' Whenever user wantsto use Alter System, he needs to enable it after first time using ALTER SYSTEM. One question here, if somebody just enables it without using ALTER SYSTEM, should it throw any error on not finding auto file or just ignore it? > We usually notice or warn, at least, on such > cases (see similar things around GRANT/REVOKE...). > >> The other part which needs suggestions is about un-safe parameters: >> I think for first version of feature, we can decide on some minimum >> list of parameters that will be considere un-safe and user will not be >> allowed to modify them with ALTER SYSTEM. > > I do feel like it's easier to add things to the 'allowed' set later, > should we see some good justification for it, than to disallow things > already allowed. Here the only consideration in my mind is about question of usefulness of this feature and getting agreement on long list of un-safe parameters. As we are already proposing/planingto give FORCE option with which user can change config parameters, the question of usefulness can be answered with that, but how about other? >> To start with we can consider below 4 parameters in that category and >> later on keep adding more parameters: >> data_directory, listen_addresses, port, shared_buffers. > > What about include directives? Sorry, I didn't get which parameters you are referring by include directives? With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Amit Kapila escribió: > 3. postgresql.conf will contain include directive in below form: > #include = 'postgresql.auto.conf' > Whenever user wants to use Alter System, he needs to enable it > after first time using ALTER SYSTEM. This seems wrong to me. If the auto file is read by an include line in postgresql.conf, what is its priority w.r.t. files placed in an hypothetical conf.d directory? Hopefully snippets put in conf.d/ by puppet/chef will override the settings in postgresql.conf (i.e. conf.d/ should be processed after postgresql.conf, not before); and hopefully ALTER SYSTEM will in turn override conf.d. I see no way to have ALTER SYSTEM handled by an include line, yet still have it override conf.d. If we want to make ALTER SYSTEM disable-able from postgresql.conf, I think it should be an explicit option, something like enable_alter_system = on or something like that. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tue, Aug 20, 2013 at 10:26 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > Amit Kapila escribió: > >> 3. postgresql.conf will contain include directive in below form: >> #include = 'postgresql.auto.conf' >> Whenever user wants to use Alter System, he needs to enable it >> after first time using ALTER SYSTEM. > > This seems wrong to me. If the auto file is read by an include line in > postgresql.conf, what is its priority w.r.t. files placed in an > hypothetical conf.d directory? Hopefully snippets put in conf.d/ by > puppet/chef will override the settings in postgresql.conf (i.e. conf.d/ > should be processed after postgresql.conf, not before); and hopefully > ALTER SYSTEM will in turn override conf.d. I see no way to have ALTER > SYSTEM handled by an include line, yet still have it override conf.d. I think the behavior of overriding can be controlled by location of include directive of conf.d directory or include location of auto file whichever is later will override. User can change the priority by changing location of include directive. > If we want to make ALTER SYSTEM disable-able from postgresql.conf, I > think it should be an explicit option, something like > enable_alter_system = on > or something like that. I think this would be an easier way to handle disabling, but the only question is whether this feature deserves a new config parameter? With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila16@gmail.com) wrote:
> Okay, let us decide how things will be if we disable by default:
> 1. initdb won't create any empty auto file, rather the file will be
> created first time user uses ALTER SYSTEM.
I'm not particularly set on this.. Why not create the file initially?
> 2. Alter system should issue warning, if it detects that feature is
> disabled (for this we need to error detection code
> during parsing of postgresql.conf as it was previously)
I agree that it should complain through a warning or perhaps an error
(to be honest, I like error more, but there may be good reasons against
that).
> 3. postgresql.conf will contain include directive in below form:
> #include = 'postgresql.auto.conf'
> Whenever user wants to use Alter System, he needs to enable it
> after first time using ALTER SYSTEM.
That I don't like.. You should be able to enable it and have things
"work" without having to run ALTER SYSTEM first..
> One question here, if somebody just enables it without using ALTER
> SYSTEM, should it throw any error on not finding auto file or just
> ignore it?
I'd prefer that it error if it can't find an included file.
> > What about include directives?
>
> Sorry, I didn't get which parameters you are referring by include directives?
Literally, the above 'include' in postgresql.conf. Would that only be
dealt with as a parse-time thing, or should it be more like a GUC which
could then be set through ALTER SYSTEM?
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Amit Kapila escribió:
> > 3. postgresql.conf will contain include directive in below form:
> > #include = 'postgresql.auto.conf'
> > Whenever user wants to use Alter System, he needs to enable it
> > after first time using ALTER SYSTEM.
>
> This seems wrong to me. If the auto file is read by an include line in
> postgresql.conf, what is its priority w.r.t. files placed in an
> hypothetical conf.d directory?
This could be handled by a simple ordering with "last one wins". I
don't particularly like that though.. My gut feeling is that I'd like
something to complain if there's more than one value set for the same
GUC..
> Hopefully snippets put in conf.d/ by
> puppet/chef will override the settings in postgresql.conf (i.e. conf.d/
> should be processed after postgresql.conf, not before); and hopefully
> ALTER SYSTEM will in turn override conf.d. I see no way to have ALTER
> SYSTEM handled by an include line, yet still have it override conf.d.
With includedir and include directives, and postgresql.conf setting a
defined ordering, with last-wins, you could simply have the 'includedir'
for conf.d come before the 'include' for auto.conf.
> If we want to make ALTER SYSTEM disable-able from postgresql.conf, I
> think it should be an explicit option, something like
> enable_alter_system = on
> or something like that.
I really don't like this approach- I'd much rather we have a general
include mechanism than special "alter-system" GUCs.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tue, Aug 20, 2013 at 6:21 PM, Stephen Frost <sfrost@snowman.net> wrote: > * Amit Kapila (amit.kapila16@gmail.com) wrote: >> Okay, let us decide how things will be if we disable by default: >> 1. initdb won't create any empty auto file, rather the file will be >> created first time user uses ALTER SYSTEM. > > I'm not particularly set on this.. Why not create the file initially? If by default this feature needs to be disabled,then it is not must to have at initdb time. OTOH if we want to enable it by default, then we need to get this created at initdb time else it will give error during system startup (an error can occur during startup when it will parse postgresql.conf and didn't find the file mentioned in include directive). Also you mentioned below line upthread which I understood as you don't like idea of creating empty file at initdb time: "If it's enabled by default, then we need to ship an 'auto' filewhich is empty by default... I don't particularly like that" >> 2. Alter system should issue warning, if it detects that feature is >> disabled (for this we need to error detection code >> during parsing of postgresql.conf as it was previously) > > I agree that it should complain through a warning or perhaps an error > (to be honest, I like error more, but there may be good reasons against > that). > >> 3. postgresql.conf will contain include directive in below form: >> #include = 'postgresql.auto.conf' >> Whenever user wants to use Alter System, he needs to enable it >> after first time using ALTER SYSTEM. > > That I don't like.. You should be able to enable it and have things > "work" without having to run ALTER SYSTEM first.. This was actually linked to point 1 mentioned above, if we create empty file at initdb time, then there should not be any problem. One other option to disable as suggested by Alvaro is have another config parameter to enable/disable Alter System, if we can do that way, the solution will be much neater. >> One question here, if somebody just enables it without using ALTER >> SYSTEM, should it throw any error on not finding auto file or just >> ignore it? > > I'd prefer that it error if it can't find an included file. > >> > What about include directives? >> >> Sorry, I didn't get which parameters you are referring by include directives? > > Literally, the above 'include' in postgresql.conf. Would that only be > dealt with as a parse-time thing, or should it be more like a GUC which > could then be set through ALTER SYSTEM? I think if we choose to use include directive as a way to enable/disable this feature, it will not be good to allow change of this parameter with Alter System. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila16@gmail.com) wrote:
> On Tue, Aug 20, 2013 at 6:21 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > I'm not particularly set on this.. Why not create the file initially?
> If by default this feature needs to be disabled, then it is not
> must to have at initdb time.
I don't see how the two are related. You could never use two-phase
commit (could even disable it), yet we still create things in the data
directory to support it. Having a file in the data directory which
isn't used by default isn't that big a deal, imv.
> Also you mentioned below line upthread which I understood as you
> don't like idea of creating empty file at initdb
> time:
> "If it's enabled by default, then we need to ship an 'auto' file which is
> empty by default... I don't particularly like that"
What I didn't like was having an empty file be accepted as 'valid', but
you need to have some kind of bootstrap mechanism. Telling users "run
this command and then ignore the warning" is certainly bad. A better
option might be to have a *non-empty* auto.conf be generated, where all
it has in it is some kind of identifier, perhaps even just
"enable_alter_system = on" which we could then key off of to see if
ALTER SYSTEM has been set up to be allowed or not.
> I think if we choose to use include directive as a way to
> enable/disable this feature, it will not be good to allow change of
> this parameter with Alter System.
I agree, but then we need to add it to the list of things ALTER SYSTEM
can't modify (if the include is implemented as a GUC, that is; I've not
looked).
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tue, Aug 20, 2013 at 6:55 PM, Stephen Frost <sfrost@snowman.net> wrote: > * Amit Kapila (amit.kapila16@gmail.com) wrote: >> On Tue, Aug 20, 2013 at 6:21 PM, Stephen Frost <sfrost@snowman.net> wrote: >> > I'm not particularly set on this.. Why not create the file initially? >> If by default this feature needs to be disabled, then it is not >> must to have at initdb time. > > I don't see how the two are related. You could never use two-phase > commit (could even disable it), yet we still create things in the data > directory to support it. Having a file in the data directory which > isn't used by default isn't that big a deal, imv. Yes, it can be done, what I wanted to say it is not a "must", rather a good to have thing. >> Also you mentioned below line upthread which I understood as you >> don't like idea of creating empty file at initdb >> time: >> "If it's enabled by default, then we need to ship an 'auto' file which is >> empty by default... I don't particularly like that" > > What I didn't like was having an empty file be accepted as 'valid', but > you need to have some kind of bootstrap mechanism. Telling users "run > this command and then ignore the warning" is certainly bad. A better > option might be to have a *non-empty* auto.conf be generated, where all > it has in it is some kind of identifier, perhaps even just > "enable_alter_system = on" which we could then key off of to see if > ALTER SYSTEM has been set up to be allowed or not. Wouldn't it be complicated to handle this way rather then by having include directive. If include directive is enabled then the autofile will be read else no need to read it. If we want to have separate identifier to judge ALTER SYSTEM is enable or not, then it is better to go with a new GUC. >> I think if we choose to use include directive as a way to >> enable/disable this feature, it will not be good to allow change of >> this parameter with Alter System. > > I agree, but then we need to add it to the list of things ALTER SYSTEM > can't modify (if the include is implemented as a GUC, that is; I've not > looked). I have checked and it seems to be just parse time thing. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila16@gmail.com) wrote:
> Wouldn't it be complicated to handle this way rather then by having
> include directive.
You misunderstood. I was suggesting a setup along these lines:
postgresql.conf: # include = 'auto.conf' # Disabled by default
auto.conf: #################################################### # MANAGED BY ALTER SYSTEM -- DO NOT MODIFY BY HAND
####################################################
# auto.conf not processed unless included by postgresql.conf # If included by postgresql.conf, then this will turn on
the# ALTER SYSTEM command. enable_alter_system = on
Which would give us the ability to independently turn on/off ALTER
SYSTEM from including auto.conf.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tue, Aug 20, 2013 at 7:51 PM, Stephen Frost <sfrost@snowman.net> wrote: > * Amit Kapila (amit.kapila16@gmail.com) wrote: >> Wouldn't it be complicated to handle this way rather then by having >> include directive. > > You misunderstood. I was suggesting a setup along these lines: > > postgresql.conf: > # include = 'auto.conf' # Disabled by default > > auto.conf: > #################################################### > # MANAGED BY ALTER SYSTEM -- DO NOT MODIFY BY HAND > #################################################### > > # auto.conf not processed unless included by postgresql.conf > # If included by postgresql.conf, then this will turn on the > # ALTER SYSTEM command. > enable_alter_system = on > > Which would give us the ability to independently turn on/off ALTER > SYSTEM from including auto.conf. So let me try to explain what I understood from above: 1. enable_alter_system a new GUC whose default value =off. 2. Alter System will check this variable and return error (not allowed), if this parameter is off. 3. Now if user enables include directive in postgresql.conf, it will enable Alter System as value of enable_alter_system is on. 4. User can run Alter System command to disable Alter System "enable_alter_system = off". Now even though include directive is enabled, but new Alter System commands will not work, however existing parameter's take into effect on restart/sighup. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila16@gmail.com) wrote:
> So let me try to explain what I understood from above:
>
> 1. enable_alter_system a new GUC whose default value =off.
> 2. Alter System will check this variable and return error (not
> allowed), if this parameter is off.
> 3. Now if user enables include directive in postgresql.conf, it will
> enable Alter System as value of
> enable_alter_system is on.
> 4. User can run Alter System command to disable Alter System
> "enable_alter_system = off".
> Now even though include directive is enabled, but new Alter System
> commands will not work, however
> existing parameter's take into effect on restart/sighup.
Yes. Not sure that it'd be terribly likely for a user to do that, but
if they do it, so be it.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Stephen Frost escribió: > * Alvaro Herrera (alvherre@2ndquadrant.com) wrote: > > Hopefully snippets put in conf.d/ by > > puppet/chef will override the settings in postgresql.conf (i.e. conf.d/ > > should be processed after postgresql.conf, not before); and hopefully > > ALTER SYSTEM will in turn override conf.d. I see no way to have ALTER > > SYSTEM handled by an include line, yet still have it override conf.d. > > With includedir and include directives, and postgresql.conf setting a > defined ordering, with last-wins, you could simply have the 'includedir' > for conf.d come before the 'include' for auto.conf. Uh, I was thinking that conf.d would be read by the system automatically, not because of an includedir line in postgresql.conf. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Stephen Frost escribió:
> > With includedir and include directives, and postgresql.conf setting a
> > defined ordering, with last-wins, you could simply have the 'includedir'
> > for conf.d come before the 'include' for auto.conf.
>
> Uh, I was thinking that conf.d would be read by the system
> automatically, not because of an includedir line in postgresql.conf.
I'd much rather have an includedir directive than some hard-coded or
command-line option to read the directory.. The directory should live
in /etc/postgresql/X.Y/cluster/ on at least Debian derivatives..
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Stephen Frost escribió: > * Amit Kapila (amit.kapila16@gmail.com) wrote: > > So let me try to explain what I understood from above: > > > > 1. enable_alter_system a new GUC whose default value =off. > > 2. Alter System will check this variable and return error (not > > allowed), if this parameter is off. > > 3. Now if user enables include directive in postgresql.conf, it will > > enable Alter System as value of > > enable_alter_system is on. > > 4. User can run Alter System command to disable Alter System > > "enable_alter_system = off". > > Now even though include directive is enabled, but new Alter System > > commands will not work, however > > existing parameter's take into effect on restart/sighup. > > Yes. Not sure that it'd be terribly likely for a user to do that, but > if they do it, so be it. With this design, if you put enable_alter_system=off in auto.conf, there is no way for the user to enable alter system again short of editing a file in the data directory. I think this is one of the things that was "forbidden" by policy; only files in the config directory needs to be edited. What I was proposing upthread is that enable_alter_system=off/on would be present in postgresql.conf, and there is no include line for auto.conf. That way, if the user wishes to enable/disable the feature, they need to edit postgresql.conf to do so. ALTER SYSTEM doesn't offer a way to disable itself. If ALTER SYSTEM is disabled via enable_alter_system=off in postgresql.conf, the settings in auto.conf are not read. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Stephen Frost escribió: > * Alvaro Herrera (alvherre@2ndquadrant.com) wrote: > > Stephen Frost escribió: > > > With includedir and include directives, and postgresql.conf setting a > > > defined ordering, with last-wins, you could simply have the 'includedir' > > > for conf.d come before the 'include' for auto.conf. > > > > Uh, I was thinking that conf.d would be read by the system > > automatically, not because of an includedir line in postgresql.conf. > > I'd much rather have an includedir directive than some hard-coded or > command-line option to read the directory.. The directory should live > in /etc/postgresql/X.Y/cluster/ on at least Debian derivatives.. The conf.d/ path would be relative to postgresql.conf, so there's no need for Debian to patch anything. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> What I was proposing upthread is that enable_alter_system=off/on would
> be present in postgresql.conf, and there is no include line for
> auto.conf. That way, if the user wishes to enable/disable the feature,
> they need to edit postgresql.conf to do so. ALTER SYSTEM doesn't offer
> a way to disable itself. If ALTER SYSTEM is disabled via
> enable_alter_system=off in postgresql.conf, the settings in auto.conf
> are not read.
Personally, I'd get rid of any explicit "enable_alter_system" variable,
and instead have an include directive for auto.conf. The functionality
you describe above is then available by commenting or uncommenting the
directive, plus the user has the option to decide where to put the
directive (and thus control which settings can or can't be overridden).
Anything else seems like it's going to be less flexible or else a lot
more complicated.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> With this design, if you put enable_alter_system=off in auto.conf, there
> is no way for the user to enable alter system again short of editing a
> file in the data directory. I think this is one of the things that was
> "forbidden" by policy; only files in the config directory needs to be
> edited.
If you edit it by hand to begin with (setting that parameter to 'off')
then it's reasonable that you may have to edit it by hand again to fix
it. If we don't want people to "lock themselves out" by using ALTER
SYSTEM to turn it off, then we just disallow that.
> What I was proposing upthread is that enable_alter_system=off/on would
> be present in postgresql.conf, and there is no include line for
> auto.conf.
I really think that's a terrible approach, to be honest. I want to see
an 'include' line in postgresql.conf for auto.conf, so the hapless
sysadmin who is trying to figure out what the crazy DBA did has some
clue what to look for. "enable_alter_system" doesn't tell him diddly
about an 'auto.conf' file which is included in the system config.
> That way, if the user wishes to enable/disable the feature,
> they need to edit postgresql.conf to do so. ALTER SYSTEM doesn't offer
> a way to disable itself.
We can simply disallow ALTER SYSTEM from modifying enable_alter_system;
that strikes me as a reasonable thing to do anyway. What I find a bit
more worrying is what happens if they decide to put
enable_alter_system=off into the postgresql.conf but keep the 'include'
line for auto.conf.. Which goes right back to the question that I had
before around if we want to complain when the same GUC is seen multiple
times during parsing. It seems like there's no hope for it, given the
way this has been designed, because you *must* set certain parameters
and so you can't simply have them commented out, but those are likely to
be parameters which DBAs will want to change through ALTER SYSTEM.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Stephen Frost escribió:
> > * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> > > Stephen Frost escribió:
> > > > With includedir and include directives, and postgresql.conf setting a
> > > > defined ordering, with last-wins, you could simply have the 'includedir'
> > > > for conf.d come before the 'include' for auto.conf.
> > >
> > > Uh, I was thinking that conf.d would be read by the system
> > > automatically, not because of an includedir line in postgresql.conf.
> >
> > I'd much rather have an includedir directive than some hard-coded or
> > command-line option to read the directory.. The directory should live
> > in /etc/postgresql/X.Y/cluster/ on at least Debian derivatives..
>
> The conf.d/ path would be relative to postgresql.conf, so there's no
> need for Debian to patch anything.
Uhhh, I really don't see that working, at all...
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Stephen Frost <sfrost@snowman.net> writes:
> * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
>> The conf.d/ path would be relative to postgresql.conf, so there's no
>> need for Debian to patch anything.
> Uhhh, I really don't see that working, at all...
Why not? conf.d is for installable files, AIUI. What we need to
be writable is auto.conf, but I thought we'd agreed that would live
inside $PGDATA.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> >> The conf.d/ path would be relative to postgresql.conf, so there's no
> >> need for Debian to patch anything.
>
> > Uhhh, I really don't see that working, at all...
>
> Why not? conf.d is for installable files, AIUI. What we need to
> be writable is auto.conf, but I thought we'd agreed that would live
> inside $PGDATA.
I agree that auto.conf should live in $PGDATA, but I really don't like
the idea of conf.d being relative to some other existing file. It
should be included through an 'includedir' option, not just picked up as
some magic directory name, and therefore consider the current
arrangement of parameters in Debian:
data_directory = '/var/lib/postgresql/9.2/main'
hba_file = '/etc/postgresql/9.2/main/pg_hba.conf'
ident_file = '/etc/postgresql/9.2/main/pg_ident.conf'
and postgres is started like so:
/usr/lib/postgresql/9.2/bin/postgres -D /var/lib/postgresql/9.2/main -c
config_file=/etc/postgresql/9.2/main/postgresql.conf
With the proposed include line for auto.conf, which lives in $PGDATA,
we'd have:
include 'auto.conf'
Would we then have
includedir 'conf.d'
which is relative to postgresql.conf instead?
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Stephen Frost escribió: > * Alvaro Herrera (alvherre@2ndquadrant.com) wrote: > > Stephen Frost escribió: > > > I'd much rather have an includedir directive than some hard-coded or > > > command-line option to read the directory.. The directory should live > > > in /etc/postgresql/X.Y/cluster/ on at least Debian derivatives.. > > > > The conf.d/ path would be relative to postgresql.conf, so there's no > > need for Debian to patch anything. > > Uhhh, I really don't see that working, at all... I don't understand why not. Care to explain? -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Stephen Frost escribió:
> > * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> > > Stephen Frost escribió:
>
> > > > I'd much rather have an includedir directive than some hard-coded or
> > > > command-line option to read the directory.. The directory should live
> > > > in /etc/postgresql/X.Y/cluster/ on at least Debian derivatives..
> > >
> > > The conf.d/ path would be relative to postgresql.conf, so there's no
> > > need for Debian to patch anything.
> >
> > Uhhh, I really don't see that working, at all...
>
> I don't understand why not. Care to explain?
Tried to in my other mail, but let me also point out that we
("PGDG"/Upstream) don't "own" the directory in which postgresql.conf
lives. At least on Debian and relatives, that directory isn't under
$PGDATA and it already has other files in it beyond postgresql.conf or
even the other PostgreSQL config files of hba and ident. Here's the
default directory setup on Debian for /etc/postgresql/9.2/main/:
-rw-r--r-- 1 postgres postgres 316 Jun 29 22:07 environment
-rw-r--r-- 1 postgres postgres 143 Jun 29 22:07 pg_ctl.conf
-rw-r----- 1 postgres postgres 4649 Jun 29 22:07 pg_hba.conf
-rw-r----- 1 postgres postgres 1636 Jun 29 22:07 pg_ident.conf
-rw-r--r-- 1 postgres postgres 19770 Jun 29 22:07 postgresql.conf
-rw-r--r-- 1 postgres postgres 378 Jun 29 22:07 start.conf
There's three other files there and some sysadmins may have already
created their own 'conf.d' directory, perhaps to use for building the
postgresql.conf file or similar. We must have a way to disable the
conf.d option and a way to name it something other than 'conf.d', imv.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Stephen Frost escribió: > I agree that auto.conf should live in $PGDATA, but I really don't like > the idea of conf.d being relative to some other existing file. It > should be included through an 'includedir' option, not just picked up as > some magic directory name, and therefore consider the current > arrangement of parameters in Debian: > > data_directory = '/var/lib/postgresql/9.2/main' > hba_file = '/etc/postgresql/9.2/main/pg_hba.conf' > ident_file = '/etc/postgresql/9.2/main/pg_ident.conf' > > and postgres is started like so: > > /usr/lib/postgresql/9.2/bin/postgres -D /var/lib/postgresql/9.2/main -c config_file=/etc/postgresql/9.2/main/postgresql.conf > > With the proposed include line for auto.conf, which lives in $PGDATA, > we'd have: > > include 'auto.conf' > > Would we then have > > includedir 'conf.d' > > which is relative to postgresql.conf instead? Well, all the relative paths in include/includedir directives would be relative to the directory specified by the -c config_file param, which makes perfect sense. So the conf.d would work fine in your example. The only problem I see in your snippet above is the "include auto.conf" line, which doesn't make any sense because that file would not be found. Hence my proposal that the file ought to be read automatically, not via an include line. Sadly I don't think we cannot just make it an absolute path, in case the data dir is moved or whatever; the only choice I see would be to have something like include-data 'auto.conf' or something like that which tells the system that the file is not in the config dir but in the data dir instead. A nearby comment could let the user know about this file being in the data directory instead of the config directory. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Stephen Frost escribió:
> Tried to in my other mail,
Yep, got it and replied, thanks.
> but let me also point out that we
> ("PGDG"/Upstream) don't "own" the directory in which postgresql.conf
> lives. At least on Debian and relatives, that directory isn't under
> $PGDATA and it already has other files in it beyond postgresql.conf or
> even the other PostgreSQL config files of hba and ident. Here's the
> default directory setup on Debian for /etc/postgresql/9.2/main/:
>
> -rw-r--r-- 1 postgres postgres 316 Jun 29 22:07 environment
> -rw-r--r-- 1 postgres postgres 143 Jun 29 22:07 pg_ctl.conf
> -rw-r----- 1 postgres postgres 4649 Jun 29 22:07 pg_hba.conf
> -rw-r----- 1 postgres postgres 1636 Jun 29 22:07 pg_ident.conf
> -rw-r--r-- 1 postgres postgres 19770 Jun 29 22:07 postgresql.conf
> -rw-r--r-- 1 postgres postgres 378 Jun 29 22:07 start.conf
>
> There's three other files there and some sysadmins may have already
> created their own 'conf.d' directory, perhaps to use for building the
> postgresql.conf file or similar. We must have a way to disable the
> conf.d option and a way to name it something other than 'conf.d', imv.
Uhm. I find it hard to care much for this position. Surely config
files are not migrated automatically from one major version to the next,
so if somebody created a 9.3/main/conf.d directory, they will have to
change it when they migrate to 9.4.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Well, all the relative paths in include/includedir directives would be
> relative to the directory specified by the -c config_file param, which
> makes perfect sense. So the conf.d would work fine in your example.
Why would include/includedir directives be relative to where the
'config_file' GUC is set to instead of relative to where all the other
GUCs in postgresql.conf are relative to? That is a recipe for
confusion, imv.
Of course, the current situation is quite terrible anyway, imv. Having
the GUCs be relative to whereever the user happens to run pg_ctl from is
pretty ugly- not to mention that the commented out 'defaults' won't
actually work if you uncomment them because the *actual* default/unset
value *is* in the data directory. I'm starting to wish for a way to do
variable substitution inside postgresql.conf, so we could have defaults
that would actually work if uncommented (eg: $DataDir/pg_hba.conf).
That would help with auto.conf also.
> The only problem I see in your snippet above is the "include auto.conf"
> line, which doesn't make any sense because that file would not be found.
To be honest, I was considering 'include' to be relative to the data
directory and handled similar to how we process recovery.conf, but as we
consider paths in postgresql.conf to be relative to $PWD, that's not
ideal because it'd be different from the rest of the file references.
In any case, while the current situation sucks, I don't think we can go
changing how relative files are treated in postgresql.conf, nor should
we make the way a path is processed change depending on which GUC is
being set.
While I really like the 'include auto.conf' style, I'm starting to think
it may not be workable after all. Another thing to consider is if the
user decides to change that include line.. What happens when the DBA
tries to do a 'ALTER SYSTEM'? It'd still use the hard-coded auto.conf
file and happily update it, I imagine, but it won't actually get
included...
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Stephen Frost escribió: > * Alvaro Herrera (alvherre@2ndquadrant.com) wrote: > > Well, all the relative paths in include/includedir directives would be > > relative to the directory specified by the -c config_file param, which > > makes perfect sense. So the conf.d would work fine in your example. > > Why would include/includedir directives be relative to where the > 'config_file' GUC is set to instead of relative to where all the other > GUCs in postgresql.conf are relative to? That is a recipe for > confusion, imv. > > Of course, the current situation is quite terrible anyway, imv. Having > the GUCs be relative to whereever the user happens to run pg_ctl from is > pretty ugly- not to mention that the commented out 'defaults' won't > actually work if you uncomment them because the *actual* default/unset > value *is* in the data directory. Uh? See the docs: http://www.postgresql.org/docs/devel/static/config-setting.html#CONFIG-INCLUDES " ... the postgresql.conf file can contain include directives, ... If the file name is not an absolute path, it is takenas relative to the directory containing the referencing configuration file." > I'm starting to wish for a way to do > variable substitution inside postgresql.conf, so we could have defaults > that would actually work if uncommented (eg: $DataDir/pg_hba.conf). > > That would help with auto.conf also. Grumble. I don't think we need this. At least, not for ALTER SYSTEM or conf.d. > > The only problem I see in your snippet above is the "include auto.conf" > > line, which doesn't make any sense because that file would not be found. > > To be honest, I was considering 'include' to be relative to the data > directory and handled similar to how we process recovery.conf, Well, recovery.conf is a special case that doesn't follow to normal rules. > but as we > consider paths in postgresql.conf to be relative to $PWD, that's not > ideal because it'd be different from the rest of the file references. I don't know where you got that idea from, but it's wrong. > While I really like the 'include auto.conf' style, I'm starting to think > it may not be workable after all. Another thing to consider is if the > user decides to change that include line.. What happens when the DBA > tries to do a 'ALTER SYSTEM'? It'd still use the hard-coded auto.conf > file and happily update it, I imagine, but it won't actually get > included... Well, this whole line of discussion started because I objected to the whole code path that was trying to detect whether auto.conf had been parsed, and raised a warning if ALTER SYSTEM was executed and the file wasn't parsed. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Stephen Frost escribi�:
>> While I really like the 'include auto.conf' style, I'm starting to think
>> it may not be workable after all. Another thing to consider is if the
>> user decides to change that include line.. What happens when the DBA
>> tries to do a 'ALTER SYSTEM'? It'd still use the hard-coded auto.conf
>> file and happily update it, I imagine, but it won't actually get
>> included...
> Well, this whole line of discussion started because I objected to the
> whole code path that was trying to detect whether auto.conf had been
> parsed, and raised a warning if ALTER SYSTEM was executed and the file
> wasn't parsed.
I really, really don't think that we should be trying to detect or prevent
any such thing. If the user breaks it like that, he gets to keep both
pieces --- and who's to say it's broken, anyway? Disabling ALTER SYSTEM
might have been exactly his intent.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Alvaro Herrera (alvherre@2ndquadrant.com) wrote: > Stephen Frost escribió: > > Of course, the current situation is quite terrible anyway, imv. Having > > the GUCs be relative to whereever the user happens to run pg_ctl from is > > pretty ugly- not to mention that the commented out 'defaults' won't > > actually work if you uncomment them because the *actual* default/unset > > value *is* in the data directory. > > Uh? See the docs: > http://www.postgresql.org/docs/devel/static/config-setting.html#CONFIG-INCLUDES > > " ... the postgresql.conf file can contain include directives, ... > If the file name is not an absolute path, it is taken as relative to > the directory containing the referencing configuration file." And what about http://www.postgresql.org/docs/9.3/static/runtime-config-file-locations.html ... ? "When setting any of these parameters, a relative path will be interpreted with respect to the directory in which postgresis started." > > I'm starting to wish for a way to do > > variable substitution inside postgresql.conf, so we could have defaults > > that would actually work if uncommented (eg: $DataDir/pg_hba.conf). > > > > That would help with auto.conf also. > > Grumble. I don't think we need this. At least, not for ALTER SYSTEM or > conf.d. imv, it would be handy. Along with a $ConfDir which is the postgresql.conf location- it would certainly make things clearer about what files are being included from where. > > To be honest, I was considering 'include' to be relative to the data > > directory and handled similar to how we process recovery.conf, > > Well, recovery.conf is a special case that doesn't follow to normal > rules. I don't see why it should be. > > but as we > > consider paths in postgresql.conf to be relative to $PWD, that's not > > ideal because it'd be different from the rest of the file references. > > I don't know where you got that idea from, but it's wrong. Not sure which you're referring to, but see above from the docs? Also, please tias.. I was amazed that we use $PWD also, but we do.. > Well, this whole line of discussion started because I objected to the > whole code path that was trying to detect whether auto.conf had been > parsed, and raised a warning if ALTER SYSTEM was executed and the file > wasn't parsed. I like the idea that we complain if ALTER SYSTEM ends up being idempotent.. Not sure how far we should take that, but accepting commands which end up doing nothing is bad, imv. Thanks, Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Stephen Frost escribió: > And what about > > http://www.postgresql.org/docs/9.3/static/runtime-config-file-locations.html > > ... ? > > "When setting any of these parameters, a relative path will be > interpreted with respect to the directory in which postgres is > started." That's talking about the data_directory and the various foo_file settings, though; it doesn't apply to the include settings. Note especially that config_file says it can only be set on the postgres command line. (I think a saner definition would have been to state that relative paths are not allowed in the command line. But that ship has already sailed. And relative paths seem useful in the config file; and maintaining the distinction that they are allowed within the config file but not in the command line might be awkward.) (Uhm, when a command line contains stuff, is the stuff "in" the command line or "on" it?) -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Alvaro Herrera (alvherre@2ndquadrant.com) wrote: > Stephen Frost escribió: > > http://www.postgresql.org/docs/9.3/static/runtime-config-file-locations.html > > That's talking about the data_directory and the various foo_file > settings, though; it doesn't apply to the include settings. Right- that's what I'm bitching about. We have various references to file locations, with defined handling of relative locations, and the 'include' system completely ignores that and instead invents its own making the result a confusing mess. > Note > especially that config_file says it can only be set on the postgres > command line. (I think a saner definition would have been to state that > relative paths are not allowed in the command line. But that ship has > already sailed. And relative paths seem useful in the config file; and > maintaining the distinction that they are allowed within the config file > but not in the command line might be awkward.) Relative paths based on $PWD are useful? Really? Not on my systems anyway.. > (Uhm, when a command line contains stuff, is the stuff "in" the command > line or "on" it?) A just question- I vote 'on'. :) Thanks, Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Martijn van Oosterhout
Date:
On Tue, Aug 20, 2013 at 11:23:06AM -0400, Stephen Frost wrote: > > What I was proposing upthread is that enable_alter_system=off/on would > > be present in postgresql.conf, and there is no include line for > > auto.conf. > > I really think that's a terrible approach, to be honest. I want to see > an 'include' line in postgresql.conf for auto.conf, so the hapless > sysadmin who is trying to figure out what the crazy DBA did has some > clue what to look for. "enable_alter_system" doesn't tell him diddly > about an 'auto.conf' file which is included in the system config. ISTM you want some kind of hybrid setting like: #include_system auto.conf which simultaneously does three things: 1. Sets the enable_alter_system flag 2. Indicates the file to use 3. Indicates the priority of the setting re other settings. Comment it out, ALTER SYSTEM stop working. Put it back and it's immediately clear what it means. And the user can control where the settings go. Syntax is a bit fugly though. Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > He who writes carelessly confesses thereby at the very outset that he does > not attach much importance to his own thoughts. -- Arthur Schopenhauer
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Tue, Aug 20, 2013 at 08:38:52AM +0530, Amit Kapila wrote: > On Tue, Aug 20, 2013 at 8:27 AM, Stephen Frost <sfrost@snowman.net> wrote: > > * Amit Kapila (amit.kapila16@gmail.com) wrote: > >> If both are okay, then I would like to go with second option (include > >> file mechanism). > >> I think by default, we should allow auto file to be included and if > >> user faces any problem or otherwise, > >> then he can decide to disable it. > > > > If it's enabled by default, then we need to ship an 'auto' file which is > > empty by default... > > initdb will create the empty auto file. If we don't enable by default, > then if user uses > ALTER SYSTEM and do sighup/restart, changed config parameter values > will not come into affect > until he manually enables it by removing '#' from '#include'. Just a heads-up, but a lot of C language folks are going to look at: #include abc and think that is enabled. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tue, Aug 20, 2013 at 10:02 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > Stephen Frost escribió: >> * Alvaro Herrera (alvherre@2ndquadrant.com) wrote: >> > Well, all the relative paths in include/includedir directives would be >> > relative to the directory specified by the -c config_file param, which >> > makes perfect sense. So the conf.d would work fine in your example. >> >> Why would include/includedir directives be relative to where the >> 'config_file' GUC is set to instead of relative to where all the other >> GUCs in postgresql.conf are relative to? That is a recipe for >> confusion, imv. >> >> Of course, the current situation is quite terrible anyway, imv. Having >> the GUCs be relative to whereever the user happens to run pg_ctl from is >> pretty ugly- not to mention that the commented out 'defaults' won't >> actually work if you uncomment them because the *actual* default/unset >> value *is* in the data directory. > > Uh? See the docs: > http://www.postgresql.org/docs/devel/static/config-setting.html#CONFIG-INCLUDES > > " ... the postgresql.conf file can contain include directives, ... > If the file name is not an absolute path, it is taken as relative to > the directory containing the referencing configuration file." You are right and I have checked code as well, it works in above way for include directives. Now the question I have in mind is that even if we can't directly use include directive to enable/disable Alter System and reading of auto file, how would a new GUC enable_alter_system can solve all the things. It can allow/disallow Alter System command, but how about reading of auto file? If we directly read auto file without considering enable_alter_system, it can lead to chaos due to some un-safe values, on the other side if we want to consider enable_alter_system before reading file, it can complicate the ProcessConfigFile() code. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Wed, Aug 21, 2013 at 7:29 AM, Bruce Momjian <bruce@momjian.us> wrote: > On Tue, Aug 20, 2013 at 08:38:52AM +0530, Amit Kapila wrote: >> On Tue, Aug 20, 2013 at 8:27 AM, Stephen Frost <sfrost@snowman.net> wrote: >> > * Amit Kapila (amit.kapila16@gmail.com) wrote: >> >> If both are okay, then I would like to go with second option (include >> >> file mechanism). >> >> I think by default, we should allow auto file to be included and if >> >> user faces any problem or otherwise, >> >> then he can decide to disable it. >> > >> > If it's enabled by default, then we need to ship an 'auto' file which is >> > empty by default... >> >> initdb will create the empty auto file. If we don't enable by default, >> then if user uses >> ALTER SYSTEM and do sighup/restart, changed config parameter values >> will not come into affect >> until he manually enables it by removing '#' from '#include'. > > Just a heads-up, but a lot of C language folks are going to look at: > > #include abc > > and think that is enabled. True, but generally for conf and script file, symbol '#' is treated as commented portion of content. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
Martijn,
* Martijn van Oosterhout (kleptog@svana.org) wrote:
> ISTM you want some kind of hybrid setting like:
>
> #include_system auto.conf
>
> which simultaneously does three things:
>
> 1. Sets the enable_alter_system flag
> 2. Indicates the file to use
> 3. Indicates the priority of the setting re other settings.
>
> Comment it out, ALTER SYSTEM stop working. Put it back and it's
> immediately clear what it means. And the user can control where the
> settings go.
Yeah, that's certainly an interesting idea. I might call it
'auto_conf_file auto.conf' to avoid the '#include' concern and to
perhaps clarify that it's more than just a regular 'include'.
> Syntax is a bit fugly though.
Agreed.
Thanks,
Stephen (who is still unhappy about the GUC-specific handling for relative paths in postgresql.conf)
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Wed, Aug 21, 2013 at 8:22 PM, Stephen Frost <sfrost@snowman.net> wrote: > Martijn, > > * Martijn van Oosterhout (kleptog@svana.org) wrote: >> ISTM you want some kind of hybrid setting like: >> >> #include_system auto.conf >> >> which simultaneously does three things: >> >> 1. Sets the enable_alter_system flag >> 2. Indicates the file to use >> 3. Indicates the priority of the setting re other settings. >> >> Comment it out, ALTER SYSTEM stop working. Put it back and it's >> immediately clear what it means. And the user can control where the >> settings go. > > Yeah, that's certainly an interesting idea. I might call it > 'auto_conf_file auto.conf' to avoid the '#include' concern and to > perhaps clarify that it's more than just a regular 'include'. This can resolve the problem of whether to read auto file rather cleanly, so the idea is: Enable/Disable reading of auto file ----------------------------------------------------- a. Have a new include in postresql.conf #include_auto_conf_file postgresql.auto.conf as it is a special include, wecan read this file relative to data directory. Enable/Disable Alter System command ----------------------------------------------------------- This can be achieved in 3 ways: a. Check before executing Alter System if include directive is disabled, then just issue a warning to user and proceed with command. b. Check before executing Alter System if include directive is disabled, then just issue an error and stop. c. Have a new guc enable_alter_system which will behave as described in my previous mail and below: >1. enable_alter_system a new GUC whose default value =off. > 2. Alter System will checkthis variable and return error (not > allowed), if this parameter is off. > 3. Now if user enables include directivein postgresql.conf, it will > enable Alter System as value of enable_alter_system is on. > 4. User can run AlterSystem command to disable Alter System > "enable_alter_system = off". > Now even though include directive is enabled,but new Alter System > commands will not work, however > existing parameter's take into effect on restart/sighup. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila16@gmail.com) wrote:
> This can resolve the problem of whether to read auto file rather
> cleanly, so the idea is:
>
> Enable/Disable reading of auto file
> -----------------------------------------------------
> a. Have a new include in postresql.conf
> #include_auto_conf_file postgresql.auto.conf
> as it is a special include, we can read this file relative to data
> directory.
>
> Enable/Disable Alter System command
> -----------------------------------------------------------
> This can be achieved in 3 ways:
> a. Check before executing Alter System if include directive is
> disabled, then just issue a warning to user and proceed with command.
> b. Check before executing Alter System if include directive is
> disabled, then just issue an error and stop.
It doesn't make sense for it to be a 'warning' with this- the
parameter specifies the file to use. If you don't know what file to
use, how you can possibly do anything but return an error?
Note that I *like* that about this approach.
There are a few other considerations with this-
- What should the default be? (Still thinking 'off' myself)
- What happens if the user specifies 'postgresql.conf'? I'm thinking we would disallow such insanity (as that's what
itis, imv..) by having an identifier in the file that this is the PG "auto conf" file.
- Should we have such an identifier in auto.conf to indicate that we created it, to prevent the user from setting it to
somethingthey shouldn't?
- What's the "bootstrap" mode; iow, if a user enables the option but the file doesn't exist, what do we do? With this
approach,I'd be inclined to say we simply create it and put the marker to indicate it's "our" file.
- Should we allow it to be outside of the data dir? We could simply log an error and ignore the parameter if it's more
thana simple filename.
There are probably other considerations also..
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Thu, Aug 22, 2013 at 08:36:37AM -0400, Stephen Frost wrote: > * Amit Kapila (amit.kapila16@gmail.com) wrote: > > This can resolve the problem of whether to read auto file rather > > cleanly, so the idea is: > > > > Enable/Disable reading of auto file > > ----------------------------------------------------- > > a. Have a new include in postresql.conf > > #include_auto_conf_file postgresql.auto.conf > > as it is a special include, we can read this file relative to data > > directory. The big advantage of using 'include_auto_conf_file' and not simply 'include' is that we can issue an error from ALTER SYSTEM SET if that is not set. > > Enable/Disable Alter System command > > ----------------------------------------------------------- > > This can be achieved in 3 ways: > > a. Check before executing Alter System if include directive is > > disabled, then just issue a warning to user and proceed with command. > > b. Check before executing Alter System if include directive is > > disabled, then just issue an error and stop. > > It doesn't make sense for it to be a 'warning' with this- the > parameter specifies the file to use. If you don't know what file to > use, how you can possibly do anything but return an error? Agreed. No sense in allowing users to add things to the 'auto' file when the auto file is inactive. > Note that I *like* that about this approach. > > There are a few other considerations with this- > > - What should the default be? (Still thinking 'off' myself) Probably, but we might need to wait until we have a final API for a decision on that. > - What happens if the user specifies 'postgresql.conf'? I'm thinking we > would disallow such insanity (as that's what it is, imv..) by having > an identifier in the file that this is the PG "auto conf" file. I am thinking they can't include a value equal to 'config_file', which is normally postgresql.conf. I am not a big fan of looking for special text in files. This might be complex to check, though, because of path changes --- we might just disallow the basement from matching the basename of config_file. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Thu, Aug 22, 2013 at 6:06 PM, Stephen Frost <sfrost@snowman.net> wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> This can resolve the problem of whether to read auto file rather
>> cleanly, so the idea is:
>>
>> Enable/Disable reading of auto file
>> -----------------------------------------------------
>> a. Have a new include in postresql.conf
>> #include_auto_conf_file postgresql.auto.conf
>> as it is a special include, we can read this file relative to data
>> directory.
>>
>> Enable/Disable Alter System command
>> -----------------------------------------------------------
>> This can be achieved in 3 ways:
>> a. Check before executing Alter System if include directive is
>> disabled, then just issue a warning to user and proceed with command.
>> b. Check before executing Alter System if include directive is
>> disabled, then just issue an error and stop.
>
> It doesn't make sense for it to be a 'warning' with this- the
> parameter specifies the file to use. If you don't know what file to
> use, how you can possibly do anything but return an error?
As the file and location are fixed, we can go-ahead and write to
it, but I think now we are deciding if someone disables include dir, then we can just disable Alter
System, so it is better to return error in such situation.
> Note that I *like* that about this approach.
>
> There are a few other considerations with this-
>
> - What should the default be? (Still thinking 'off' myself) default 'off' is a safe option, as it won't allow
usersto make
any change to parameter values until/unless they read from manual, how to use it and what can go wrong, on the
other side it will be bit hassle for user to use this command. I think 'on' would be better.
> - What happens if the user specifies 'postgresql.conf'? I'm thinking we
> would disallow such insanity (as that's what it is, imv..) by having
> an identifier in the file that this is the PG "auto conf" file. I think we can detect by name and give
error.
> - Should we have such an identifier in auto.conf to indicate that we
> created it, to prevent the user from setting it to something they
> shouldn't? I think if user plays with this file manually, it can lead to
problems, that's why earlier we have decided to keep a note on top of file which will indicate, do not
edit this file manually. I believe that should be sufficient.
> - What's the "bootstrap" mode; iow, if a user enables the option but the
> file doesn't exist, what do we do? With this approach, I'd be
> inclined to say we simply create it and put the marker to indicate
> it's "our" file.
Alter System will create the file if doesn't exist.
> - Should we allow it to be outside of the data dir? We could simply log
> an error and ignore the parameter if it's more than a simple filename.
This should be an error, the file location and name will be fixed.
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Thu, Aug 22, 2013 at 9:34 PM, Bruce Momjian <bruce@momjian.us> wrote: > On Thu, Aug 22, 2013 at 08:36:37AM -0400, Stephen Frost wrote: >> * Amit Kapila (amit.kapila16@gmail.com) wrote: >> > This can resolve the problem of whether to read auto file rather >> > cleanly, so the idea is: >> > >> > Enable/Disable reading of auto file >> > ----------------------------------------------------- >> > a. Have a new include in postresql.conf >> > #include_auto_conf_file postgresql.auto.conf >> > as it is a special include, we can read this file relative to data >> > directory. > > The big advantage of using 'include_auto_conf_file' and not simply > 'include' is that we can issue an error from ALTER SYSTEM SET if that is > not set. > >> > Enable/Disable Alter System command >> > ----------------------------------------------------------- >> > This can be achieved in 3 ways: >> > a. Check before executing Alter System if include directive is >> > disabled, then just issue a warning to user and proceed with command. >> > b. Check before executing Alter System if include directive is >> > disabled, then just issue an error and stop. >> >> It doesn't make sense for it to be a 'warning' with this- the >> parameter specifies the file to use. If you don't know what file to >> use, how you can possibly do anything but return an error? > > Agreed. No sense in allowing users to add things to the 'auto' file > when the auto file is inactive. > >> Note that I *like* that about this approach. >> >> There are a few other considerations with this- >> >> - What should the default be? (Still thinking 'off' myself) > > Probably, but we might need to wait until we have a final API for a > decision on that. > >> - What happens if the user specifies 'postgresql.conf'? I'm thinking we >> would disallow such insanity (as that's what it is, imv..) by having >> an identifier in the file that this is the PG "auto conf" file. > > I am thinking they can't include a value equal to 'config_file', which > is normally postgresql.conf. I am not a big fan of looking for special > text in files. This might be complex to check, though, because of path > changes --- we might just disallow the basement from matching the > basename of config_file. Right, I also think that as file and location are fixed, so it can be detected with name. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Amit Kapila (amit.kapila16@gmail.com) wrote:
> On Thu, Aug 22, 2013 at 6:06 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > * Amit Kapila (amit.kapila16@gmail.com) wrote:
> >> Enable/Disable reading of auto file
> >> -----------------------------------------------------
> >> a. Have a new include in postresql.conf
> >> #include_auto_conf_file postgresql.auto.conf
> >> as it is a special include, we can read this file relative to data
> >> directory.
> >>
> >> Enable/Disable Alter System command
> >> -----------------------------------------------------------
> >> This can be achieved in 3 ways:
> >> a. Check before executing Alter System if include directive is
> >> disabled, then just issue a warning to user and proceed with command.
> >> b. Check before executing Alter System if include directive is
> >> disabled, then just issue an error and stop.
> >
> > It doesn't make sense for it to be a 'warning' with this- the
> > parameter specifies the file to use. If you don't know what file to
> > use, how you can possibly do anything but return an error?
>
> As the file and location are fixed, we can go-ahead and write to
> it, but I think now we are deciding
> if someone disables include dir, then we can just disable Alter
> System, so it is better to return error in such
> situation.
It wouldn't be fixed with this approach.
> > Note that I *like* that about this approach.
> >
> > There are a few other considerations with this-
> >
> > - What should the default be? (Still thinking 'off' myself)
> default 'off' is a safe option, as it won't allow users to make
> any change to parameter values until/unless they
> read from manual, how to use it and what can go wrong, on the
> other side it will be bit hassle for user to use this
> command. I think 'on' would be better.
Yeah, no, I still think 'off' would be best for this particular option.
> > - What happens if the user specifies 'postgresql.conf'? I'm thinking we
> > would disallow such insanity (as that's what it is, imv..) by having
> > an identifier in the file that this is the PG "auto conf" file.
> I think we can detect by name and give error.
> > - Should we have such an identifier in auto.conf to indicate that we
> > created it, to prevent the user from setting it to something they
> > shouldn't?
> I think if user plays with this file manually, it can lead to
> problems, that's why earlier we have
> decided to keep a note on top of file which will indicate, do not
> edit this file manually.
> I believe that should be sufficient.
I agree that having such a disclaimer at the top of the file is a good
idea. I'm not completely convinced that's sufficient but it's certainly
better than nothing.
> > - What's the "bootstrap" mode; iow, if a user enables the option but the
> > file doesn't exist, what do we do? With this approach, I'd be
> > inclined to say we simply create it and put the marker to indicate
> > it's "our" file.
>
> Alter System will create the file if doesn't exist.
... Only if it's enabled though.
> > - Should we allow it to be outside of the data dir? We could simply log
> > an error and ignore the parameter if it's more than a simple filename.
>
> This should be an error, the file location and name will be fixed.
Not with this proposal... If it's fixed then it makes no sense to make it
look like it can be modified.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Fri, Aug 23, 2013 at 6:01 PM, Stephen Frost <sfrost@snowman.net> wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> On Thu, Aug 22, 2013 at 6:06 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> > * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> >> Enable/Disable reading of auto file
>> >> -----------------------------------------------------
>> >> a. Have a new include in postresql.conf
>> >> #include_auto_conf_file postgresql.auto.conf
>> >> as it is a special include, we can read this file relative to data
>> >> directory.
>> >>
>> >> Enable/Disable Alter System command
>> >> -----------------------------------------------------------
>> >> This can be achieved in 3 ways:
>> >> a. Check before executing Alter System if include directive is
>> >> disabled, then just issue a warning to user and proceed with command.
>> >> b. Check before executing Alter System if include directive is
>> >> disabled, then just issue an error and stop.
>> >
>> > It doesn't make sense for it to be a 'warning' with this- the
>> > parameter specifies the file to use. If you don't know what file to
>> > use, how you can possibly do anything but return an error?
>>
>> As the file and location are fixed, we can go-ahead and write to
>> it, but I think now we are deciding
>> if someone disables include dir, then we can just disable Alter
>> System, so it is better to return error in such
>> situation.
>
> It wouldn't be fixed with this approach.
>
>> > Note that I *like* that about this approach.
>> >
>> > There are a few other considerations with this-
>> >
>> > - What should the default be? (Still thinking 'off' myself)
>> default 'off' is a safe option, as it won't allow users to make
>> any change to parameter values until/unless they
>> read from manual, how to use it and what can go wrong, on the
>> other side it will be bit hassle for user to use this
>> command. I think 'on' would be better.
>
> Yeah, no, I still think 'off' would be best for this particular option.
>
>> > - What happens if the user specifies 'postgresql.conf'? I'm thinking we
>> > would disallow such insanity (as that's what it is, imv..) by having
>> > an identifier in the file that this is the PG "auto conf" file.
>> I think we can detect by name and give error.
>> > - Should we have such an identifier in auto.conf to indicate that we
>> > created it, to prevent the user from setting it to something they
>> > shouldn't?
>> I think if user plays with this file manually, it can lead to
>> problems, that's why earlier we have
>> decided to keep a note on top of file which will indicate, do not
>> edit this file manually.
>> I believe that should be sufficient.
>
> I agree that having such a disclaimer at the top of the file is a good
> idea. I'm not completely convinced that's sufficient but it's certainly
> better than nothing.
>
>> > - What's the "bootstrap" mode; iow, if a user enables the option but the
>> > file doesn't exist, what do we do? With this approach, I'd be
>> > inclined to say we simply create it and put the marker to indicate
>> > it's "our" file.
>>
>> Alter System will create the file if doesn't exist.
>
> ... Only if it's enabled though.
Yes.
>
>> > - Should we allow it to be outside of the data dir? We could simply log
>> > an error and ignore the parameter if it's more than a simple filename.
>>
>> This should be an error, the file location and name will be fixed.
>
> Not with this proposal... If it's fixed then it makes no sense to make it
> look like it can be modified.
This proposal is to have a special include which user can only use
for enable/disable which means he can remove symbol '#' or add '#'. We cannot stop user from changing file name or
addsome different
location, but that can lead to problems. We can have a note on top of this include indicating it is only for
enable/disable.
I think if we add more meaning to it, like allow user to change it,
handling and defining of that will be bit complex.
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Martijn van Oosterhout
Date:
On Fri, Aug 23, 2013 at 06:41:04PM +0530, Amit Kapila wrote: > > Not with this proposal... If it's fixed then it makes no sense to make it > > look like it can be modified. > > This proposal is to have a special include which user can only use > for enable/disable > which means he can remove symbol '#' or add '#'. > We cannot stop user from changing file name or add some different > location, but that can lead to problems. > We can have a note on top of this include indicating it is only for > enable/disable. Note, my whole purpose for suggesting something like: include_auto_conf_file postgresql.auto.conf is because I want the file location to be configurable. If I put in my configuration: include_auto_conf_file /etc/postgresql/9.4/postgresql.auto.conf it better put the options there. And if I comment the line out ALTER SYSTEM should stop working. If I put it at the beginning of the config then any other option in the file will override it (which we can detect and warn about). If I put it at the end of the file, ALTER SYSTEM overrides any statically configured options. And I can imagine hosting providers putting the configuration for memory usage, shared library directories and other such options after, and options like cpu_cost, enable_merge_join, etc before. That way they have fine grained control over which options the user can set and which not. > I think if we add more meaning to it, like allow user to change it, > handling and defining of that will be bit complex. Letting the user configure the location seems like common curtesy. Note this line isn't in itself a GUC, so you can't configure it via ALTER SYSTEM. Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > He who writes carelessly confesses thereby at the very outset that he does > not attach much importance to his own thoughts. -- Arthur Schopenhauer
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Sat, Aug 24, 2013 at 6:08 PM, Martijn van Oosterhout <kleptog@svana.org> wrote: > On Fri, Aug 23, 2013 at 06:41:04PM +0530, Amit Kapila wrote: >> > Not with this proposal... If it's fixed then it makes no sense to make it >> > look like it can be modified. >> >> This proposal is to have a special include which user can only use >> for enable/disable >> which means he can remove symbol '#' or add '#'. >> We cannot stop user from changing file name or add some different >> location, but that can lead to problems. >> We can have a note on top of this include indicating it is only for >> enable/disable. > > Note, my whole purpose for suggesting something like: > > include_auto_conf_file postgresql.auto.conf > > is because I want the file location to be configurable. If I put in my > configuration: > > include_auto_conf_file /etc/postgresql/9.4/postgresql.auto.conf > > it better put the options there. And if I comment the line out ALTER > SYSTEM should stop working. If I put it at the beginning of the config > then any other option in the file will override it (which we can detect > and warn about). If I put it at the end of the file, ALTER SYSTEM > overrides any statically configured options. > > And I can imagine hosting providers putting the configuration for > memory usage, shared library directories and other such options after, > and options like cpu_cost, enable_merge_join, etc before. That way > they have fine grained control over which options the user can set and > which not. Thanks for your suggestion. Above usecase can be achieved even if the file is not configurable. >> I think if we add more meaning to it, like allow user to change it, >> handling and defining of that will be bit complex. > > Letting the user configure the location seems like common curtesy. Note > this line isn't in itself a GUC, so you can't configure it via ALTER > SYSTEM. In general yes it is better to give control to user for configuring the location, but as this file will be used for internal purpose and will be modified by system and not by user, so it is better to be in data directory. The similar thing was discussed previously on this thread and it is suggested to have this file in data directory. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
Martijn,
* Martijn van Oosterhout (kleptog@svana.org) wrote:
> Note, my whole purpose for suggesting something like:
>
> include_auto_conf_file postgresql.auto.conf
>
> is because I want the file location to be configurable. If I put in my
> configuration:
>
> include_auto_conf_file /etc/postgresql/9.4/postgresql.auto.conf
I'm not following the use-case here.
Why would it make sense for a file which is entirely managed by PG
automatically to be in /etc? Of course, by the same token you might ask
why it makes sense to let the user pick it at all, which holds some
merit- but I liked your idea because then an admin doesn't have to go
looking for the file but instead knows where it is. Perhaps comments in
the file stating where the auto.conf lives would be sufficient, but
those are too often nuked. :(
> it better put the options there. And if I comment the line out ALTER
> SYSTEM should stop working. If I put it at the beginning of the config
> then any other option in the file will override it (which we can detect
> and warn about). If I put it at the end of the file, ALTER SYSTEM
> overrides any statically configured options.
>
> And I can imagine hosting providers putting the configuration for
> memory usage, shared library directories and other such options after,
> and options like cpu_cost, enable_merge_join, etc before. That way
> they have fine grained control over which options the user can set and
> which not.
For my part, I'd honestly rather have the first things found be what's
picked and later things be ignored. If you want it controlled by ALTER
SYSTEM, then don't set it in postgresql.conf. The problem with that is
there's no "bootstrap" mechanism without actually modifying such configs
or making the configs be in auto.conf to begin with, neither of which is
ideal, imv.
I really hate the idea that someone could configure 'X' in
postgresql.conf and because the auto.conf line is later in the file,
it's able to override the original setting. Does not strike me as
intuitive at all.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Mon, Aug 26, 2013 at 10:50 PM, Stephen Frost <sfrost@snowman.net> wrote: > Martijn, > > * Martijn van Oosterhout (kleptog@svana.org) wrote: >> Note, my whole purpose for suggesting something like: >> >> include_auto_conf_file postgresql.auto.conf >> >> is because I want the file location to be configurable. If I put in my >> configuration: >> >> include_auto_conf_file /etc/postgresql/9.4/postgresql.auto.conf > > I'm not following the use-case here. > > Why would it make sense for a file which is entirely managed by PG > automatically to be in /etc? Of course, by the same token you might ask > why it makes sense to let the user pick it at all, which holds some > merit- but I liked your idea because then an admin doesn't have to go > looking for the file but instead knows where it is. Perhaps comments in > the file stating where the auto.conf lives would be sufficient, but > those are too often nuked. :( > >> it better put the options there. And if I comment the line out ALTER >> SYSTEM should stop working. If I put it at the beginning of the config >> then any other option in the file will override it (which we can detect >> and warn about). If I put it at the end of the file, ALTER SYSTEM >> overrides any statically configured options. >> >> And I can imagine hosting providers putting the configuration for >> memory usage, shared library directories and other such options after, >> and options like cpu_cost, enable_merge_join, etc before. That way >> they have fine grained control over which options the user can set and >> which not. > > For my part, I'd honestly rather have the first things found be what's > picked and later things be ignored. If you want it controlled by ALTER > SYSTEM, then don't set it in postgresql.conf. The problem with that is > there's no "bootstrap" mechanism without actually modifying such configs > or making the configs be in auto.conf to begin with, neither of which is > ideal, imv. > > I really hate the idea that someone could configure 'X' in > postgresql.conf and because the auto.conf line is later in the file, > it's able to override the original setting. Does not strike me as > intuitive at all. This is currently how include mechanism works in postgresql.conf, changing that for this special case can be costly and moreover the specs for this patch were layout from beginning that way. However, we can have some mechanism for appending values (if feasible) as suggested in mail below, after initial patch is done http://www.postgresql.org/message-id/52015414.9060900@cybertec.at With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Tue, Aug 27, 2013 at 09:04:00AM +0530, Amit Kapila wrote: > > For my part, I'd honestly rather have the first things found be what's > > picked and later things be ignored. If you want it controlled by ALTER > > SYSTEM, then don't set it in postgresql.conf. The problem with that is > > there's no "bootstrap" mechanism without actually modifying such configs > > or making the configs be in auto.conf to begin with, neither of which is > > ideal, imv. > > > > I really hate the idea that someone could configure 'X' in > > postgresql.conf and because the auto.conf line is later in the file, > > it's able to override the original setting. Does not strike me as > > intuitive at all. > > This is currently how include mechanism works in postgresql.conf, > changing that for this special case can be costly and moreover the > specs for this patch were layout from beginning that way. Agreed. If you are worried about ALTER SYSTEM changing postgresql.conf settings, you should move the include_auto line to the top of postgresql.conf, but I don't think that should be the default. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Aug 27, 2013 at 09:04:00AM +0530, Amit Kapila wrote:
> > > I really hate the idea that someone could configure 'X' in
> > > postgresql.conf and because the auto.conf line is later in the file,
> > > it's able to override the original setting. Does not strike me as
> > > intuitive at all.
> >
> > This is currently how include mechanism works in postgresql.conf,
> > changing that for this special case can be costly and moreover the
> > specs for this patch were layout from beginning that way.
>
> Agreed. If you are worried about ALTER SYSTEM changing postgresql.conf
> settings, you should move the include_auto line to the top of
> postgresql.conf, but I don't think that should be the default.
While I appreciate that there are bootstrap-type issues with this, I
really don't like this idea of "later stuff can just override earlier
stuff".
include files and conf.d-style options are for breaking the config up,
not to allow you to override options because a file came later than an
earlier file. Our particular implementation of config-file reading
happens to lend itself to later-definition-wins, but that's really
counter-intuitive for anyone unfamiliar with PG, imv.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Stephen Frost <sfrost@snowman.net> writes:
> While I appreciate that there are bootstrap-type issues with this, I
> really don't like this idea of "later stuff can just override earlier
> stuff".
> include files and conf.d-style options are for breaking the config up,
> not to allow you to override options because a file came later than an
> earlier file. Our particular implementation of config-file reading
> happens to lend itself to later-definition-wins, but that's really
> counter-intuitive for anyone unfamiliar with PG, imv.
I don't follow this argument at all. Do you know any software with text
config files that will act differently from this if the same setting is
listed twice? "Last one wins" is certainly what I'd expect.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Wed, Aug 28, 2013 at 02:30:41PM -0400, Stephen Frost wrote: > * Bruce Momjian (bruce@momjian.us) wrote: > > On Tue, Aug 27, 2013 at 09:04:00AM +0530, Amit Kapila wrote: > > > > I really hate the idea that someone could configure 'X' in > > > > postgresql.conf and because the auto.conf line is later in the file, > > > > it's able to override the original setting. Does not strike me as > > > > intuitive at all. > > > > > > This is currently how include mechanism works in postgresql.conf, > > > changing that for this special case can be costly and moreover the > > > specs for this patch were layout from beginning that way. > > > > Agreed. If you are worried about ALTER SYSTEM changing postgresql.conf > > settings, you should move the include_auto line to the top of > > postgresql.conf, but I don't think that should be the default. > > While I appreciate that there are bootstrap-type issues with this, I > really don't like this idea of "later stuff can just override earlier > stuff". > > include files and conf.d-style options are for breaking the config up, > not to allow you to override options because a file came later than an > earlier file. Our particular implementation of config-file reading > happens to lend itself to later-definition-wins, but that's really > counter-intuitive for anyone unfamiliar with PG, imv. Agreed, but I think this is a much larger issue than ALTER SYSTEM SET. I think changing behavior to first-seen would only add to confusion. What we really need is a WARNING when a later postgresql.conf setting overrides an earlier one, and ALTER SYSTEM SET's config file could behave the same, so you would know right away when you were overriding something in postgresql.conf that appeared earlier. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > While I appreciate that there are bootstrap-type issues with this, I
> > really don't like this idea of "later stuff can just override earlier
> > stuff".
>
> > include files and conf.d-style options are for breaking the config up,
> > not to allow you to override options because a file came later than an
> > earlier file. Our particular implementation of config-file reading
> > happens to lend itself to later-definition-wins, but that's really
> > counter-intuitive for anyone unfamiliar with PG, imv.
>
> I don't follow this argument at all. Do you know any software with text
> config files that will act differently from this if the same setting is
> listed twice? "Last one wins" is certainly what I'd expect.
Have you tried having the same mount specified in multiple files in
fstab.d..? Also, aiui (not a big exim fan personally), duplicate
definitions in an exim4/conf.d would result in an error. Playing around
in apache2/sites-enabled, it appears to be "first wins" wrt virtual
hosts.
There's a number of cases where only a single value is being set and
subseqeunt files are 'additive' (eg: ld.so.conf.d), so they don't have
this issue. Similar to that are script directories, which simply run a
set of scripts in the $DIR.d and, as it's the scripts themselves which
are the 'parameters', and being files in a directory, you can't
duplicate them. Others (eg: pam.d) define the file name to be an
enclosing context, again preventing duplication by using the filename
itself.
There are counter-examples also; sysctl.d will replace what's already
been set, so perhaps it simply depends on an individual's experience.
For my part, I'd much prefer an error or warning saying "you've got
duplicate definitions of X" than a last-wins approach, though perhaps
that's just because I like things to be very explicit and clear.
Allowing multiple definitions of the same parameter to be valid isn't
'clear' to me.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Bruce Momjian (bruce@momjian.us) wrote:
> Agreed, but I think this is a much larger issue than ALTER SYSTEM SET.
Yeah, true.
> I think changing behavior to first-seen would only add to confusion.
> What we really need is a WARNING when a later postgresql.conf setting
> overrides an earlier one, and ALTER SYSTEM SET's config file could
> behave the same, so you would know right away when you were overriding
> something in postgresql.conf that appeared earlier.
A warning would be nice. If only everyone read the log files. :/
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Alvaro Herrera
Date:
Stephen Frost escribió: > There are counter-examples also; sysctl.d will replace what's already > been set, so perhaps it simply depends on an individual's experience. > For my part, I'd much prefer an error or warning saying "you've got > duplicate definitions of X" than a last-wins approach, though perhaps > that's just because I like things to be very explicit and clear. > Allowing multiple definitions of the same parameter to be valid isn't > 'clear' to me. One of the elements that had to be in place for this whole thing of multiple configuration files to be allowed was for pg_settings to include precise information about, for each setting, where is its value coming from. I doubt any of the systems you mentioned have comparable functionality. If the admin has trouble figuring it out, he needs only look into that view. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Bruce Momjian
Date:
On Wed, Aug 28, 2013 at 03:15:14PM -0400, Stephen Frost wrote: > * Bruce Momjian (bruce@momjian.us) wrote: > > Agreed, but I think this is a much larger issue than ALTER SYSTEM SET. > > Yeah, true. > > > I think changing behavior to first-seen would only add to confusion. > > What we really need is a WARNING when a later postgresql.conf setting > > overrides an earlier one, and ALTER SYSTEM SET's config file could > > behave the same, so you would know right away when you were overriding > > something in postgresql.conf that appeared earlier. > > A warning would be nice. If only everyone read the log files. :/ I would expect ALTER SYSTEM SET to return a WARNING in such cases too. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Wed, Aug 28, 2013 at 3:10 PM, Stephen Frost <sfrost@snowman.net> wrote: > * Tom Lane (tgl@sss.pgh.pa.us) wrote: >> Stephen Frost <sfrost@snowman.net> writes: >> > While I appreciate that there are bootstrap-type issues with this, I >> > really don't like this idea of "later stuff can just override earlier >> > stuff". >> >> > include files and conf.d-style options are for breaking the config up, >> > not to allow you to override options because a file came later than an >> > earlier file. Our particular implementation of config-file reading >> > happens to lend itself to later-definition-wins, but that's really >> > counter-intuitive for anyone unfamiliar with PG, imv. >> >> I don't follow this argument at all. Do you know any software with text >> config files that will act differently from this if the same setting is >> listed twice? "Last one wins" is certainly what I'd expect. > > Have you tried having the same mount specified in multiple files in > fstab.d..? Also, aiui (not a big exim fan personally), duplicate > definitions in an exim4/conf.d would result in an error. Playing around > in apache2/sites-enabled, it appears to be "first wins" wrt virtual > hosts. > > There's a number of cases where only a single value is being set and > subseqeunt files are 'additive' (eg: ld.so.conf.d), so they don't have > this issue. Similar to that are script directories, which simply run a > set of scripts in the $DIR.d and, as it's the scripts themselves which > are the 'parameters', and being files in a directory, you can't > duplicate them. Others (eg: pam.d) define the file name to be an > enclosing context, again preventing duplication by using the filename > itself. > > There are counter-examples also; sysctl.d will replace what's already > been set, so perhaps it simply depends on an individual's experience. > For my part, I'd much prefer an error or warning saying "you've got > duplicate definitions of X" than a last-wins approach, though perhaps > that's just because I like things to be very explicit and clear. > Allowing multiple definitions of the same parameter to be valid isn't > 'clear' to me. This may be true, but I think it's got little if anything to do with the current patch. There are lots of things that I don't like about our GUC machinery, too, but the goal of this thread shouldn't be to fix them all, but to get a useful piece of functionality added. The thing that we should keep in mind here is that it's *already* possible - TODAY - for users to overwrite postgresql.conf with a new set of settings. pgAdmin has this functionality built-in; it only requires that you install the "adminpack" contrib module, which we ship. According to Dave Page, pgAdmin will even offer to run CREATE EXTENSION adminpack for you, if you haven't done that already. Also according to Dave Page, if two users try to use it concurrently, you will get exactly the sort of awful results that you might expect. If it gets killed halfway through rewriting the file, too bad for you. The sorts of watered-down half-features being proposed here are not going to do anything to address that situation. If there are restrictions on what GUCs can be changed with this feature, or if the feature is disabled by default, or if you can only use it when the moon is full and you hop on your left foot while spinning around sideways, then pgAdmin (and any other, similar tools) are just going to keep doing what they do today in the same crappy, unsafe way they currently do it. Meanwhile, people who use psql are going to continue to find themselves without a reasonable API for doing the same thing that can be done easily using pgAdmin. I think the goals of this patch should be to (1) let users of other clients manipulate the startup configuration just as easily as we can already do using pgAdmin and (2) remove some of the concurrency hazards that pgAdmin introduces, for example by using locking and atomic renames. Restricting the functionality to something less than what pgAdmin provides will just cause people to ignore the new mechanism and use the one that already exists and, by and large, works. And trying to revise other aspects of how GUCs and config files work as part of this effort is not reasonably related to this patch, and should be kept out of the discussion. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
Robert,
* Robert Haas (robertmhaas@gmail.com) wrote:
> The sorts of watered-down half-features being proposed here are not
> going to do anything to address that situation. If there are
> restrictions on what GUCs can be changed with this feature, or if the
> feature is disabled by default, or if you can only use it when the
> moon is full and you hop on your left foot while spinning around
> sideways, then pgAdmin (and any other, similar tools) are just going
> to keep doing what they do today in the same crappy, unsafe way they
> currently do it. Meanwhile, people who use psql are going to continue
> to find themselves without a reasonable API for doing the same thing
> that can be done easily using pgAdmin.
To be honest, I don't find the arguments of "pgAdmin does it badly"
nor "psql users want this ability" (which I find difficult to believe)
to be particularlly compelling reasons to have a dangerous
implementation (even if it's better than 'adminpack' today) in core.
If it's in core rather than in contrib it's going to be deployed a great
deal farther with a large increase in userbase. I've already stated
that if this is in contrib that my concerns are much less.
> I think the goals of this patch should be to (1) let users of other
> clients manipulate the startup configuration just as easily as we can
> already do using pgAdmin,
Which could be done already through use of adminpack.. The capabilities
exposed there could be used by other clients. The fact that none of the
other clients have chosen to do so could be an indication that this
ability isn't terribly 'in demand'.
> and (2) remove some of the concurrency
> hazards that pgAdmin introduces, for example by using locking and
> atomic renames.
Why can't adminpack do that..?
> Restricting the functionality to something less than
> what pgAdmin provides will just cause people to ignore the new
> mechanism and use the one that already exists and, by and large,
> works. And trying to revise other aspects of how GUCs and config
> files work as part of this effort is not reasonably related to this
> patch, and should be kept out of the discussion.
We're talking about modifying config files through an interface and you
wish to exclude all discussion about how those config files are handled.
That leads to a result that only an adminpack-like solution is an
option, to which I respond "use and improve adminpack then". If we want
something that works well in *core* then I don't think we can exclude
how core reads and handles config files from the discussion. We need a
real solution, not another adminpack-like hack.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Thu, Aug 29, 2013 at 1:42 PM, Stephen Frost <sfrost@snowman.net> wrote: > To be honest, I don't find the arguments of "pgAdmin does it badly" > nor "psql users want this ability" (which I find difficult to believe) > to be particularlly compelling reasons to have a dangerous > implementation (even if it's better than 'adminpack' today) in core. I don't understand how you can find that difficult to believe. I'm a psql user and I want it. Josh Berkus is a psql user and he wants it. And there are numerous statements of support on these threads from other people as well. The sheer volume of discussion on this topic, and the fact that it has not gone away after years of wrangling, is a clear indication that people do, in fact, want it. To be honest, I think the argument that this is dangerous is pretty ridiculous. AFAICS, the only argument anyone's advanced for this being dangerous is the theory that you might accidentally put something in your postgresql.conf file that makes the server not start. However, the reality is that the superuser has MANY, MANY ways of killing the database cluster as things stand. Consider the ever-popular "DELETE FROM pg_proc". That will not only render your database unusable, but it's a hell of a lot harder to recover from than anything you might do to postgresql.conf. Therefore, from where I'm sitting, this is like telling a head of state with the ability to launch nuclear weapons which will destroy the planet that he isn't allowed to bring his swiss army knife on board an aircraft. Now, you can argue that people are more likely to render the database nonfunctional by changing GUC settings than they are to do it by corrupting the system catalogs, but I'm not sure I believe it. We can, after all, validate that any setting a user supplies is a valid value for that setting before writing it out to the configuration file. It might still make the system fail to start if - for example - it causes too many semaphores to be reserved, or something like that. But why should we think that such mistakes will be common? If anything, it sounds less error-prone to me than hand-editing the configuration file, where typing something like on_exit_error=maybe will make the server fail to start. We can eliminate that whole category of error, at least for people who choose to use the new tools. If you're using the term "dangerous" to refer to a security exposure, I concede there is some incremental exposure from allowing this by default. But it's not a terribly large additional exposure. It's already the case that if adminpack is available the super-user can do whatever he or she wants, because she or he can write to arbitrary files inside the data directory. Even if not, for most intents and purposes, ALTER DATABASE my_main_database SET whatever = thing is functionally equivalent to modifying postgresql.conf. Some settings can't be modified that way, but so what? AFAICS, about the worst thing the user can do is ALTER SYSTEM SET shared_preload_libraries = 'rootkit'. But if the user has the ability to install rootkit.so, the sysadmin is already screwed. And aside from that sort of thing, I don't really see what can happen that's any worse than what a rogue superuser can already do as things stand today. > If it's in core rather than in contrib it's going to be deployed a great > deal farther with a large increase in userbase. I've already stated > that if this is in contrib that my concerns are much less. I don't really see a compelling reason why it can't or shouldn't be in core. But having something better in contrib would still be an improvement on the status quo. >> I think the goals of this patch should be to (1) let users of other >> clients manipulate the startup configuration just as easily as we can >> already do using pgAdmin, > > Which could be done already through use of adminpack.. The capabilities > exposed there could be used by other clients. The fact that none of the > other clients have chosen to do so could be an indication that this > ability isn't terribly 'in demand'. Huh? The problem with adminpack is that it doesn't let you modify individual configuration settings. All you can do is rewrite an entire file. I guess somebody could write a specialized client that just uses that infrastructure to rewrite postgresql.conf. For all I know, someone has. Even if not, I don't think that you can use that to prove that people don't care about this feature. If nobody cares, why are there 400 emails on this topic?! >> and (2) remove some of the concurrency >> hazards that pgAdmin introduces, for example by using locking and >> atomic renames. > > Why can't adminpack do that..? It could. But it doesn't. We could improve it to do that, and that might be worthwhile, but it still wouldn't be as nice as what's being proposed here. >> Restricting the functionality to something less than >> what pgAdmin provides will just cause people to ignore the new >> mechanism and use the one that already exists and, by and large, >> works. And trying to revise other aspects of how GUCs and config >> files work as part of this effort is not reasonably related to this >> patch, and should be kept out of the discussion. > > We're talking about modifying config files through an interface and you > wish to exclude all discussion about how those config files are handled. > That leads to a result that only an adminpack-like solution is an > option, to which I respond "use and improve adminpack then". If we want > something that works well in *core* then I don't think we can exclude > how core reads and handles config files from the discussion. We need a > real solution, not another adminpack-like hack. I'm not sure what you mean about "all discussion about how those config files are handled". I think it's entirely reasonable to discuss how where the automatically-written configuration settings will be stored, and how they'll interact with manually-generated files. But IIUC you have proposed: 1. disabling the feature by default, and providing no way for it to be turned on remotely, and 2. even when the feature is enabled, only allowing a subset of the settings to be changed with it, and 3. also changing the interpretation of the config files so that we have a first-wins rules instead of a last-wins rule. If we do either #1 or #2, this won't be a plausible substitute for the functionality that adminpack offers today. #3 is a bad idea in my book - it would break some of my existing benchmarking scripts, which do initdb; cat >>$PGDATA/postgresql.conf <<EOM. But even if it were a good idea, it isn't a necessary prerequisite for this patch. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-29 15:07:35 -0400, Robert Haas wrote: > On Thu, Aug 29, 2013 at 1:42 PM, Stephen Frost <sfrost@snowman.net> wrote: > > To be honest, I don't find the arguments of "pgAdmin does it badly" > > nor "psql users want this ability" (which I find difficult to believe) > > to be particularlly compelling reasons to have a dangerous > > implementation (even if it's better than 'adminpack' today) in core. > > I don't understand how you can find that difficult to believe. I'm a > psql user and I want it. Josh Berkus is a psql user and he wants it. > And there are numerous statements of support on these threads from > other people as well. The sheer volume of discussion on this topic, > and the fact that it has not gone away after years of wrangling, is a > clear indication that people do, in fact, want it. > > To be honest, I think the argument that this is dangerous is pretty > ridiculous. +waytoomuch. > > If it's in core rather than in contrib it's going to be deployed a great > > deal farther with a large increase in userbase. I've already stated > > that if this is in contrib that my concerns are much less. > > I don't really see a compelling reason why it can't or shouldn't be in > core. But having something better in contrib would still be an > improvement on the status quo. I don't see much argument for putting it into contrib. One class of users this will benefit is relatively new ones, possibly using some GUI. Adding some additional complexity for them to enable the feature seems pointless to me. If you don't want your installation to use it, tell you ops people not to do so. They are superusers, they need to have the ability to follow some rules you make up internally. The energy wasted in a good part of this massive 550+ messages thread is truly saddening. We all (c|sh)ould have spent that time making PG more awesome instead. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
On Thursday, August 29, 2013, Andres Freund wrote:
On 2013-08-29 15:07:35 -0400, Robert Haas wrote:
> I don't really see a compelling reason why it can't or shouldn't be in
> core. But having something better in contrib would still be an
> improvement on the status quo.
I don't see much argument for putting it into contrib. One class of
users this will benefit is relatively new ones, possibly using some
GUI. Adding some additional complexity for them to enable the feature
seems pointless to me.
I keep wondering what this fantastic new GUI that isn't pgAdmin is, and why it wouldn't be able to use the exact same mechanism that pgAdmin uses and provides in a few simple steps to enable this.
If you don't want your installation to use it, tell you ops people not
to do so. They are superusers, they need to have the ability to follow
some rules you make up internally.
The OPs people are the ones that will be upset with this because the DBAs will be modifying configs which OPs rightfully claim as theirs. If they have a way to prevent it then perhaps it's not terrible but they'd also need to know to disable this new "feature". As for ALTER DATABASE- I would be happier with encouraging use of that (or providing an ALTER CLUSTER) for those thing it makes sense and works for and removing those from being in postgresql.conf. I still feel things like listen_addresses shouldn't be changed through this.
The energy wasted in a good part of this massive 550+ messages thread is
truly saddening. We all (c|sh)ould have spent that time making PG more
awesome instead.
Perhaps not understood by all, but keeping PG awesome involves more than adding every feature proposed- it also means saying no sometimes; to features, to new GUCs, even to micro-optimizations when they're overly complicated and offer only minimal or questionable improvements.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-29 18:42:13 -0400, Stephen Frost wrote: > On Thursday, August 29, 2013, Andres Freund wrote: > > The energy wasted in a good part of this massive 550+ messages thread is > > truly saddening. We all (c|sh)ould have spent that time making PG more > > awesome instead. > Perhaps not understood by all, but keeping PG awesome involves more than > adding every feature proposed- it also means saying no sometimes; to > features, to new GUCs, even to micro-optimizations when they're overly > complicated and offer only minimal or questionable improvements. But that doesn't mean that endlessly discussing in circles is a worthwile thing to do. The discussion essentially hasn't progressed towards concensus in the last months at all besides involving most active hackers at some point. If anything it has become more contentious. Obviously lots of people have strong opinions. Now we need to make decisions. Even if it ends up being ones I judge to be ridiculously bad. > > On 2013-08-29 15:07:35 -0400, Robert Haas wrote: > > > I don't really see a compelling reason why it can't or shouldn't be in > > > core. But having something better in contrib would still be an > > > improvement on the status quo. > > > > I don't see much argument for putting it into contrib. One class of > > users this will benefit is relatively new ones, possibly using some > > GUI. Adding some additional complexity for them to enable the feature > > seems pointless to me. > I keep wondering what this fantastic new GUI that isn't pgAdmin is, and why > it wouldn't be able to use the exact same mechanism that pgAdmin uses and > provides in a few simple steps to enable this. To quote Robert two mails up: > Huh? The problem with adminpack is that it doesn't let you modify > individual configuration settings. All you can do is rewrite an > entire file. I guess somebody could write a specialized client that > just uses that infrastructure to rewrite postgresql.conf. For all I > know, someone has. Even if not, I don't think that you can use that > to prove that people don't care about this feature. If nobody cares, > why are there 400 emails on this topic?! Also, doing it the adminpack way lacks even the most basic validity checks. And that's not really changeable. Presumably one major reason why we don't have other|good GUIs is that it's ridicuously hard to make them work to an interesting extent with the current infrastructure. > > If you don't want your installation to use it, tell you ops people not > > to do so. They are superusers, they need to have the ability to follow > > some rules you make up internally. > The OPs people are the ones that will be upset with this because the DBAs > will be modifying configs which OPs rightfully claim as theirs. If they > have a way to prevent it then perhaps it's not terrible but they'd also > need to know to disable this new "feature". As for ALTER DATABASE- I would > be happier with encouraging use of that (or providing an ALTER CLUSTER) for > those thing it makes sense and works for and removing those from being in > postgresql.conf. I still feel things like listen_addresses shouldn't be > changed through this. If they give out superuser access it has to be to people who can follow rules. After all they don't DROP DATABASE; DELETE FROM pg_class; alter passwords; use adminpack (changing postgresql.conf..); ... All of which they can do. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
On Thursday, August 29, 2013, Andres Freund wrote:
To quote Robert two mails up:
> Huh? The problem with adminpack is that it doesn't let you modify
> individual configuration settings. All you can do is rewrite an
> entire file.
That's clearly fixable.
> I guess somebody could write a specialized client that
> just uses that infrastructure to rewrite postgresql.conf. For all I
> know, someone has. Even if not, I don't think that you can use that
> to prove that people don't care about this feature. If nobody cares,
> why are there 400 emails on this topic?!
Having 400 emails about it means it's contentious. That's quite different from having a large demand. It does speak to the author's persistence as well, but that shouldn't be a surprise.
Also, doing it the adminpack way lacks even the most basic validity
checks. And that's not really changeable.
I don't see why..? Admin pack could certainly be modified to take a parameter and do appropriate verification before locking an object and rewriting the file. It's what we're being expected to do in core, after all. Indeed, we can't even do validity checks on all the options, which is the crux of what I'm concerned about.
Presumably one major reason why we don't have other|good GUIs is that
it's ridicuously hard to make them work to an interesting extent with
the current infrastructure.
Yet no one has tried to improve admin pack?
If they give out superuser access it has to be to people who can follow
rules. After all they don't DROP DATABASE; DELETE FROM pg_class; alter
passwords; use adminpack (changing postgresql.conf..); ... All of which
they can do.
This completely misses, or perhaps just ignores, the point. Disallowing super user access can be difficult because there's a lot of *normal* DBA activities which can't be easily done without it (like changing table ownership or similar). The "createrole" option definitely improved things but we aren't there yet. It's certainly easy to simply not install the adminpack. The other concerns above are strawmen because they attack a malicious DBA. I'm not talking about malicious DBAs but rather a generally knowledgable DBA who changed shared_buffers up too high and then leaves on vacation, while the OPs guys need to do a database restart for whatever reason and then discover it doesn't start.
I bring up these concerns because I have environments where I can see exactly this happening and I have a hard time believing that I'm somehow alone.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Thu, Aug 29, 2013 at 8:07 PM, Stephen Frost <sfrost@snowman.net> wrote: > Having 400 emails about it means it's contentious. That's quite different > from having a large demand. It does speak to the author's persistence as > well, but that shouldn't be a surprise. Yet you can't ignore the fact that many people on these threads want some form of this. >> Presumably one major reason why we don't have other|good GUIs is that >> it's ridicuously hard to make them work to an interesting extent with >> the current infrastructure. > > Yet no one has tried to improve admin pack? No, but multiple people have tried to do ALTER SYSTEM .. SET. Peter had a crack at this problem, I fooled around with it (though I don't think I ever got as far as publishing), and there were various others as well (Greg Smith?). > I'm not talking about malicious DBAs but rather a generally > knowledgable DBA who changed shared_buffers up too high and then leaves on > vacation, while the OPs guys need to do a database restart for whatever > reason and then discover it doesn't start. /me looks at Stephen incredulously. In the first place, modifying postgresql.conf and not immediately restarting the server to test your changes is probably the single most basic DBA error imaginable. I have a hard time viewing such a person as generally knowledgeable. I hope the DBA in question doesn't have access to the switches, because he's probably the sort of guy who reassigns switch ports and doesn't type "wr m" afterwards. In the second place, the same guy can do the same thing today. He just has to use "vi". In fact, I think this is a pretty common failure mode in poorly-controlled environments where too many people have access to the configuration files. Now maybe you're going to tell me that the ops guys can't modify the configuration file because they only have SQL-level access, but then how are they managing to restart the database? They need to be able to run pg_ctl *as the postgres user* to do that, and if they have shell access to that account, all bets are off. Sure, you can construct a scenario where this matters. The ops guys have "sudo postgres pg_ctl" access but adminpack isn't installed and they have no other way to modify the configuration file. But that's just bizarre. And if that's really the environment you have, then you can install a loadable module that grabs ProcessUtility_hook and uses it to forbid ALTER SYSTEM on that machine. Hell, we can ship such a thing in contrib. Problem solved. But it's surely too obscure a combination of circumstances to justify disabling this by default. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
Robert,
Was working on replying to this, but got distracted..
* Robert Haas (robertmhaas@gmail.com) wrote:
> To be honest, I think the argument that this is dangerous is pretty
> ridiculous. AFAICS, the only argument anyone's advanced for this
> being dangerous is the theory that you might accidentally put
> something in your postgresql.conf file that makes the server not
> start. However, the reality is that the superuser has MANY, MANY ways
> of killing the database cluster as things stand. Consider the
> ever-popular "DELETE FROM pg_proc".
You will not find me argueing to allow that in normal operation, or most
direct-catalog hacking. I'd be much happier if we had all the ALTER,
etc, options necessary to prevent any need to ever touch the catalogs.
Similairly, it'd be nice to have more permission options which reduce
the need for anyone to be superuser.
> Now, you can argue that people are more likely to render the database
> nonfunctional by changing GUC settings than they are to do it by
> corrupting the system catalogs, but I'm not sure I believe it. We
> can, after all, validate that any setting a user supplies is a valid
> value for that setting before writing it out to the configuration
> file. It might still make the system fail to start if - for example -
> it causes too many semaphores to be reserved, or something like that.
> But why should we think that such mistakes will be common? If
> anything, it sounds less error-prone to me than hand-editing the
> configuration file, where typing something like on_exit_error=maybe
> will make the server fail to start. We can eliminate that whole
> category of error, at least for people who choose to use the new
> tools.
That category of error is much more likely to get caught by the sysadmin
doing the restart after the update.. ALTER SYSTEM doesn't provide any
way to restart the DB to make sure it worked. That, in turn, encourages
modification of the config parameters w/o a restart, which is a pretty
bad idea, imv.
> If you're using the term "dangerous" to refer to a security exposure,
> I concede there is some incremental exposure from allowing this by
> default. But it's not a terribly large additional exposure. It's
> already the case that if adminpack is available the super-user can do
> whatever he or she wants, because she or he can write to arbitrary
> files inside the data directory.
This is only true if -contrib is installed, which a *lot* of admins
refuse to do, specifically because of such insane modules as this.
Therefore, I would argue that it's not nearly as small a change wrt
exposure as you believe. Strictly speaking, I don't believe it's a huge
security hole or something, but I do think it's going to surprise admins
who aren't following the lists.
> Even if not, for most intents and
> purposes, ALTER DATABASE my_main_database SET whatever = thing is
> functionally equivalent to modifying postgresql.conf. Some settings
> can't be modified that way, but so what? AFAICS, about the worst
> thing the user can do is ALTER SYSTEM SET shared_preload_libraries =
> 'rootkit'.
This is assuming a malicious user, and one who has access to the
filesystem, which are pretty big assumptions to be making. I'm not
trying to make any argument about a malicious user and in general I
don't believe the users of this capability are people who have root
on the box.
> Huh? The problem with adminpack is that it doesn't let you modify
> individual configuration settings. All you can do is rewrite an
> entire file. I guess somebody could write a specialized client that
> just uses that infrastructure to rewrite postgresql.conf. For all I
> know, someone has. Even if not, I don't think that you can use that
> to prove that people don't care about this feature. If nobody cares,
> why are there 400 emails on this topic?!
Addressed this in my reply to Andres.
> > Why can't adminpack do that..?
>
> It could. But it doesn't. We could improve it to do that, and that
> might be worthwhile, but it still wouldn't be as nice as what's being
> proposed here.
This wouldn't be the first time we've said "do it externally and when
we figure out a better answer then we'll put it into core."
> 1. disabling the feature by default, and providing no way for it to be
> turned on remotely, and
Yes, that's quite intentional because it would make someone knowledgable
about the *overall system* be aware of this rather than having it sprung
on them when they discover some change has made the database unable to
start.
> 2. even when the feature is enabled, only allowing a subset of the
> settings to be changed with it, and
> 3. also changing the interpretation of the config files so that we
> have a first-wins rules instead of a last-wins rule.
>
> If we do either #1 or #2, this won't be a plausible substitute for the
> functionality that adminpack offers today.
While I appreciate that the original thrust behind this may be trying to
replace adminpack, I find that module to be utterly horrendous in what
it does and if we don't replicate all that it can do, so much the
better. Consider a similar proposal as ALTER SYSTEM which simply
rewrites pg_hba.conf, ala how adminpack could. Would we punt on the
hard questions around things like how to specify the order in which
entries in pg_hba.conf are maintained? We could have an ALTER HBA today
which takes a line number to insert a string of text into, but is it a
good idea or design? Certainly not. My point here is simply that
focusing on replacing adminpack is not the basis for a feature or
design.
> #3 is a bad idea in my
> book - it would break some of my existing benchmarking scripts, which
> do initdb; cat >>$PGDATA/postgresql.conf <<EOM. But even if it were a
> good idea, it isn't a necessary prerequisite for this patch.
Too often do we consider sloppy handling like this, which is great for a
dev or testing environment, to be appropriate for production systems. :(
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Peter Geoghegan
Date:
On Thu, Aug 29, 2013 at 9:13 AM, Robert Haas <robertmhaas@gmail.com> wrote: > I think the goals of this patch should be to (1) let users of other > clients manipulate the startup configuration just as easily as we can > already do using pgAdmin and (2) remove some of the concurrency > hazards that pgAdmin introduces, for example by using locking and > atomic renames. +1 -- Peter Geoghegan
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Robert Haas (robertmhaas@gmail.com) wrote:
> On Thu, Aug 29, 2013 at 8:07 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > I'm not talking about malicious DBAs but rather a generally
> > knowledgable DBA who changed shared_buffers up too high and then leaves on
> > vacation, while the OPs guys need to do a database restart for whatever
> > reason and then discover it doesn't start.
>
> /me looks at Stephen incredulously.
>
> In the first place, modifying postgresql.conf and not immediately
> restarting the server to test your changes is probably the single most
> basic DBA error imaginable.
You're not modifying postgresql.conf with ALTER SYSTEM, now are you?
Admins are going to realize the need to restart or at least reload the
service after updating such, but a DBA who has focused mainly on
normalization, query optimization, etc, isn't going to have the same
experience that a sysadmin has.
A DBA updating a GUC, something they are likely to do frequently
day-in and day-out, isn't necessairly going to consider that it's a
reboot-needing change. Indeed, it's not unreasonable to consider that
we may, some day, actually be able to change or increase
shared_buffers on the fly (or error in the trying); I'd think your
work with the dynamic backends would actually make that likely to
happen sooner rather than later.
> I have a hard time viewing such a person
> as generally knowledgeable. I hope the DBA in question doesn't have
> access to the switches, because he's probably the sort of guy who
> reassigns switch ports and doesn't type "wr m" afterwards.
I'd consider the DBAs who I work with as generally knowledgable, and
thankfully also cautious enough to talk to me before making changes, but
they have superuser access (they need to be able to do database
'releases' to our production environments and those scripts need to
change table ownership and do other things which can be annoying to
maintain the permissions for) and certainly understand changing GUCs
(mostly things like work_mem).
> In the second place, the same guy can do the same thing today. He
> just has to use "vi". In fact, I think this is a pretty common
> failure mode in poorly-controlled environments where too many people
> have access to the configuration files. Now maybe you're going to
> tell me that the ops guys can't modify the configuration file because
> they only have SQL-level access, but then how are they managing to
> restart the database? They need to be able to run pg_ctl *as the
> postgres user* to do that, and if they have shell access to that
> account, all bets are off.
You seem to be confused here between the DBA/data team and the OPs or
Infrastructure team. I'm making a clear distinction between them and
feel quite comfortable with it because it's the environment that I live
in today. My job, in fact, is exactly to straddle that line and work
with both.
In the above example, the DBA hasn't got root on the box and can't
simply go change or modify postgresql.conf, not even with "vi", and is
true for most of the DBAs on my team, even though they have superuser
access in PG.
> Sure, you can construct a scenario where this matters. The ops guys
> have "sudo postgres pg_ctl" access but adminpack isn't installed and
> they have no other way to modify the configuration file. But that's
> just bizarre. And if that's really the environment you have, then you
> can install a loadable module that grabs ProcessUtility_hook and uses
> it to forbid ALTER SYSTEM on that machine. Hell, we can ship such a
> thing in contrib. Problem solved. But it's surely too obscure a
> combination of circumstances to justify disabling this by default.
It's not the OPs guy that I'm worried about using ALTER SYSTEM- I don't
expect them to have any clue about it or care about it, except where it
can be used to modify things under /etc which they, rightfully, consider
their domain.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Cédric Villemain
Date:
Le jeudi 29 août 2013 18:42:13 Stephen Frost a écrit : > On Thursday, August 29, 2013, Andres Freund wrote: > > If you don't want your installation to use it, tell you ops people not > > to do so. They are superusers, they need to have the ability to follow > > some rules you make up internally. > > The OPs people are the ones that will be upset with this because the DBAs > will be modifying configs which OPs rightfully claim as theirs. If they > have a way to prevent it then perhaps it's not terrible but they'd also > need to know to disable this new "feature". As for ALTER DATABASE- I would > be happier with encouraging use of that (or providing an ALTER CLUSTER) for > those thing it makes sense and works for and removing those from being in > postgresql.conf. I still feel things like listen_addresses shouldn't be > changed through this. ALTER ROLE ALL may be good enougth to handle every GUC that we can also remove from postgresql.conf (I suppose all GUC needing only a reload, not a restart). It may needs some improvement to handle changing default for ALL and adding new role. > > The energy wasted in a good part of this massive 550+ messages thread is > > truly saddening. We all (c|sh)ould have spent that time making PG more > > awesome instead. > > Perhaps not understood by all, but keeping PG awesome involves more than > adding every feature proposed- it also means saying no sometimes; to > features, to new GUCs, even to micro-optimizations when they're overly > complicated and offer only minimal or questionable improvements. Agreed, the current feature and proposal does not include pg_reload, and it introduces a full machinery we absolutely don't need. Grammar can be added later when the feature is stable. So far, we can achieve the goal by using adminpack, by using a file_fdw or a config_fdw. IMHO it is the job of a FDW to be able to handle atomic write or anything like that. I've commented one of the proposed patch adding some helpers to validate GUC change, I claimed this part was good enough to be added without ALTER SYSTEM (so a contrib can use it). -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Dimitri Fontaine
Date:
Stephen Frost <sfrost@snowman.net> writes: > The OPs people are the ones that will be upset with this because the DBAs > will be modifying configs which OPs rightfully claim as theirs. If that's the problem you want to solve, there's no technical solution that will put you at ease. That's a people and trust problem. I don't think typical (or less typical) organisation should drive our technical choices too much, and I'm pretty confident they didn't in the past: pg_hba.conf is a file not because it's meant for this or that team but because it makes sense technically to manage the settings to allow using some resources *outside* of said resources. We currently have no way that I know of to disable ALTER ROLE SET and ALTER DATABASE SET effects, why do we need to provide that feature for ALTER SYSTEM SET so much? Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
Hi, On 2013-08-30 10:20:48 +0200, Cédric Villemain wrote: > > > The energy wasted in a good part of this massive 550+ messages thread is > > > truly saddening. We all (c|sh)ould have spent that time making PG more > > > awesome instead. > > > > Perhaps not understood by all, but keeping PG awesome involves more than > > adding every feature proposed- it also means saying no sometimes; to > > features, to new GUCs, even to micro-optimizations when they're overly > > complicated and offer only minimal or questionable improvements. > > Agreed, the current feature and proposal does not include pg_reload, and it > introduces a full machinery we absolutely don't need. The complexity in the last version of the patch I looked at wasn't in the added grammar or pg_reload() (well, it didn't have that). It was the logic to read (from memory)/write the config file and validate the GUCs. That's needed even if you put it into some module. And it requires support from guc.c/guc-file.l > Grammar can be added later when the feature is stable. Could you explain the advantages of this? It will require users to get used to different interfaces and we will end up maintaining both just about forever. And I don't see the grammar being that contentious? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Dimitri Fontaine (dimitri@2ndQuadrant.fr) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > The OPs people are the ones that will be upset with this because the DBAs
> > will be modifying configs which OPs rightfully claim as theirs.
>
> If that's the problem you want to solve, there's no technical solution
> that will put you at ease. That's a people and trust problem.
I really just don't buy that- I've already put forward suggestions for
how to deal with it, but no one here seems to understand the
distinction. Modifying listen_addresses through ALTER SYSTEM is akin to
ISC/bind allowing changes to its listen_addresses equivilant through
dynamic DNS updates. Would it be possible to implement? Sure. Does it
make any sense? Certainly not.
> I don't think typical (or less typical) organisation should drive our
> technical choices too much, and I'm pretty confident they didn't in the
> past: pg_hba.conf is a file not because it's meant for this or that team
> but because it makes sense technically to manage the settings to allow
> using some resources *outside* of said resources.
I'm all for moving pg_hba.conf into the database properly, actually. We
already handle permissions and user access in the DB and that's one of
the DBA's responsibilities. The same goes for pg_ident.conf.
> We currently have no way that I know of to disable ALTER ROLE SET and
> ALTER DATABASE SET effects, why do we need to provide that feature for
> ALTER SYSTEM SET so much?
Because we've got crap mixed into postgresql.conf which are bootstrap
configs needed to get the system started. Those things, in my view
anyway, fall much more into the category of "resources which should be
managed outside the database" than pg_hba.conf.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Cédric Villemain (cedric@2ndquadrant.com) wrote:
> ALTER ROLE ALL may be good enougth to handle every GUC that we can also remove
> from postgresql.conf (I suppose all GUC needing only a reload, not a restart).
> It may needs some improvement to handle changing default for ALL and adding
> new role.
Yes, one of the issues with the existing system is that you can't
specify a default to be applied to new roles. Also, there are
parameters which are not per-role yet which it probably makes sense to
be in the database and we'd need a way to deal with that. Although, at
the same time, considering it role based does make for a nice
distinction. Unfortunately, those clammoring for this will argue that
it wouldn't replace adminpack and that they couldn't use it to modify
their /etc/network/interfaces file, which is the obvious next step to
all of this.
> Grammar can be added later when the feature is stable.
I tend to agree w/ Andres on this point- the grammar isn't really the
contentious part. I think I see where you were going with this in that
excluding the grammar makes it able to live as a module, but that's a
different consideration.
> I've commented one of the proposed patch adding some helpers to validate GUC
> change, I claimed this part was good enough to be added without ALTER SYSTEM
> (so a contrib can use it).
Perhaps..
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Andres Freund (andres@2ndquadrant.com) wrote:
> On 2013-08-30 10:20:48 +0200, Cédric Villemain wrote:
> > Grammar can be added later when the feature is stable.
>
> Could you explain the advantages of this? It will require users to get
> used to different interfaces and we will end up maintaining both just
> about forever. And I don't see the grammar being that contentious?
The "different interfaces" issue already exists, to some extent, as
adminpack does exist already and this is clearly a different interface
from that. That said, as I mentioned just now elsewhere, I agree that
the grammar itself (being rather simple anyway...) isn't the contentious
part of this.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-30 08:48:21 -0400, Stephen Frost wrote: > * Dimitri Fontaine (dimitri@2ndQuadrant.fr) wrote: > > Stephen Frost <sfrost@snowman.net> writes: > > > The OPs people are the ones that will be upset with this because the DBAs > > > will be modifying configs which OPs rightfully claim as theirs. > > > > If that's the problem you want to solve, there's no technical solution > > that will put you at ease. That's a people and trust problem. > > I really just don't buy that- I've already put forward suggestions for > how to deal with it, but no one here seems to understand the > distinction. Modifying listen_addresses through ALTER SYSTEM is akin to > ISC/bind allowing changes to its listen_addresses equivilant through > dynamic DNS updates. Would it be possible to implement? Sure. Does it > make any sense? Certainly not. I very much want to change stuff like wal_level, listen_addresses and shared_buffers via ALTER SYSTEM. Configuration variables like that (PGC_POSTMASTER stuff mostly) are the prime reason why you actually need to change postgresql.conf instead of changing per user/database settings. And you don't even need to do anything special to implement it. Because it's already there. > > We currently have no way that I know of to disable ALTER ROLE SET and > > ALTER DATABASE SET effects, why do we need to provide that feature for > > ALTER SYSTEM SET so much? > > Because we've got crap mixed into postgresql.conf which are bootstrap > configs needed to get the system started. Those things, in my view > anyway, fall much more into the category of "resources which should be > managed outside the database" than pg_hba.conf. I think the problem with your position in this thread is that you want to overhaul the way our configuration works in a pretty radical way. Which is fair enough, there certainly are deficiencies. But it's not the topic of this thread. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-29 21:26:48 -0400, Stephen Frost wrote: > > Sure, you can construct a scenario where this matters. The ops guys > > have "sudo postgres pg_ctl" access but adminpack isn't installed and > > they have no other way to modify the configuration file. But that's > > just bizarre. And if that's really the environment you have, then you > > can install a loadable module that grabs ProcessUtility_hook and uses > > it to forbid ALTER SYSTEM on that machine. Hell, we can ship such a > > thing in contrib. Problem solved. But it's surely too obscure a > > combination of circumstances to justify disabling this by default. > > It's not the OPs guy that I'm worried about using ALTER SYSTEM- I don't > expect them to have any clue about it or care about it, except where it > can be used to modify things under /etc which they, rightfully, consider > their domain. I think for the scenarios you describe it makes far, far much more sense to add the ability to easily monitor for two things: * on-disk configuration isn't the same as the currently loaded (not trivially possible yet) * Configuration variables only come from locations that are approved for in your scenario (Already possible, we might wantto make it even easier) Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Andres Freund (andres@2ndquadrant.com) wrote:
> On 2013-08-30 08:48:21 -0400, Stephen Frost wrote:
> > I really just don't buy that- I've already put forward suggestions for
> > how to deal with it, but no one here seems to understand the
> > distinction. Modifying listen_addresses through ALTER SYSTEM is akin to
> > ISC/bind allowing changes to its listen_addresses equivilant through
> > dynamic DNS updates. Would it be possible to implement? Sure. Does it
> > make any sense? Certainly not.
>
> I very much want to change stuff like wal_level, listen_addresses and
> shared_buffers via ALTER SYSTEM. Configuration variables like that
> (PGC_POSTMASTER stuff mostly) are the prime reason why you actually need
> to change postgresql.conf instead of changing per user/database
> settings.
wal_level and shared_buffers I can buy, but listen_addresses? The most
typical change there is going from localhost -> '*', but you've got to
be on the box to do that. Anything else and you're going to need to be
adding interfaces to the box anyway and hacking around in
/etc/network/interfaces or what-have-you.
> > Because we've got crap mixed into postgresql.conf which are bootstrap
> > configs needed to get the system started. Those things, in my view
> > anyway, fall much more into the category of "resources which should be
> > managed outside the database" than pg_hba.conf.
>
> I think the problem with your position in this thread is that you want
> to overhaul the way our configuration works in a pretty radical
> way. Which is fair enough, there certainly are deficiencies. But it's
> not the topic of this thread.
You and Robert both seem to be of the opinion that this hack which
brings postgresql.conf into the database via ALTER SYSTEM is a-ok
because it's moving us "forward" in someone's mind, but it really is
developing a system configuration management system which *looks* like a
first-class citizen when it actually falls woefully short of that.
There is absolutely no question in my mind that this will be a huge
support pain, from the first "ALTER SYSTEM SET shared_buffers = blah;
SHOW shared_buffers;" to the "why can't my database start?!? it's
complaining it can't allocate memory but I keep changing postgresql.conf
and nothing works!" I'm simply not convinced that this is moving us
forward nor that we will end up with more benefit than pain from it.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Thu, Aug 29, 2013 at 9:12 PM, Stephen Frost <sfrost@snowman.net> wrote: > You will not find me argueing to allow that in normal operation, or most > direct-catalog hacking. I'd be much happier if we had all the ALTER, > etc, options necessary to prevent any need to ever touch the catalogs. > Similairly, it'd be nice to have more permission options which reduce > the need for anyone to be superuser. Sure, I agree. But you can't burden this patch with the job of conforming to what you and I think the project policy ought to be, but isn't. Right now, as things stand today, the superuser is one step short of God Almighty. The only way in which we even attempt to restrict the superuser is to try to keep her from hijacking the postgres login account. But we don't even try to do that very thoroughly, as has recently been pointed out on other threads; there are several plausible attack vectors for her to do just that. This patch fits right into that existing philosophy. If we take this patch, and I think we should, and later change our policy, then the permissions around this may require adjustment to fit the new policy. Fine! But that policy change is properly a separate discussion. >> Now, you can argue that people are more likely to render the database >> nonfunctional by changing GUC settings than they are to do it by >> corrupting the system catalogs, but I'm not sure I believe it. We >> can, after all, validate that any setting a user supplies is a valid >> value for that setting before writing it out to the configuration >> file. It might still make the system fail to start if - for example - >> it causes too many semaphores to be reserved, or something like that. >> But why should we think that such mistakes will be common? If >> anything, it sounds less error-prone to me than hand-editing the >> configuration file, where typing something like on_exit_error=maybe >> will make the server fail to start. We can eliminate that whole >> category of error, at least for people who choose to use the new >> tools. > > That category of error is much more likely to get caught by the sysadmin > doing the restart after the update.. If you edit postgresql.conf manually today, you will have no warning of ANY sort of error you might make. With a feature like this, we can catch a large subset of things you might do wrong *before we even modify the file*. Yes, there will be some that slip through, but an imperfect mechanism for catching mistakes is better than no mechanism at all. > ALTER SYSTEM doesn't provide any > way to restart the DB to make sure it worked. > That, in turn, encourages > modification of the config parameters w/o a restart, which is a pretty > bad idea, imv. "vi" doesn't provide any way to restart the DB to make sure it worked, either, and never will. However, ALTER SYSTEM could be extended in exactly that way: we could have ALTER SYSTEM RESTART. Of course some people are likely to argue that's a bad idea, so we'd have to have that discussion and think carefully about the issues, but there's nothing intrinsic that restricts us from giving ALTER SYSTEM any arbitrary set of capabilities we might want it to have. Even if we don't ever do that, we're no worse off than today. Well, with one exception. If we make it easier to modify the configuration file, then people are more likely to do it, and thus more likely to do it wrong. But you could use that argument against any change that improves ease-of-use. And ease-of-use is a feature, not a bug. >> If you're using the term "dangerous" to refer to a security exposure, >> I concede there is some incremental exposure from allowing this by >> default. But it's not a terribly large additional exposure. It's >> already the case that if adminpack is available the super-user can do >> whatever he or she wants, because she or he can write to arbitrary >> files inside the data directory. > > This is only true if -contrib is installed, which a *lot* of admins > refuse to do, specifically because of such insane modules as this. > Therefore, I would argue that it's not nearly as small a change wrt > exposure as you believe. Strictly speaking, I don't believe it's a huge > security hole or something, but I do think it's going to surprise admins > who aren't following the lists. If we take the patch, it will be in the release notes, just like any other feature. I suspect there will be more people *pleasantly* surprised than anything else. If there's anything I've learned about ease-of-use as a 10+-year veteran of PostgreSQL, it's that being able to do everything via a database connection is really, really, really convenient. That's why foreign data wrappers are awesome. Instead of installing a foreign data wrapper for Oracle and pointing it at an Oracle server and then sucking data down over the link to store or process or whatever, you put all that logic in a client which knows how to talk to both PostgreSQL and Oracle. It turns out, though, that having that capability *in the server* is really valuable to people. It doesn't let you do anything you couldn't do otherwise, but it's extremely convenient. It's also why we keep hearing requests to put a scheduler in core. It's not that you *can't* schedule tasks using cron or Scheduled Tasks on Windows; it's that it's much more convenient to be able to do it all in one place, namely PostgreSQL. It is 100% possible for us to tell all of our users, sorry, you've got to keep using vi to change configuration settings, because we don't trust you to do that via SQL. But I think it's the WRONG decision. Users like being trusted. And it's not that hard for the DBA to disable this or any other feature if he or she feels a real need to do so. > This wouldn't be the first time we've said "do it externally and when > we figure out a better answer then we'll put it into core." No, but I don't think there's anything terribly wrong with *this* answer. It's the product of a lot of people hashing out a lot of issues, and I think it's pretty good. Pushing things out externally is appropriate when the best design is unclear and especially when little or no core support is required, but I think we have a pretty broad consensus on the general contours of how this should work, and some core support is needed. Someone could certainly do something better than adminpack without core integration, but what's being proposed here is better still. >> 1. disabling the feature by default, and providing no way for it to be >> turned on remotely, and > > Yes, that's quite intentional because it would make someone knowledgable > about the *overall system* be aware of this rather than having it sprung > on them when they discover some change has made the database unable to > start. We add new features in every release. I don't see that this one is dangerous enough that it needs to be hedged about with disclaimers we don't attach to any of our other features. My favorite example is the release where we changed the default behavior of TRUNCATE so that it cascades to child tables instead of affecting only the parent. Talk about springing changes on people that could have catastrophic consequences. People have to read the release notes before upgrading to new major versions, or they will get surprised. That's why we have release notes, and that's why we have stable minor versions. I would also just like to note in passing that I believe the shared memory changes in 9.3 eliminate the #1 reason for "my database won't start after a config change", which IME has always been that the change pushed up the shared memory allocation just beyond what the OS would allow. That hazard affected a large number of different configuration parameters, and it doesn't exist any more. You can still bump max_connections or max_autovacuum_workers or the new max_worker_processes high enough to eat up all the system's semaphores, and I'm sure someone has killed their system that way before, but I've personally not run into it. You might also increase the size of shared memory so much that you actually overrun what's available on the box, rather than just overrunning the limit on System V shared memory, but that requires a much larger adjustment to the parameter values and is therefore likely to be the result of abject stupidity rather than simple bad luck. > While I appreciate that the original thrust behind this may be trying to > replace adminpack, I find that module to be utterly horrendous in what > it does and if we don't replicate all that it can do, so much the > better. Consider a similar proposal as ALTER SYSTEM which simply > rewrites pg_hba.conf, ala how adminpack could. Would we punt on the > hard questions around things like how to specify the order in which > entries in pg_hba.conf are maintained? We could have an ALTER HBA today > which takes a line number to insert a string of text into, but is it a > good idea or design? Certainly not. My point here is simply that > focusing on replacing adminpack is not the basis for a feature or > design. I partially agree with that, but the point of mentioning adminpack is that the horse is already halfway out of the barn. I think this feature is pretty good as designed and not half-baked at all, but even if you feel otherwise it's surely a heck of a lot better than what exists today. >> #3 is a bad idea in my >> book - it would break some of my existing benchmarking scripts, which >> do initdb; cat >>$PGDATA/postgresql.conf <<EOM. But even if it were a >> good idea, it isn't a necessary prerequisite for this patch. > > Too often do we consider sloppy handling like this, which is great for a > dev or testing environment, to be appropriate for production systems. :( Well, IMHO, the solution to that is to stop relying on hand-edited files. I've never really liked the fact that postgresql.conf is a text file rather than somehow part of the database guts, requiring a special tool (that can do the appropriate validation) to make changes.But that ship has already sailed and doesn't seem likelyto reverse course any time soon. ...Robert
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Andres Freund (andres@2ndquadrant.com) wrote:
> On 2013-08-29 21:26:48 -0400, Stephen Frost wrote:
> > It's not the OPs guy that I'm worried about using ALTER SYSTEM- I don't
> > expect them to have any clue about it or care about it, except where it
> > can be used to modify things under /etc which they, rightfully, consider
> > their domain.
>
> I think for the scenarios you describe it makes far, far much more sense
> to add the ability to easily monitor for two things:
> * on-disk configuration isn't the same as the currently loaded (not
> trivially possible yet)
That would certainly help. I don't know that it needs to be technically
'trivial', but at least having check_postgres.pl or something which can
alert on it would be an improvement. Clearly, that would be valuable
today (assuming it doesn't already exist somewhere.. it might).
> * Configuration variables only come from locations that are approved for
> in your scenario (Already possible, we might want to make it even easier)
That an interesting notion; do you have something specific in mind..?
The easiest, imv anyway, would be that options set in postgresql.conf
can't be overridden, but that gets us into the bootstrap problem that
people seem to be concerned about. It would also be a change to how
postgresql.conf is parsed today which some people would be annoyed by.
Having some configuration option which says what can be modified by
alter system doesn't strike me as a terribly good solution either.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-30 09:19:42 -0400, Stephen Frost wrote: > * Andres Freund (andres@2ndquadrant.com) wrote: > > On 2013-08-30 08:48:21 -0400, Stephen Frost wrote: > > > I really just don't buy that- I've already put forward suggestions for > > > how to deal with it, but no one here seems to understand the > > > distinction. Modifying listen_addresses through ALTER SYSTEM is akin to > > > ISC/bind allowing changes to its listen_addresses equivilant through > > > dynamic DNS updates. Would it be possible to implement? Sure. Does it > > > make any sense? Certainly not. > > > > I very much want to change stuff like wal_level, listen_addresses and > > shared_buffers via ALTER SYSTEM. Configuration variables like that > > (PGC_POSTMASTER stuff mostly) are the prime reason why you actually need > > to change postgresql.conf instead of changing per user/database > > settings. > > wal_level and shared_buffers I can buy, but listen_addresses? The most > typical change there is going from localhost -> '*', but you've got to > be on the box to do that. Anything else and you're going to need to be > adding interfaces to the box anyway and hacking around in > /etc/network/interfaces or what-have-you. Even if it requires to be on the box locally, I only need libpq for it. And it's not infrequent to allow additional, already configured interfaces. And even if not, what's the point of prohibiting listen_interfaces specifically? When the other very interesting variables have the same dangers? Doing this on a variable-by-variable basis will a) be a ridiculous amount of effort, b) confuse users which may not share our judgement of individual variables. > > > Because we've got crap mixed into postgresql.conf which are bootstrap > > > configs needed to get the system started. Those things, in my view > > > anyway, fall much more into the category of "resources which should be > > > managed outside the database" than pg_hba.conf. > > > > I think the problem with your position in this thread is that you want > > to overhaul the way our configuration works in a pretty radical > > way. Which is fair enough, there certainly are deficiencies. But it's > > not the topic of this thread. > > You and Robert both seem to be of the opinion that this hack which > brings postgresql.conf into the database via ALTER SYSTEM is a-ok > because it's moving us "forward" in someone's mind, but it really is > developing a system configuration management system which *looks* like a > first-class citizen when it actually falls woefully short of that. It's what plenty of people want and it doesn't hurt people who do not want it. Yes. I think that's a step forward. > There is absolutely no question in my mind that this will be a huge > support pain, from the first "ALTER SYSTEM SET shared_buffers = blah; > SHOW shared_buffers;" to the "why can't my database start?!? it's > complaining it can't allocate memory but I keep changing postgresql.conf > and nothing works!" I'm simply not convinced that this is moving us > forward nor that we will end up with more benefit than pain from it. That will not show the changed shared_buffers. And it (afair) will throw a WARNING that shared_buffers couldn't be adjusted at this point. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Thu, Aug 29, 2013 at 9:26 PM, Stephen Frost <sfrost@snowman.net> wrote: >> In the first place, modifying postgresql.conf and not immediately >> restarting the server to test your changes is probably the single most >> basic DBA error imaginable. > > You're not modifying postgresql.conf with ALTER SYSTEM, now are you? > Admins are going to realize the need to restart or at least reload the > service after updating such, but a DBA who has focused mainly on > normalization, query optimization, etc, isn't going to have the same > experience that a sysadmin has. I refuse to reject any feature on the basis of the possibility that people might use it without reading the documentation. Nearly anything anyone will ever propose is so idiot-proof that we can't conceive of a scenario in which someone shoots their foot off with it.The question is whether the usefulness of the featureexceeds, by a sufficient amount, the harm it's likely to do, and I think the answer is clearly yes. > A DBA updating a GUC, something they are likely to do frequently > day-in and day-out, isn't necessairly going to consider that it's a > reboot-needing change. Indeed, it's not unreasonable to consider that > we may, some day, actually be able to change or increase > shared_buffers on the fly (or error in the trying); I'd think your > work with the dynamic backends would actually make that likely to > happen sooner rather than later. I wouldn't hold my breath. We have way too many separate, variable-length data structures in shared memory. Increasing shared_buffers means making the lwlock array bigger, making the buffer header array bigger, allocating more space for the buffer mapping hash, etc. I doubt we'd accept a design that involves each of those data structures being a separate shared memory segment, but if they're all in the same segment one after another, then you can't easily extend them on the fly. There are also a lot of problems around unmapping-and-remapping a segment. If the remap fails, it's going to be at least a FATAL, but if you had any shared memory state in the unmapped segment (like a buffer pin) that has to be unwound, you have to PANIC; there's no other way to fix it. My initial design for dynamic shared memory allowed for an unbounded number of dynamic shared memory segments by growing the control segment on the fly. However, this introduced PANIC hazards in corner cases that I couldn't get rid of, so now there's a fairly generous but fixed-at-startup-time limit on how many segments you can have. In practice I don't think this matters much, but it was a sobering reminder that the main shared memory segment, with all of its inflexibility, has important reliability properties that are hard to replicate in more dynamic scenarios. > It's not the OPs guy that I'm worried about using ALTER SYSTEM- I don't > expect them to have any clue about it or care about it, except where it > can be used to modify things under /etc which they, rightfully, consider > their domain. Under the currently-proposed design, it can't be used to do any such thing. It can only be used to modify some auto.conf file which lives in $PGDATA. It's therefore no different from the ops perspective than ALTER DATABASE or ALTER USER - except that it allows all settings to be changed rather than only a subset. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-30 09:26:32 -0400, Stephen Frost wrote: > * Andres Freund (andres@2ndquadrant.com) wrote: > > On 2013-08-29 21:26:48 -0400, Stephen Frost wrote: > > > It's not the OPs guy that I'm worried about using ALTER SYSTEM- I don't > > > expect them to have any clue about it or care about it, except where it > > > can be used to modify things under /etc which they, rightfully, consider > > > their domain. > > > > I think for the scenarios you describe it makes far, far much more sense > > to add the ability to easily monitor for two things: > > * on-disk configuration isn't the same as the currently loaded (not > > trivially possible yet) > > That would certainly help. I don't know that it needs to be technically > 'trivial', but at least having check_postgres.pl or something which can > alert on it would be an improvement. Clearly, that would be valuable > today (assuming it doesn't already exist somewhere.. it might). Technically trivial in the sense that it should be queryable from SQL without having to write code in an untrusted PL ;). I guess storing the file modification date along the file/location a GUC is originating from would be good enough. Then you could write a query using pg_stat_file() to make sure they are up2date. > > * Configuration variables only come from locations that are approved for > > in your scenario (Already possible, we might want to make it even easier) > > That an interesting notion; do you have something specific in mind..? > The easiest, imv anyway, would be that options set in postgresql.conf > can't be overridden, but that gets us into the bootstrap problem that > people seem to be concerned about. It would also be a change to how > postgresql.conf is parsed today which some people would be annoyed by. > Having some configuration option which says what can be modified by > alter system doesn't strike me as a terribly good solution either. I think changing the precedence of options in postgresql.conf has about zero chance. Currently you can do something like: SELECT name FROM pg_settings WHERE sourcefile IS NOT NULL AND sourcefile <> '/etc/postgresql/9.2/main/postgresql.conf'; To be easily usable we should have two new columns in pg_settings: * reset_sourcefile * reset_sourceline And an easy way to get the absolute path of * postgresql.conf * auto.conf or whatever we name it. That would make it possible to easily write a query that works across intallation that warns about any values stored in auto.conf, even if they are overwritten by a per-user config or similar. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Fri, Aug 30, 2013 at 8:53 AM, Stephen Frost <sfrost@snowman.net> wrote: > Yes, one of the issues with the existing system is that you can't > specify a default to be applied to new roles. Also, there are > parameters which are not per-role yet which it probably makes sense to > be in the database and we'd need a way to deal with that. Although, at > the same time, considering it role based does make for a nice > distinction. Unfortunately, those clammoring for this will argue that > it wouldn't replace adminpack and that they couldn't use it to modify > their /etc/network/interfaces file, which is the obvious next step to > all of this. This is a straw-man. adminpack doesn't allow reading or writing files outside of the configured data and log directories, as a security precaution. But suppose it did. If your permissions on /etc/network/interfaces allow the postgres user to modify it, then you pretty much deserve exactly what you get. Likening this to the feature being proposed is silly. What is being asked for here is the ability to easily modify system-wide settings from the SQL prompt, just as today you can modify settings per-user or per-database. That's not the same thing as rewriting the entire system configuration. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Andres Freund (andres@2ndquadrant.com) wrote:
> On 2013-08-30 09:19:42 -0400, Stephen Frost wrote:
> > wal_level and shared_buffers I can buy, but listen_addresses? The most
> > typical change there is going from localhost -> '*', but you've got to
> > be on the box to do that. Anything else and you're going to need to be
> > adding interfaces to the box anyway and hacking around in
> > /etc/network/interfaces or what-have-you.
>
> Even if it requires to be on the box locally, I only need libpq for
> it. And it's not infrequent to allow additional, already configured
> interfaces. And even if not, what's the point of prohibiting
> listen_interfaces specifically? When the other very interesting
> variables have the same dangers?
I'd like to see those dangers removed from the other very interesting
variables. We're making progress towards that, for example with
shared_buffers. I've commented on that already up-thread. Hell, you
could even make things such that PG would start w/ a misconfigured
listen_addresses- but if we don't like that then I would argue that it
shouldn't be included here. I'm not entirely sure how wal_level has the
same danger as the others..
> Doing this on a variable-by-variable basis will a) be a ridiculous
> amount of effort, b) confuse users which may not share our judgement of
> individual variables.
I don't think the effort involved is nearly as much as you claim and we
already have the issue that users don't like our choices around what can
be reconfigured on the fly and what can't (perhaps they understand that
there are technical challenges to some of them, but that doesn't make
them agree with them).
> > You and Robert both seem to be of the opinion that this hack which
> > brings postgresql.conf into the database via ALTER SYSTEM is a-ok
> > because it's moving us "forward" in someone's mind, but it really is
> > developing a system configuration management system which *looks* like a
> > first-class citizen when it actually falls woefully short of that.
>
> It's what plenty of people want and it doesn't hurt people who do not
> want it. Yes. I think that's a step forward.
It will be quite interesting to see if people decide they really wanted
this once they actually *get* it.
> > There is absolutely no question in my mind that this will be a huge
> > support pain, from the first "ALTER SYSTEM SET shared_buffers = blah;
> > SHOW shared_buffers;" to the "why can't my database start?!? it's
> > complaining it can't allocate memory but I keep changing postgresql.conf
> > and nothing works!" I'm simply not convinced that this is moving us
> > forward nor that we will end up with more benefit than pain from it.
>
> That will not show the changed shared_buffers. And it (afair) will throw
> a WARNING that shared_buffers couldn't be adjusted at this point.
Not showing the change is what I was getting at. As has been said
elsewhere, throwing a warning on every interesting invokation of a
command can speak to it not being exactly 'ready'.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Robert Haas (robertmhaas@gmail.com) wrote:
> there's a fairly generous but fixed-at-startup-time limit on how many
> segments you can have. In practice I don't think this matters much,
> but it was a sobering reminder that the main shared memory segment,
> with all of its inflexibility, has important reliability properties
> that are hard to replicate in more dynamic scenarios.
Why wouldn't it be possible to have the same arrangment for
shared_buffers, where you have more entires than you 'need' at startup
but which then allows you to add more shared segments later? I do see
that we would need an additional bit of indirection to handle that,
which might be too much overhead, but the concept seems possible. Isn't
that more-or-less how the kernel handles dynamic memory..?
> Under the currently-proposed design, it can't be used to do any such
> thing. It can only be used to modify some auto.conf file which lives
> in $PGDATA. It's therefore no different from the ops perspective than
> ALTER DATABASE or ALTER USER - except that it allows all settings to
> be changed rather than only a subset.
Claiming that modifying a file *included from a file in /etc* doesn't
modify things under /etc is disingenuous, imv.
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Andres Freund
Date:
On 2013-08-30 09:43:01 -0400, Stephen Frost wrote: > * Andres Freund (andres@2ndquadrant.com) wrote: > > On 2013-08-30 09:19:42 -0400, Stephen Frost wrote: > > > wal_level and shared_buffers I can buy, but listen_addresses? The most > > > typical change there is going from localhost -> '*', but you've got to > > > be on the box to do that. Anything else and you're going to need to be > > > adding interfaces to the box anyway and hacking around in > > > /etc/network/interfaces or what-have-you. > > > > Even if it requires to be on the box locally, I only need libpq for > > it. And it's not infrequent to allow additional, already configured > > interfaces. And even if not, what's the point of prohibiting > > listen_interfaces specifically? When the other very interesting > > variables have the same dangers? > > I'd like to see those dangers removed from the other very interesting > variables. We're making progress towards that, for example with > shared_buffers. I've commented on that already up-thread. Hell, you > could even make things such that PG would start w/ a misconfigured > listen_addresses- but if we don't like that then I would argue that it > shouldn't be included here. Yes, we're making progress. But that's an independent thing, partially constrained by implementation complexity, partially constrained by cross platform worries. You're free to work on reducing the dangers around other variables, but I don't understand in the very least why that should stop this feature. > I'm not entirely sure how wal_level has the same danger as the > others.. Try setting max_wal_senders = 10 and wal_level = minimal. > > > There is absolutely no question in my mind that this will be a huge > > > support pain, from the first "ALTER SYSTEM SET shared_buffers = blah; > > > SHOW shared_buffers;" to the "why can't my database start?!? it's > > > complaining it can't allocate memory but I keep changing postgresql.conf > > > and nothing works!" I'm simply not convinced that this is moving us > > > forward nor that we will end up with more benefit than pain from it. > > > > That will not show the changed shared_buffers. And it (afair) will throw > > a WARNING that shared_buffers couldn't be adjusted at this point. > > Not showing the change is what I was getting at. As has been said > elsewhere, throwing a warning on every interesting invokation of a > command can speak to it not being exactly 'ready'. Then pg isn't ready already, because it has done that for a long time. Set some PGC_POSTMASTER variable and reload the configuration. Guiding a user that to perform required further action doesn't imply nonreadyness in the least. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Fri, Aug 30, 2013 at 9:19 AM, Stephen Frost <sfrost@snowman.net> wrote: > You and Robert both seem to be of the opinion that this hack which > brings postgresql.conf into the database via ALTER SYSTEM is a-ok > because it's moving us "forward" in someone's mind, but it really is > developing a system configuration management system which *looks* like a > first-class citizen when it actually falls woefully short of that. It's not a system configuration management system, and it doesn't pretend to be. It's an analogue of ALTER USER and ALTER DATABASE that papers over their shortcomings, and a safer alternative to adminpack's kludgy way of enabling the same functionality. > There is absolutely no question in my mind that this will be a huge > support pain, from the first "ALTER SYSTEM SET shared_buffers = blah; > SHOW shared_buffers;" They'd have the same problem with ALTER USER SET work_mem = blah. You're acting like ALTER commands that don't take immediate effect are something brand-new, but we've had them for years. > to the "why can't my database start?!? it's > complaining it can't allocate memory but I keep changing postgresql.conf > and nothing works!" As of 9.3, that failure mode doesn't really happen any more, unless you maybe set shared_buffers equal to 100% of system memory. > I'm simply not convinced that this is moving us > forward nor that we will end up with more benefit than pain from it. Fair enough, but I'm not convinced that we'll derive any pain at all from it. The existing similar features haven't been a notable source of complaints AFAIK. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Andres Freund (andres@2ndquadrant.com) wrote:
> Technically trivial in the sense that it should be queryable from SQL
> without having to write code in an untrusted PL ;).
hah.
> I guess storing the file modification date along the file/location a GUC
> is originating from would be good enough. Then you could write a query
> using pg_stat_file() to make sure they are up2date.
Wouldn't you want to actually look at the GUC and see if the value is
different also? Just knowing that postgresql.conf changed doesn't mean
you want every value to say it's different... It's entirely possible
that the file was rewritten or touch'd but the configuration is
identical.
> > > * Configuration variables only come from locations that are approved for
> > > in your scenario (Already possible, we might want to make it even easier)
> >
> > That an interesting notion; do you have something specific in mind..?
> > The easiest, imv anyway, would be that options set in postgresql.conf
> > can't be overridden, but that gets us into the bootstrap problem that
> > people seem to be concerned about. It would also be a change to how
> > postgresql.conf is parsed today which some people would be annoyed by.
> > Having some configuration option which says what can be modified by
> > alter system doesn't strike me as a terribly good solution either.
>
> I think changing the precedence of options in postgresql.conf has about
> zero chance.
Sadly, you're probably right.
> That would make it possible to easily write a query that works across
> intallation that warns about any values stored in auto.conf, even if
> they are overwritten by a per-user config or similar.
I had the impression that 'approved for' above meant something which
actually *enforced* it rather than just another monitoring check..
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Robert Haas (robertmhaas@gmail.com) wrote:
> On Fri, Aug 30, 2013 at 8:53 AM, Stephen Frost <sfrost@snowman.net> wrote:
> > Yes, one of the issues with the existing system is that you can't
> > specify a default to be applied to new roles. Also, there are
> > parameters which are not per-role yet which it probably makes sense to
> > be in the database and we'd need a way to deal with that. Although, at
> > the same time, considering it role based does make for a nice
> > distinction. Unfortunately, those clammoring for this will argue that
> > it wouldn't replace adminpack and that they couldn't use it to modify
> > their /etc/network/interfaces file, which is the obvious next step to
> > all of this.
>
> This is a straw-man.
It was intended to be and I wasn't particularly expecting a response to
the sarcasm, at the same time..
> Likening this to the
> feature being proposed is silly. What is being asked for here is the
> ability to easily modify system-wide settings from the SQL prompt,
> just as today you can modify settings per-user or per-database.
/etc/network/interfaces would be considered part of the "system" by
some.. :) PG isn't an OS, yet..
> That's not the same thing as rewriting the entire system
> configuration.
And here you're using the *other* meaning of system, no? Glad there
won't be any confusion here! ;) (yes, I'm kidding..).
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Robert Haas
Date:
On Fri, Aug 30, 2013 at 9:49 AM, Stephen Frost <sfrost@snowman.net> wrote: > * Robert Haas (robertmhaas@gmail.com) wrote: >> there's a fairly generous but fixed-at-startup-time limit on how many >> segments you can have. In practice I don't think this matters much, >> but it was a sobering reminder that the main shared memory segment, >> with all of its inflexibility, has important reliability properties >> that are hard to replicate in more dynamic scenarios. > > Why wouldn't it be possible to have the same arrangment for > shared_buffers, where you have more entires than you 'need' at startup > but which then allows you to add more shared segments later? I do see > that we would need an additional bit of indirection to handle that, > which might be too much overhead, but the concept seems possible. Isn't > that more-or-less how the kernel handles dynamic memory..? Yeah, I think that something like would be possible, but the synchronization would be tricky, and it wouldn't work at all with System V shared memory or Windows shared memory, which apparently can't be resized after creation. Also, it would rely on the system administrator having a sensible setting for max_on_the_fly_shared_buffers and only having the wrong setting for initial_shared_buffers_until_i_change_it_later, which might not cover a very high percentage of the cases that arise in practice. >> Under the currently-proposed design, it can't be used to do any such >> thing. It can only be used to modify some auto.conf file which lives >> in $PGDATA. It's therefore no different from the ops perspective than >> ALTER DATABASE or ALTER USER - except that it allows all settings to >> be changed rather than only a subset. > > Claiming that modifying a file *included from a file in /etc* doesn't > modify things under /etc is disingenuous, imv. I think you're getting way too hung up on the fact that the proposed auto.conf will be stored as a flat file. From your comments upthread, I gather that you'd be rejoicing if it were a table. The only reason we're not doing that is because of the possibility that something in auto.conf might keep the server from starting. I don't think that's gonna be very common now that we're mostly out from under the System V shared memory limits, but we'll see. As you point out, it might also be worth fixing: maybe we can find some way to arrange things so that the server will always be able to start up regardless of how badly messed-up auto.conf is... but that's a different patch. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
* Robert Haas (robertmhaas@gmail.com) wrote:
> I think you're getting way too hung up on the fact that the proposed
> auto.conf will be stored as a flat file. From your comments upthread,
> I gather that you'd be rejoicing if it were a table.
I'd be happy if it was a table which managed an *independent* set of
parameters from those used to bootstrap the system, but no one seems to
like breaking up the options between "things that can be sanely modified
without other OS changes" and "things which require OS support".
Thanks,
Stephen
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Tom Lane
Date:
Stephen Frost <sfrost@snowman.net> writes:
> * Robert Haas (robertmhaas@gmail.com) wrote:
>> I think you're getting way too hung up on the fact that the proposed
>> auto.conf will be stored as a flat file. From your comments upthread,
>> I gather that you'd be rejoicing if it were a table.
> I'd be happy if it was a table which managed an *independent* set of
> parameters from those used to bootstrap the system, but no one seems to
> like breaking up the options between "things that can be sanely modified
> without other OS changes" and "things which require OS support".
I agree with Robert's comments upthread that if the new facility can't do
everything that can be done today by editing postgresql.conf, it's not
going to be adequate. So I'm not in favor of having two sets of
parameters. It's also not clear to me that we can make a reliable
distinction between parameters that can prevent a server restart vs those
that can't; or at least, the set of the latter will be much smaller than
one could wish.
regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Stephen Frost
Date:
Tom, all,
I'm not one to give up a fight (I hope that's something ya'll like about me ;), but in this case I'm gonna have to
concede. Clearly, I'm in the minority about this, at least on the lists and among the active hackers. Let me just say
thatI hope all the happy users of this will outweigh the complaints. I suppose there's only one way to find out.
Thanks!
Stephen
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > * Robert Haas (robertmhaas@gmail.com) wrote:
> >> I think you're getting way too hung up on the fact that the proposed
> >> auto.conf will be stored as a flat file. From your comments upthread,
> >> I gather that you'd be rejoicing if it were a table.
>
> > I'd be happy if it was a table which managed an *independent* set of
> > parameters from those used to bootstrap the system, but no one seems to
> > like breaking up the options between "things that can be sanely modified
> > without other OS changes" and "things which require OS support".
>
> I agree with Robert's comments upthread that if the new facility can't do
> everything that can be done today by editing postgresql.conf, it's not
> going to be adequate. So I'm not in favor of having two sets of
> parameters. It's also not clear to me that we can make a reliable
> distinction between parameters that can prevent a server restart vs those
> that can't; or at least, the set of the latter will be much smaller than
> one could wish.
>
> regards, tom lane
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Fri, Aug 30, 2013 at 8:24 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Stephen Frost <sfrost@snowman.net> writes: >> * Robert Haas (robertmhaas@gmail.com) wrote: >>> I think you're getting way too hung up on the fact that the proposed >>> auto.conf will be stored as a flat file. From your comments upthread, >>> I gather that you'd be rejoicing if it were a table. > >> I'd be happy if it was a table which managed an *independent* set of >> parameters from those used to bootstrap the system, but no one seems to >> like breaking up the options between "things that can be sanely modified >> without other OS changes" and "things which require OS support". > > I agree with Robert's comments upthread that if the new facility can't do > everything that can be done today by editing postgresql.conf, it's not > going to be adequate. So I'm not in favor of having two sets of > parameters. It's also not clear to me that we can make a reliable > distinction between parameters that can prevent a server restart vs those > that can't; or at least, the set of the latter will be much smaller than > one could wish. Now as we have an agreement, I had updated patch for below left issues: 1. ALTER SYSTEM SET should be constrained to only set known GUCs, this point has been discussed on another mail thread (http://www.postgresql.org/message-id/14857.1378523977@sss.pgh.pa.us) In function AlterSystemSetConfigFile(), when we try to get the record using find_option(), pass second parameter as false which will make sure if the parameter doesn't exist it will return NULL. 2. Some indentation issues. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Attachment
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Peter Eisentraut
Date:
On 9/28/13 3:05 AM, Amit Kapila wrote: > Now as we have an agreement, I had updated patch for below left issues: Regression tests are failing.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Mon, Sep 30, 2013 at 9:07 PM, Peter Eisentraut <peter_e@gmx.net> wrote: > On 9/28/13 3:05 AM, Amit Kapila wrote: >> Now as we have an agreement, I had updated patch for below left issues: > > Regression tests are failing. Thanks for informing. I am sorry for not running regression before sending patch. Reason for failure was that source for GUC in new function validate_conf_option() was hardcoded to PGC_S_FILE which was okay for Alter System, but not for SET path. I had added new parameter source in this function and get the value of source when this is called from SET path. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Attachment
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Haribabu kommi
Date:
On 01 October 2013 00:56 Amit Kapila wrote: > On Mon, Sep 30, 2013 at 9:07 PM, Peter Eisentraut <peter_e@gmx.net> > wrote: > > On 9/28/13 3:05 AM, Amit Kapila wrote: > >> Now as we have an agreement, I had updated patch for below left > issues: > > > > Regression tests are failing. > > Thanks for informing. I am sorry for not running regression before > sending patch. > > Reason for failure was that source for GUC in new function > validate_conf_option() was hardcoded to PGC_S_FILE which was okay for > Alter System, but > not for SET path. I had added new parameter source in this function > and get the value of source when this is called from SET path. Some of the initial observation of the patch are, 1. Patch is not applying against git head, needs a rebase. 2. Patch doesn't contain the tests. I started reviewing the patch, will share the details once I finish. Regards, Hari babu.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Wed, Nov 13, 2013 at 4:05 PM, Haribabu kommi <haribabu.kommi@huawei.com> wrote: > On 01 October 2013 00:56 Amit Kapila wrote: >> On Mon, Sep 30, 2013 at 9:07 PM, Peter Eisentraut <peter_e@gmx.net> >> wrote: >> > On 9/28/13 3:05 AM, Amit Kapila wrote: >> >> Now as we have an agreement, I had updated patch for below left >> issues: >> > >> > Regression tests are failing. >> >> Thanks for informing. I am sorry for not running regression before >> sending patch. >> >> Reason for failure was that source for GUC in new function >> validate_conf_option() was hardcoded to PGC_S_FILE which was okay for >> Alter System, but >> not for SET path. I had added new parameter source in this function >> and get the value of source when this is called from SET path. > > Some of the initial observation of the patch are, > 1. Patch is not applying against git head, needs a rebase. > 2. Patch doesn't contain the tests. It was intentional and as per feedback for this patch. As for testing this feature, we need to put sleep after operation, so it was suggested to remove tests. > I started reviewing the patch, will share the details once I finish. Thanks. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Wed, Nov 13, 2013 at 7:23 PM, Amit Kapila <amit.kapila16@gmail.com> wrote: > On Wed, Nov 13, 2013 at 4:05 PM, Haribabu kommi > <haribabu.kommi@huawei.com> wrote: >> On 01 October 2013 00:56 Amit Kapila wrote: >>> On Mon, Sep 30, 2013 at 9:07 PM, Peter Eisentraut <peter_e@gmx.net> >>> wrote: >>> > On 9/28/13 3:05 AM, Amit Kapila wrote: >>> >> Now as we have an agreement, I had updated patch for below left >>> issues: >>> > >> >> Some of the initial observation of the patch are, >> 1. Patch is not applying against git head, needs a rebase. Find the rebased version attached with this mail. I have removed header inclusions of port.h and lwlock.h from guc.c, it works fine without them. >> 2. Patch doesn't contain the tests. > It was intentional and as per feedback for this patch. As for > testing this feature, we need to put sleep after operation, so it was > suggested to remove tests. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Attachment
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Peter Eisentraut
Date:
On 11/14/13, 2:50 AM, Amit Kapila wrote: > Find the rebased version attached with this mail. Doesn't build: openjade -wall -wno-unused-param -wno-empty -wfully-tagged -D . -D . -c /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog-d stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml openjade:reference.sgml:61:3:E: cannot find "alter_system.sgml"; tried "ref/alter_system.sgml", "./alter_system.sgml", "./alter_system.sgml","/usr/local/share/sgml/alter_system.sgml", "/usr/share/sgml/alter_system.sgml" openjade:config.sgml:164:27:X: reference to non-existent ID "SQL-ALTERSYSTEM" make[3]: *** [HTML.index] Error 1 make[3]: *** Deleting file `HTML.index' osx -D. -x lower -i include-xslt-index postgres.sgml >postgres.xmltmp osx:reference.sgml:61:3:E: cannot find "alter_system.sgml"; tried "ref/alter_system.sgml", "./alter_system.sgml", "/usr/local/share/sgml/alter_system.sgml","/usr/share/sgml/alter_system.sgml" osx:config.sgml:164:27:X: reference to non-existent ID "SQL-ALTERSYSTEM" make[3]: *** [postgres.xml] Error 1 New file missing in patch?
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Fri, Nov 15, 2013 at 10:18 PM, Peter Eisentraut <peter_e@gmx.net> wrote: > On 11/14/13, 2:50 AM, Amit Kapila wrote: >> Find the rebased version attached with this mail. > > Doesn't build: > > openjade -wall -wno-unused-param -wno-empty -wfully-tagged -D . -D . -c /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog-d stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml > openjade:reference.sgml:61:3:E: cannot find "alter_system.sgml"; tried "ref/alter_system.sgml", "./alter_system.sgml","./alter_system.sgml", "/usr/local/share/sgml/alter_system.sgml", "/usr/share/sgml/alter_system.sgml" > openjade:config.sgml:164:27:X: reference to non-existent ID "SQL-ALTERSYSTEM" > make[3]: *** [HTML.index] Error 1 > make[3]: *** Deleting file `HTML.index' > osx -D. -x lower -i include-xslt-index postgres.sgml >postgres.xmltmp > osx:reference.sgml:61:3:E: cannot find "alter_system.sgml"; tried "ref/alter_system.sgml", "./alter_system.sgml", "/usr/local/share/sgml/alter_system.sgml","/usr/share/sgml/alter_system.sgml" > osx:config.sgml:164:27:X: reference to non-existent ID "SQL-ALTERSYSTEM" > make[3]: *** [postgres.xml] Error 1 > > New file missing in patch? Oops, missed the new sgml file in patch, updated patch to include it. Many thanks for checking it. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Attachment
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Haribabu kommi
Date:
On 16 November 2013 09:43 Amit Kapila wrote:
> On Fri, Nov 15, 2013 at 10:18 PM, Peter Eisentraut <peter_e@gmx.net>
> wrote:
> > On 11/14/13, 2:50 AM, Amit Kapila wrote:
> >> Find the rebased version attached with this mail.
> >
> > Doesn't build:
> >
> > openjade -wall -wno-unused-param -wno-empty -wfully-tagged -D . -D .
> -c /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d
> stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
> > openjade:reference.sgml:61:3:E: cannot find "alter_system.sgml";
> tried "ref/alter_system.sgml", "./alter_system.sgml",
> "./alter_system.sgml", "/usr/local/share/sgml/alter_system.sgml",
> "/usr/share/sgml/alter_system.sgml"
> > openjade:config.sgml:164:27:X: reference to non-existent ID "SQL-
> ALTERSYSTEM"
> > make[3]: *** [HTML.index] Error 1
> > make[3]: *** Deleting file `HTML.index'
> > osx -D. -x lower -i include-xslt-index postgres.sgml >postgres.xmltmp
> > osx:reference.sgml:61:3:E: cannot find "alter_system.sgml"; tried
> "ref/alter_system.sgml", "./alter_system.sgml",
> "/usr/local/share/sgml/alter_system.sgml",
> "/usr/share/sgml/alter_system.sgml"
> > osx:config.sgml:164:27:X: reference to non-existent ID "SQL-
> ALTERSYSTEM"
> > make[3]: *** [postgres.xml] Error 1
> >
> > New file missing in patch?
>
> Oops, missed the new sgml file in patch, updated patch to include it.
> Many thanks for checking it.
1. Patch applied properly
2. No warnings in the compilation
3. Regress test passed.
4. Basic tests are passed.
Please find review comments:
+ * ALTER SYSTEM SET
+ *
+ * Command to edit postgresql.conf
+ *****************************************************************************/
I feel it should be "Command to change the configuration parameter"
because this command is not edits the postgresql.conf file.
ereport(ERROR, (errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("configurationfile \"%s\" contains errors",
- ConfigFileName)));
+ ErrorConfFile)));
The ErrorConfFile prints "postgresql.auto.conf" only if there is any parsing problem
with postgresql.auto.conf otherwise it always print "postgresql.conf" because of any other error.
+ * A stale temporary file may be left behind in case we crash.
+ * Such files are removed on the next server restart.
The above comment is wrong, the stale temporary file will be used
in the next ALTER SYSTEM command. I didn't find any code where it gets
deleted on the next server restart.
if any postmaster setting which are set by the alter system command which
leads to failure of server start, what is the solution to user to proceed
further to start the server. As it is mentioned that the auto.conf file
shouldn't be edited manually.
Regards,
Hari babu.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Sat, Nov 16, 2013 at 4:35 PM, Haribabu kommi
<haribabu.kommi@huawei.com> wrote:
> On 16 November 2013 09:43 Amit Kapila wrote:
>> On Fri, Nov 15, 2013 at 10:18 PM, Peter Eisentraut <peter_e@gmx.net>
>> wrote:
>> > On 11/14/13, 2:50 AM, Amit Kapila wrote:
>> >> Find the rebased version attached with this mail.
>> >
> Please find review comments:
>
> + * ALTER SYSTEM SET
> + *
> + * Command to edit postgresql.conf
> + *****************************************************************************/
>
> I feel it should be "Command to change the configuration parameter"
> because this command is not edits the postgresql.conf file.
Changed the comment, but I think it is better to use persistently
in line suggested by you, so I have done that way.
> ereport(ERROR,
> (errcode(ERRCODE_CONFIG_FILE_ERROR),
> errmsg("configuration file \"%s\" contains errors",
> - ConfigFileName)));
> + ErrorConfFile)));
>
> The ErrorConfFile prints "postgresql.auto.conf" only if there is any parsing problem
> with postgresql.auto.conf otherwise it always print "postgresql.conf" because of any other error.
Changed to ensure ErrorConfFile contains proper config file name.
Note: I have not asssigned file name incase of error in below loop,
as file name in gconf is NULL in most cases and moreover this loops
over
guc_variables which doesn't contain values/parameters from
auto.conf. So I don't think it is required to assign ErrorConfFile in
this loop.
ProcessConfigFile(GucContext context)
{
..
for (i = 0; i < num_guc_variables; i++)
{
struct config_generic *gconf = guc_variables[i];
..
}
>
> + * A stale temporary file may be left behind in case we crash.
> + * Such files are removed on the next server restart.
>
> The above comment is wrong, the stale temporary file will be used
> in the next ALTER SYSTEM command. I didn't find any code where it gets
> deleted on the next server restart.
Removed the comment from top of function and modified the comment
where file is getting opened.
>
> if any postmaster setting which are set by the alter system command which
> leads to failure of server start, what is the solution to user to proceed
> further to start the server. As it is mentioned that the auto.conf file
> shouldn't be edited manually.
Yeah, but in case of emergency user can change it to get server
started. Now the question is whether to mention it in documentation, I
think we can leave this decision to committer. If he thinks that it is
better to document then I will update it.
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
Attachment
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Haribabu kommi
Date:
On 17 November 2013 12:25 Amit Kapila wrote:
> On Sat, Nov 16, 2013 at 4:35 PM, Haribabu kommi
> <haribabu.kommi@huawei.com> wrote:
> > On 16 November 2013 09:43 Amit Kapila wrote:
> >> On Fri, Nov 15, 2013 at 10:18 PM, Peter Eisentraut <peter_e@gmx.net>
> >> wrote:
> >> > On 11/14/13, 2:50 AM, Amit Kapila wrote:
> >> >> Find the rebased version attached with this mail.
> >> >
> > ereport(ERROR,
> >
> (errcode(ERRCODE_CONFIG_FILE_ERROR),
> > errmsg("configuration file
> \"%s\" contains errors",
> > -
> ConfigFileName)));
> > +
> > + ErrorConfFile)));
> >
> > The ErrorConfFile prints "postgresql.auto.conf" only if there is any
> > parsing problem with postgresql.auto.conf otherwise it always print
> "postgresql.conf" because of any other error.
>
> Changed to ensure ErrorConfFile contains proper config file name.
> Note: I have not asssigned file name incase of error in below loop,
> as file name in gconf is NULL in most cases and moreover this loops
> over
> guc_variables which doesn't contain values/parameters from
> auto.conf. So I don't think it is required to assign ErrorConfFile in
> this loop.
>
> ProcessConfigFile(GucContext context)
> {
> ..
> for (i = 0; i < num_guc_variables; i++)
> {
> struct config_generic *gconf = guc_variables[i];
>
> ..
> }
Code changes are fine.
If two or three errors are present in the configuration file, it prints the last error
Configuration parameter file only. Is it required to be mentioned in the documentation?
> >
> > if any postmaster setting which are set by the alter system command
> > which leads to failure of server start, what is the solution to user
> > to proceed further to start the server. As it is mentioned that the
> > auto.conf file shouldn't be edited manually.
>
> Yeah, but in case of emergency user can change it to get server started.
> Now the question is whether to mention it in documentation, I think we
> can leave this decision to committer. If he thinks that it is better to
> document then I will update it.
Ok fine.
Regards,
Hari babu.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Mon, Nov 18, 2013 at 6:28 PM, Haribabu kommi
<haribabu.kommi@huawei.com> wrote:
> On 17 November 2013 12:25 Amit Kapila wrote:
>> On Sat, Nov 16, 2013 at 4:35 PM, Haribabu kommi
>> >> >> Find the rebased version attached with this mail.
>> >> >
>> > ereport(ERROR,
>> >
>> (errcode(ERRCODE_CONFIG_FILE_ERROR),
>> > errmsg("configuration file
>> \"%s\" contains errors",
>> > -
>> ConfigFileName)));
>> > +
>> > + ErrorConfFile)));
>> >
>> > The ErrorConfFile prints "postgresql.auto.conf" only if there is any
>> > parsing problem with postgresql.auto.conf otherwise it always print
>> "postgresql.conf" because of any other error.
>>
>> Changed to ensure ErrorConfFile contains proper config file name.
>> Note: I have not asssigned file name incase of error in below loop,
>> as file name in gconf is NULL in most cases and moreover this loops
>> over
>> guc_variables which doesn't contain values/parameters from
>> auto.conf. So I don't think it is required to assign ErrorConfFile in
>> this loop.
>>
>> ProcessConfigFile(GucContext context)
>> {
>> ..
>> for (i = 0; i < num_guc_variables; i++)
>> {
>> struct config_generic *gconf = guc_variables[i];
>>
>> ..
>> }
>
> Code changes are fine.
> If two or three errors are present in the configuration file, it prints the last error
> Configuration parameter file only. Is it required to be mentioned in the documentation?
Do you mean to say parsing errors or some run-time error, could you
explain with example?
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Haribabu kommi
Date:
On 18 November 2013 20:01 Amit Kapila wrote:
> On Mon, Nov 18, 2013 at 6:28 PM, Haribabu kommi
> <haribabu.kommi@huawei.com> wrote:
> > On 17 November 2013 12:25 Amit Kapila wrote:
> >> On Sat, Nov 16, 2013 at 4:35 PM, Haribabu kommi
> >> >> >> Find the rebased version attached with this mail.
> >> >> >
> >> > ereport(ERROR,
> >> >
> >> (errcode(ERRCODE_CONFIG_FILE_ERROR),
> >> > errmsg("configuration
> file
> >> \"%s\" contains errors",
> >> > -
> >> ConfigFileName)));
> >> > +
> >> > + ErrorConfFile)));
> >> >
> >> > The ErrorConfFile prints "postgresql.auto.conf" only if there is
> >> > any parsing problem with postgresql.auto.conf otherwise it always
> >> > print
> >> "postgresql.conf" because of any other error.
> >>
> >> Changed to ensure ErrorConfFile contains proper config file name.
> >> Note: I have not asssigned file name incase of error in below
> >> loop, as file name in gconf is NULL in most cases and moreover this
> >> loops over
> >> guc_variables which doesn't contain values/parameters
> >> from auto.conf. So I don't think it is required to assign
> >> ErrorConfFile in this loop.
> >>
> >> ProcessConfigFile(GucContext context) { ..
> >> for (i = 0; i < num_guc_variables; i++)
> >> {
> >> struct config_generic *gconf = guc_variables[i];
> >>
> >> ..
> >> }
> >
> > Code changes are fine.
> > If two or three errors are present in the configuration file, it
> prints the last error
> > Configuration parameter file only. Is it required to be mentioned in
> the documentation?
>
> Do you mean to say parsing errors or some run-time error, could you
> explain with example?
LOG: parameter "shared_buffers" cannot be changed without restarting the server
LOG: parameter "port" cannot be changed without restarting the server
LOG: configuration file "/home/hari/installation/bin/../../data/postgresql.auto.conf" contains errors; unaffected
changeswere applied
The shared_buffers parameter is changed in postgresql.conf and port is changed in postgresql.auto.conf.
The error file displays the last error occurred file.
Is it required to mention the above behavior in the document?
Regards,
Hari babu
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Mon, Nov 18, 2013 at 8:31 PM, Haribabu kommi <haribabu.kommi@huawei.com> wrote: > On 18 November 2013 20:01 Amit Kapila wrote: >> > Code changes are fine. >> > If two or three errors are present in the configuration file, it >> prints the last error >> > Configuration parameter file only. Is it required to be mentioned in >> the documentation? >> >> Do you mean to say parsing errors or some run-time error, could you >> explain with example? > > LOG: parameter "shared_buffers" cannot be changed without restarting the server > LOG: parameter "port" cannot be changed without restarting the server > LOG: configuration file "/home/hari/installation/bin/../../data/postgresql.auto.conf" contains errors; unaffected changeswere applied > > The shared_buffers parameter is changed in postgresql.conf and port is changed in postgresql.auto.conf. > The error file displays the last error occurred file. This is only possible if user tries to change configuration parameters both by Alter System command and manually as well and the case above is not an error, it is just an information for user that there are some parameters changed in config file which can only get reflected after server restart. So definitely user can check log files to see which parameter's doesn't get reflected if he is expecting some parameter to be changed, but it's not changed. I think here even in logs if last file containing errors is mentioned is not a big problem. > Is it required to mention the above behavior in the document? It is better to show in log both the files rather than documenting it, if the the above case is helpful for user which I don't think so, also to handle case in code can complicate the error handling path of code a bit. So I think we can leave this case as-is. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Haribabu kommi
Date:
On 19 November 2013 09:59 Amit Kapila wrote: > On Mon, Nov 18, 2013 at 8:31 PM, Haribabu kommi > <haribabu.kommi@huawei.com> wrote: > > On 18 November 2013 20:01 Amit Kapila wrote: > >> > Code changes are fine. > >> > If two or three errors are present in the configuration file, it > >> prints the last error > >> > Configuration parameter file only. Is it required to be mentioned > >> > in > >> the documentation? > >> > >> Do you mean to say parsing errors or some run-time error, could you > >> explain with example? > > > > LOG: parameter "shared_buffers" cannot be changed without restarting > > the server > > LOG: parameter "port" cannot be changed without restarting the > server > > LOG: configuration file > > "/home/hari/installation/bin/../../data/postgresql.auto.conf" > contains > > errors; unaffected changes were applied > > > > The shared_buffers parameter is changed in postgresql.conf and port > is changed in postgresql.auto.conf. > > The error file displays the last error occurred file. > > This is only possible if user tries to change configuration > parameters both by Alter System command and manually as well and the > case above is > not an error, it is just an information for user that there are some > parameters changed in config file which can only get reflected after > server restart. > So definitely user can check log files to see which parameter's > doesn't get reflected if he is expecting some parameter to be changed, > but it's not > changed. I think here even in logs if last file containing errors is > mentioned is not a big problem. > > > Is it required to mention the above behavior in the document? > > It is better to show in log both the files rather than documenting > it, if the the above case is helpful for user which I don't think so, > also to handle case > in code can complicate the error handling path of code a bit. So I > think we can leave this case as-is. Ok fine I marked the patch as ready for committer. Regards, Hari babu.
Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])
From
Amit Kapila
Date:
On Tue, Nov 19, 2013 at 2:06 PM, Haribabu kommi <haribabu.kommi@huawei.com> wrote: > On 19 November 2013 09:59 Amit Kapila wrote: >> On Mon, Nov 18, 2013 at 8:31 PM, Haribabu kommi >> <haribabu.kommi@huawei.com> wrote: >> > On 18 November 2013 20:01 Amit Kapila wrote: >> >> > Code changes are fine. >> >> > If two or three errors are present in the configuration file, it >> >> prints the last error > > Ok fine I marked the patch as ready for committer. Thanks for review. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Hi, I have looked into this because it's marked as "ready for committer". > On Tue, Nov 19, 2013 at 2:06 PM, Haribabu kommi > <haribabu.kommi@huawei.com> wrote: >> On 19 November 2013 09:59 Amit Kapila wrote: >>> On Mon, Nov 18, 2013 at 8:31 PM, Haribabu kommi >>> <haribabu.kommi@huawei.com> wrote: >>> > On 18 November 2013 20:01 Amit Kapila wrote: >>> >> > Code changes are fine. >>> >> > If two or three errors are present in the configuration file, it >>> >> prints the last error >> >> Ok fine I marked the patch as ready for committer. > > Thanks for review. > > With Regards, > Amit Kapila. > EnterpriseDB: http://www.enterprisedb.com It looks like working as advertised. Great! However I have noticed a few minor issues. 1) validate_conf_option +/* + * Validates configuration parameter and value, by calling check hook functions + * depending on record's vartype. It validates if the parameter + * value given is in range of expected predefined value for that parameter. + * + * freemem - true indicates memory for newval and newextra will be + * freed in this function, false indicates it will be freed + * by caller. + * Return value: + * 1: the value is valid + * 0: the name or value is invalid + */ +int +validate_conf_option(struct config_generic * record, const char *name, + const char *value, GucSource source, int elevel, + bool freemem, void *newval, void **newextra) Is there any reason for the function returns int as it always returns 0 or 1. Maybe returns bool is better? 2) initdb.c + strcpy(tempautobuf, "# Do not edit this file manually! \n"); + autoconflines[0] = pg_strdup(tempautobuf); + strcpy(tempautobuf, "# It will be overwritten by the ALTER SYSTEM command. \n"); + autoconflines[1] = pg_strdup(tempautobuf); Is there any reason to use "tempautobuf" here? I think we can simply change to this: + autoconflines[0] = pg_strdup("# Do not edit this file manually! \n"); + autoconflines[1] = pg_strdup("# It will be overwritten by the ALTER SYSTEM command. \n"); 3) initdb.c It seems the memory allocated for autoconflines[0] and autoconflines[1] by pg_strdup is never freed. (I think there's similar problem with "conflines" as well, though it was not introduced by the patch). Best regards, -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp
On Wed, Dec 18, 2013 at 2:05 PM, Tatsuo Ishii <ishii@postgresql.org> wrote:
> Hi,
>
> I have looked into this because it's marked as "ready for committer".
Thank you.
> It looks like working as advertised. Great! However I have noticed a
> few minor issues.
>
> 1) validate_conf_option
>
> +/*
> + * Validates configuration parameter and value, by calling check hook functions
> + * depending on record's vartype. It validates if the parameter
> + * value given is in range of expected predefined value for that parameter.
> + *
> + * freemem - true indicates memory for newval and newextra will be
> + * freed in this function, false indicates it will be freed
> + * by caller.
> + * Return value:
> + * 1: the value is valid
> + * 0: the name or value is invalid
> + */
> +int
> +validate_conf_option(struct config_generic * record, const char *name,
> + const char *value, GucSource source, int elevel,
> + bool freemem, void *newval, void **newextra)
>
> Is there any reason for the function returns int as it always returns
> 0 or 1. Maybe returns bool is better?
No, return type should be bool, I have changed the same in attached patch.
> 2) initdb.c
>
> + strcpy(tempautobuf, "# Do not edit this file manually! \n");
> + autoconflines[0] = pg_strdup(tempautobuf);
> + strcpy(tempautobuf, "# It will be overwritten by the ALTER SYSTEM command. \n");
> + autoconflines[1] = pg_strdup(tempautobuf);
>
> Is there any reason to use "tempautobuf" here? I think we can simply change to this:
>
> + autoconflines[0] = pg_strdup("# Do not edit this file manually! \n");
> + autoconflines[1] = pg_strdup("# It will be overwritten by the ALTER SYSTEM command. \n");
You are right, I have changed code as per your suggestion.
> 3) initdb.c
>
> It seems the memory allocated for autoconflines[0] and
> autoconflines[1] by pg_strdup is never freed.
I think, it gets freed in writefile() in below code.
for (line = lines; *line != NULL; line++)
{
if (fputs(*line, out_file) < 0)
{
fprintf(stderr, _("%s: could not write file \"%s\": %s\n"),
progname, path, strerror(errno));
exit_nicely();
}
free(*line);
}
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
Attachment
On 12/18/2013 03:35 AM, Tatsuo Ishii wrote: > 3) initdb.c > > It seems the memory allocated for autoconflines[0] and > autoconflines[1] by pg_strdup is never freed. > > (I think there's similar problem with "conflines" as well, though it > was not introduced by the patch). > Why would we care? initdb doesn't run for very long, and the memory will be freed when it exits, usually within a few seconds. My recollection from back when I originally rewrote initdb in C was that cleaning up the memory leaks wasn't worth the trouble. cheers andrew
>> Is there any reason for the function returns int as it always returns
>> 0 or 1. Maybe returns bool is better?
>
> No, return type should be bool, I have changed the same in attached patch.
Confirmed.
>> 2) initdb.c
>>
>> + strcpy(tempautobuf, "# Do not edit this file manually! \n");
>> + autoconflines[0] = pg_strdup(tempautobuf);
>> + strcpy(tempautobuf, "# It will be overwritten by the ALTER SYSTEM command. \n");
>> + autoconflines[1] = pg_strdup(tempautobuf);
>>
>> Is there any reason to use "tempautobuf" here? I think we can simply change to this:
>>
>> + autoconflines[0] = pg_strdup("# Do not edit this file manually! \n");
>> + autoconflines[1] = pg_strdup("# It will be overwritten by the ALTER SYSTEM command. \n");
>
> You are right, I have changed code as per your suggestion.
Confirmed.
>> 3) initdb.c
>>
>> It seems the memory allocated for autoconflines[0] and
>> autoconflines[1] by pg_strdup is never freed.
>
> I think, it gets freed in writefile() in below code.
Oh, I see. Sorry for noise.
I have committed your patches. Thanks.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp
On Wed, Dec 18, 2013 at 8:25 PM, Tatsuo Ishii <ishii@postgresql.org> wrote: >>> Is there any reason for the function returns int as it always returns >>> 0 or 1. Maybe returns bool is better? >> > > I have committed your patches. Thanks. Thank you very much. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
On Thu, Dec 19, 2013 at 12:08 PM, Amit Kapila <amit.kapila16@gmail.com> wrote: > On Wed, Dec 18, 2013 at 8:25 PM, Tatsuo Ishii <ishii@postgresql.org> wrote: >>>> Is there any reason for the function returns int as it always returns >>>> 0 or 1. Maybe returns bool is better? >>> >> >> I have committed your patches. Thanks. > > Thank you very much. I found that the psql tab-completion for ALTER SYSTEM SET has not been implemented yet. Attached patch does that. Barring any objections, I will commit this patch. Regards, -- Fujii Masao
Attachment
> I found that the psql tab-completion for ALTER SYSTEM SET has not been > implemented yet. > Attached patch does that. Barring any objections, I will commit this patch. Good catch! Best regards, -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp
On Thu, Dec 19, 2013 at 2:21 PM, Tatsuo Ishii <ishii@postgresql.org> wrote: >> I found that the psql tab-completion for ALTER SYSTEM SET has not been >> implemented yet. >> Attached patch does that. Barring any objections, I will commit this patch. > > Good catch! Committed. Regards, -- Fujii Masao
On Fri, Dec 20, 2013 at 2:35 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Thu, Dec 19, 2013 at 2:21 PM, Tatsuo Ishii <ishii@postgresql.org> wrote:
>>> I found that the psql tab-completion for ALTER SYSTEM SET has not been
>>> implemented yet.
>>> Attached patch does that. Barring any objections, I will commit this patch.
>>
>> Good catch!
>
> Committed.
I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
it is here.
$ psql
=# ALTER SYSTEM SET shared_buffers = '512MB';
ALTER SYSTEM
=# \q
$ pg_ctl -D data reload
server signaled
2013-12-22 18:24:13 JST LOG: received SIGHUP, reloading configuration files
2013-12-22 18:24:13 JST LOG: parameter "shared_buffers" cannot be
changed without restarting the server
2013-12-22 18:24:13 JST LOG: configuration file "X??" contains
errors; unaffected changes were applied
The configuration file name in the last log message is broken. This problem was
introduced by the ALTER SYSTEM SET patch.
> FreeConfigVariables(head);
> <snip>
> else if (apply)
> ereport(elevel,
> (errcode(ERRCODE_CONFIG_FILE_ERROR),
> errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
> ErrorConfFile)));
The cause of the problem is that, in ProcessConfigFile(), the log
message including
the 'ErrorConfFile' is emitted after 'head' is free'd even though
'ErrorConfFile' points
to one of entry of the 'head'.
Regards,
--
Fujii Masao
On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
> it is here.
>
> $ psql
> =# ALTER SYSTEM SET shared_buffers = '512MB';
> ALTER SYSTEM
> =# \q
> $ pg_ctl -D data reload
> server signaled
> 2013-12-22 18:24:13 JST LOG: received SIGHUP, reloading configuration files
> 2013-12-22 18:24:13 JST LOG: parameter "shared_buffers" cannot be
> changed without restarting the server
> 2013-12-22 18:24:13 JST LOG: configuration file "X??" contains
> errors; unaffected changes were applied
>
> The configuration file name in the last log message is broken. This problem was
> introduced by the ALTER SYSTEM SET patch.
>
>> FreeConfigVariables(head);
>> <snip>
>> else if (apply)
>> ereport(elevel,
>> (errcode(ERRCODE_CONFIG_FILE_ERROR),
>> errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>> ErrorConfFile)));
>
> The cause of the problem is that, in ProcessConfigFile(), the log
> message including
> the 'ErrorConfFile' is emitted after 'head' is free'd even though
> 'ErrorConfFile' points
> to one of entry of the 'head'.
Your analysis is absolutely right.
The reason for this issue is that we want to display filename to which
erroneous parameter
belongs and in this case we are freeing the parameter info before actual error.
To fix, we need to save the filename value before freeing parameter list.
Please find the attached patch to fix this issue.
Note - In my m/c, I could not reproduce the issue. I think this is not
surprising because we
are not setting freed memory to NULL, so it can display anything
(sometimes right value as well)
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
Attachment
On Sun, Dec 22, 2013 at 8:32 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
>> it is here.
>>
>> $ psql
>> =# ALTER SYSTEM SET shared_buffers = '512MB';
>> ALTER SYSTEM
>> =# \q
>> $ pg_ctl -D data reload
>> server signaled
>> 2013-12-22 18:24:13 JST LOG: received SIGHUP, reloading configuration files
>> 2013-12-22 18:24:13 JST LOG: parameter "shared_buffers" cannot be
>> changed without restarting the server
>> 2013-12-22 18:24:13 JST LOG: configuration file "X??" contains
>> errors; unaffected changes were applied
>>
>> The configuration file name in the last log message is broken. This problem was
>> introduced by the ALTER SYSTEM SET patch.
>>
>>> FreeConfigVariables(head);
>>> <snip>
>>> else if (apply)
>>> ereport(elevel,
>>> (errcode(ERRCODE_CONFIG_FILE_ERROR),
>>> errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>>> ErrorConfFile)));
>>
>> The cause of the problem is that, in ProcessConfigFile(), the log
>> message including
>> the 'ErrorConfFile' is emitted after 'head' is free'd even though
>> 'ErrorConfFile' points
>> to one of entry of the 'head'.
>
> Your analysis is absolutely right.
> The reason for this issue is that we want to display filename to which
> erroneous parameter
> belongs and in this case we are freeing the parameter info before actual error.
> To fix, we need to save the filename value before freeing parameter list.
>
> Please find the attached patch to fix this issue.
>
> Note - In my m/c, I could not reproduce the issue. I think this is not
> surprising because we
> are not setting freed memory to NULL, so it can display anything
> (sometimes right value as well)
If you find that the provided patch doesn't fix the problem or you don't
find it appropriate due to some other reason, let me know the feedback.
I shall be able to provide updated patch early next as I will be on vacation for
remaining part of this week.
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
Discussion around this topic has reached hundreds of messages, and whilst I have failed to find mention of my following question, I appreciate it may have already been discussed. I have noticed that configuration parameters for extensions are only supported if the server was started with one of the parameters already being set in postgresql.conf: # without any mention in postgresql.conf postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true; ERROR: unrecognized configuration parameter "auto_explain.log_verbose" # with auto_explain.log_verbose = false added to postgresql.conf postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true; ALTER SYSTEM Removing the entry from postgresql.conf, restarting Postgres, setting it to the default, then restarting again renders it unsettable again: # removed entry from postgresql.conf and restarted postgres=# ALTER SYSTEM SET auto_explain.log_verbose = default; ALTER SYSTEM # restart postgres postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true; ERROR: unrecognized configuration parameter "auto_explain.log_verbose" Is this by design? -- Thom
On Mon, Dec 23, 2013 at 1:42 PM, Thom Brown <thom@linux.com> wrote: > Discussion around this topic has reached hundreds of messages, and > whilst I have failed to find mention of my following question, I > appreciate it may have already been discussed. > > I have noticed that configuration parameters for extensions are only > supported if the server was started with one of the parameters already > being set in postgresql.conf: > > # without any mention in postgresql.conf > postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true; > ERROR: unrecognized configuration parameter "auto_explain.log_verbose" > > # with auto_explain.log_verbose = false added to postgresql.conf > postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true; > ALTER SYSTEM > > Removing the entry from postgresql.conf, restarting Postgres, setting > it to the default, then restarting again renders it unsettable again: > > # removed entry from postgresql.conf and restarted > > postgres=# ALTER SYSTEM SET auto_explain.log_verbose = default; > ALTER SYSTEM > > # restart postgres > > postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true; > ERROR: unrecognized configuration parameter "auto_explain.log_verbose" > > Is this by design? I would think that you'd need to have auto_explain loaded in the backend where you're trying to make a change, but you shouldn't need the setting to be present in postgresql.conf, I would think. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 23 December 2013 19:14, Robert Haas <robertmhaas@gmail.com> wrote: > On Mon, Dec 23, 2013 at 1:42 PM, Thom Brown <thom@linux.com> wrote: >> Discussion around this topic has reached hundreds of messages, and >> whilst I have failed to find mention of my following question, I >> appreciate it may have already been discussed. >> >> I have noticed that configuration parameters for extensions are only >> supported if the server was started with one of the parameters already >> being set in postgresql.conf: >> >> # without any mention in postgresql.conf >> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true; >> ERROR: unrecognized configuration parameter "auto_explain.log_verbose" >> >> # with auto_explain.log_verbose = false added to postgresql.conf >> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true; >> ALTER SYSTEM >> >> Removing the entry from postgresql.conf, restarting Postgres, setting >> it to the default, then restarting again renders it unsettable again: >> >> # removed entry from postgresql.conf and restarted >> >> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = default; >> ALTER SYSTEM >> >> # restart postgres >> >> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true; >> ERROR: unrecognized configuration parameter "auto_explain.log_verbose" >> >> Is this by design? > > I would think that you'd need to have auto_explain loaded in the > backend where you're trying to make a change, but you shouldn't need > the setting to be present in postgresql.conf, I would think. This appears to be the case. I hadn't set the library to be loaded in the config. I guess therefore it follows that arbitrary configuration parameters aren't supported (e.g. moo.bark = 5), only pre-defined ones. Thanks Thom
On Mon, Dec 23, 2013 at 2:19 PM, Thom Brown <thom@linux.com> wrote: >> I would think that you'd need to have auto_explain loaded in the >> backend where you're trying to make a change, but you shouldn't need >> the setting to be present in postgresql.conf, I would think. > > This appears to be the case. I hadn't set the library to be loaded in > the config. > > I guess therefore it follows that arbitrary configuration parameters > aren't supported (e.g. moo.bark = 5), only pre-defined ones. Yeah, and that's by design. Otherwise, it would be too easy to set a config parameter to a value that wasn't legal, and therefore make the server fail to start. moo.bark = 5 likely won't cause any problems, but auto_explain.log_verbose = fasle could. By insisting that the module providing the GUC be loaded, we can sanity-check the value at the time it gets set. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 23 December 2013 19:35, Robert Haas <robertmhaas@gmail.com> wrote: > On Mon, Dec 23, 2013 at 2:19 PM, Thom Brown <thom@linux.com> wrote: >>> I would think that you'd need to have auto_explain loaded in the >>> backend where you're trying to make a change, but you shouldn't need >>> the setting to be present in postgresql.conf, I would think. >> >> This appears to be the case. I hadn't set the library to be loaded in >> the config. >> >> I guess therefore it follows that arbitrary configuration parameters >> aren't supported (e.g. moo.bark = 5), only pre-defined ones. > > Yeah, and that's by design. Otherwise, it would be too easy to set a > config parameter to a value that wasn't legal, and therefore make the > server fail to start. moo.bark = 5 likely won't cause any problems, > but auto_explain.log_verbose = fasle could. By insisting that the > module providing the GUC be loaded, we can sanity-check the value at > the time it gets set. Alles klar. :) -- Thom
On Tue, Dec 24, 2013 at 2:57 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Sun, Dec 22, 2013 at 8:32 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
>>> it is here.
>>>
>>> $ psql
>>> =# ALTER SYSTEM SET shared_buffers = '512MB';
>>> ALTER SYSTEM
>>> =# \q
>>> $ pg_ctl -D data reload
>>> server signaled
>>> 2013-12-22 18:24:13 JST LOG: received SIGHUP, reloading configuration files
>>> 2013-12-22 18:24:13 JST LOG: parameter "shared_buffers" cannot be
>>> changed without restarting the server
>>> 2013-12-22 18:24:13 JST LOG: configuration file "X??" contains
>>> errors; unaffected changes were applied
>>>
>>> The configuration file name in the last log message is broken. This problem was
>>> introduced by the ALTER SYSTEM SET patch.
>>>
>>>> FreeConfigVariables(head);
>>>> <snip>
>>>> else if (apply)
>>>> ereport(elevel,
>>>> (errcode(ERRCODE_CONFIG_FILE_ERROR),
>>>> errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>>>> ErrorConfFile)));
>>>
>>> The cause of the problem is that, in ProcessConfigFile(), the log
>>> message including
>>> the 'ErrorConfFile' is emitted after 'head' is free'd even though
>>> 'ErrorConfFile' points
>>> to one of entry of the 'head'.
>>
>> Your analysis is absolutely right.
>> The reason for this issue is that we want to display filename to which
>> erroneous parameter
>> belongs and in this case we are freeing the parameter info before actual error.
>> To fix, we need to save the filename value before freeing parameter list.
>>
>> Please find the attached patch to fix this issue.
>>
>> Note - In my m/c, I could not reproduce the issue. I think this is not
>> surprising because we
>> are not setting freed memory to NULL, so it can display anything
>> (sometimes right value as well)
>
> If you find that the provided patch doesn't fix the problem or you don't
> find it appropriate due to some other reason, let me know the feedback.
When I read ProcessConfigFile() more carefully, I found another related
problem. The procedure to reproduce the problem is here.
--------------------
$ psql
=# ALTER SYSTEM SET shared_buffers = '256MB';
=# \q
$ echo "shared_buffers = '256MB'" >> $PGDATA/postgresql.conf
$ pg_ctl -D $PGDATA reload
2013-12-25 03:05:44 JST LOG: parameter "shared_buffers" cannot be
changed without restarting the server
2013-12-25 03:05:44 JST LOG: parameter "shared_buffers" cannot be
changed without restarting the server
2013-12-25 03:05:44 JST LOG: configuration file
"/dav/alter-system/data/postgresql.auto.conf" contains errors;
unaffected changes were applied
--------------------
Both postgresql.conf and postgresql.auto.conf contain the setting of
the parameter that cannot be changed without restarting the server.
But only postgresql.auto.conf was logged, and which would be confusing,
I'm afraid. I think that this problem should be fixed together with the
problem that I reported before. Thought?
Regards,
--
Fujii Masao
On Tue, Dec 24, 2013 at 10:08 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Tue, Dec 24, 2013 at 2:57 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> On Sun, Dec 22, 2013 at 8:32 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>>>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
>>>> it is here.
>>>>
>>>> $ psql
>>>> =# ALTER SYSTEM SET shared_buffers = '512MB';
>>>> ALTER SYSTEM
>>>> =# \q
>>>> $ pg_ctl -D data reload
>>>> server signaled
>>>> 2013-12-22 18:24:13 JST LOG: received SIGHUP, reloading configuration files
>>>> 2013-12-22 18:24:13 JST LOG: parameter "shared_buffers" cannot be
>>>> changed without restarting the server
>>>> 2013-12-22 18:24:13 JST LOG: configuration file "X??" contains
>>>> errors; unaffected changes were applied
>>>>
>>>> The configuration file name in the last log message is broken. This problem was
>>>> introduced by the ALTER SYSTEM SET patch.
>>>>
>>>>> FreeConfigVariables(head);
>>>>> <snip>
>>>>> else if (apply)
>>>>> ereport(elevel,
>>>>> (errcode(ERRCODE_CONFIG_FILE_ERROR),
>>>>> errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>>>>> ErrorConfFile)));
>>>>
>>>> The cause of the problem is that, in ProcessConfigFile(), the log
>>>> message including
>>>> the 'ErrorConfFile' is emitted after 'head' is free'd even though
>>>> 'ErrorConfFile' points
>>>> to one of entry of the 'head'.
>>>
>>> Your analysis is absolutely right.
>>> The reason for this issue is that we want to display filename to which
>>> erroneous parameter
>>> belongs and in this case we are freeing the parameter info before actual error.
>>> To fix, we need to save the filename value before freeing parameter list.
>>>
>>> Please find the attached patch to fix this issue.
>>>
>>> Note - In my m/c, I could not reproduce the issue. I think this is not
>>> surprising because we
>>> are not setting freed memory to NULL, so it can display anything
>>> (sometimes right value as well)
>>
>> If you find that the provided patch doesn't fix the problem or you don't
>> find it appropriate due to some other reason, let me know the feedback.
>
> When I read ProcessConfigFile() more carefully, I found another related
> problem. The procedure to reproduce the problem is here.
>
> --------------------
> $ psql
> =# ALTER SYSTEM SET shared_buffers = '256MB';
> =# \q
>
> $ echo "shared_buffers = '256MB'" >> $PGDATA/postgresql.conf
> $ pg_ctl -D $PGDATA reload
> 2013-12-25 03:05:44 JST LOG: parameter "shared_buffers" cannot be
> changed without restarting the server
> 2013-12-25 03:05:44 JST LOG: parameter "shared_buffers" cannot be
> changed without restarting the server
> 2013-12-25 03:05:44 JST LOG: configuration file
> "/dav/alter-system/data/postgresql.auto.conf" contains errors;
> unaffected changes were applied
> --------------------
>
> Both postgresql.conf and postgresql.auto.conf contain the setting of
> the parameter that cannot be changed without restarting the server.
> But only postgresql.auto.conf was logged, and which would be confusing,
> I'm afraid. I think that this problem should be fixed together with the
> problem that I reported before. Thought?
I have run into this problem on many occasions because my benchmark
scripts usually append a hunk of stuff to postgresql.conf, and any
parameters that were already set generate this warning every time you
reload, because postgresql.conf now mentions that parameter twice.
I'm not quite sure what other problem you want to fix it together
with, and I'm not sure what the right fix is either, but +1 for coming
up with some solution that's better than what we have now.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Tue, Dec 24, 2013 at 11:38 PM, Fujii Masao <masao.fujii@gmail.com> wrote: >> On Sun, Dec 22, 2013 at 8:32 PM, Amit Kapila <amit.kapila16@gmail.com> wrote: >>> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote: >>>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce >>>> it is here. >>>> >>>> $ psql >>>> =# ALTER SYSTEM SET shared_buffers = '512MB'; >>>> ALTER SYSTEM >>>> =# \q >>>> $ pg_ctl -D data reload >>>> server signaled >>>> 2013-12-22 18:24:13 JST LOG: received SIGHUP, reloading configuration files >>>> 2013-12-22 18:24:13 JST LOG: parameter "shared_buffers" cannot be >>>> changed without restarting the server >>>> 2013-12-22 18:24:13 JST LOG: configuration file "X??" contains >>>> errors; unaffected changes were applied >>>> >>> >>> Your analysis is absolutely right. >>> The reason for this issue is that we want to display filename to which >>> erroneous parameter >>> belongs and in this case we are freeing the parameter info before actual error. >>> To fix, we need to save the filename value before freeing parameter list. >>> >>> Please find the attached patch to fix this issue. > > When I read ProcessConfigFile() more carefully, I found another related > problem. The procedure to reproduce the problem is here. > > -------------------- > $ psql > =# ALTER SYSTEM SET shared_buffers = '256MB'; > =# \q > > $ echo "shared_buffers = '256MB'" >> $PGDATA/postgresql.conf > $ pg_ctl -D $PGDATA reload > 2013-12-25 03:05:44 JST LOG: parameter "shared_buffers" cannot be > changed without restarting the server > 2013-12-25 03:05:44 JST LOG: parameter "shared_buffers" cannot be > changed without restarting the server > 2013-12-25 03:05:44 JST LOG: configuration file > "/dav/alter-system/data/postgresql.auto.conf" contains errors; > unaffected changes were applied > -------------------- > > Both postgresql.conf and postgresql.auto.conf contain the setting of > the parameter that cannot be changed without restarting the server. > But only postgresql.auto.conf was logged, and which would be confusing, > I'm afraid. The reason for above behaviour is that while applying configuration parameters, it just note down the last file which has error and ignore others. Now if we just want to change behaviour for above case that it display both the files, then during processing of parameters, we can save the errorconffile if it is different from previous. However, let us say if user uses include mechanism of conf files and used new file for specifying some of the changed parameters, then again similar thing can be observed. I think here basic idea is user should avoid using multiple methods to change configuration parameters, however we cannot stop them from doing the same. So in some cases like above where user use multiple methods to change configuration can add more work for him to understand and detect the error. Also, some similar behaviour can be observed if user uses include mechanism to specify some config parameters without even Alter System changes. >I think that this problem should be fixed together with the > problem that I reported before. Thought? Here I think you might be worried that if we want to fix the behaviour reported in this mail, it might overlap with the changes of previous fix. So I think let us first decide if we want to keep the current behaviour as it is or would like to change it and if it needs change, what should be the new behaviour? With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
On Fri, Dec 27, 2013 at 5:01 AM, Robert Haas <robertmhaas@gmail.com> wrote: > On Tue, Dec 24, 2013 at 10:08 AM, Fujii Masao <masao.fujii@gmail.com> wrote: >> When I read ProcessConfigFile() more carefully, I found another related >> problem. The procedure to reproduce the problem is here. >> >> -------------------- >> $ psql >> =# ALTER SYSTEM SET shared_buffers = '256MB'; >> =# \q >> >> $ echo "shared_buffers = '256MB'" >> $PGDATA/postgresql.conf >> $ pg_ctl -D $PGDATA reload >> 2013-12-25 03:05:44 JST LOG: parameter "shared_buffers" cannot be >> changed without restarting the server >> 2013-12-25 03:05:44 JST LOG: parameter "shared_buffers" cannot be >> changed without restarting the server >> 2013-12-25 03:05:44 JST LOG: configuration file >> "/dav/alter-system/data/postgresql.auto.conf" contains errors; >> unaffected changes were applied >> -------------------- >> >> Both postgresql.conf and postgresql.auto.conf contain the setting of >> the parameter that cannot be changed without restarting the server. >> But only postgresql.auto.conf was logged, and which would be confusing, >> I'm afraid. I think that this problem should be fixed together with the >> problem that I reported before. Thought? > > I have run into this problem on many occasions because my benchmark > scripts usually append a hunk of stuff to postgresql.conf, and any > parameters that were already set generate this warning every time you > reload, because postgresql.conf now mentions that parameter twice. > I'm not quite sure what other problem you want to fix it together > with, The 2 problems are: 1. First is that if user changes the config parameter (for ex. shared_buffers) both by manually editing postgresql.conf and by using Alter System command and then reload the config, it will show the message for parameter 'shared_buffers' twice, but will only show the last file where it exists. The detailed description of problem is in current mail. 2. The other problem is in some cases the name of file it display is junk, problem and fix can be found at link: http://www.postgresql.org/message-id/CAA4eK1+-yD2p=+6tdj_xPruHN4m_CkvFr51aH0gv3SXLGOACGg@mail.gmail.com >and I'm not sure what the right fix is either, but +1 for coming > up with some solution that's better than what we have now. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
On Sun, Dec 22, 2013 at 10:02 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
>> it is here.
>>
>> $ psql
>> =# ALTER SYSTEM SET shared_buffers = '512MB';
>> ALTER SYSTEM
>> =# \q
>> $ pg_ctl -D data reload
>> server signaled
>> 2013-12-22 18:24:13 JST LOG: received SIGHUP, reloading configuration files
>> 2013-12-22 18:24:13 JST LOG: parameter "shared_buffers" cannot be
>> changed without restarting the server
>> 2013-12-22 18:24:13 JST LOG: configuration file "X??" contains
>> errors; unaffected changes were applied
>>
>> The configuration file name in the last log message is broken. This problem was
>> introduced by the ALTER SYSTEM SET patch.
>>
>>> FreeConfigVariables(head);
>>> <snip>
>>> else if (apply)
>>> ereport(elevel,
>>> (errcode(ERRCODE_CONFIG_FILE_ERROR),
>>> errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>>> ErrorConfFile)));
>>
>> The cause of the problem is that, in ProcessConfigFile(), the log
>> message including
>> the 'ErrorConfFile' is emitted after 'head' is free'd even though
>> 'ErrorConfFile' points
>> to one of entry of the 'head'.
>
> Your analysis is absolutely right.
> The reason for this issue is that we want to display filename to which
> erroneous parameter
> belongs and in this case we are freeing the parameter info before actual error.
> To fix, we need to save the filename value before freeing parameter list.
>
> Please find the attached patch to fix this issue.
>
> Note - In my m/c, I could not reproduce the issue. I think this is not
> surprising because we
> are not setting freed memory to NULL, so it can display anything
> (sometimes right value as well)
Couldn't we also handle this by postponing FreeConfigVariables until
after the if (error) block?
Also, what's the point of this:
+ snprintf(ConfigAutoFileName,sizeof(ConfigAutoFileName),"%s",
PG_AUTOCONF_FILENAME);
Why copy PG_AUTOCONF_FILENAME into another buffer? Why not just pass
PG_AUTOCONF_FILENAME directly to ParseConfigFile?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Mon, Jan 6, 2014 at 7:58 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sun, Dec 22, 2013 at 10:02 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
>>> it is here.
>>>
>>>> FreeConfigVariables(head);
>>>> <snip>
>>>> else if (apply)
>>>> ereport(elevel,
>>>> (errcode(ERRCODE_CONFIG_FILE_ERROR),
>>>> errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>>>> ErrorConfFile)));
>>>
>>> The cause of the problem is that, in ProcessConfigFile(), the log
>>> message including
>>> the 'ErrorConfFile' is emitted after 'head' is free'd even though
>>> 'ErrorConfFile' points
>>> to one of entry of the 'head'.
>>
>> Your analysis is absolutely right.
>> The reason for this issue is that we want to display filename to which
>> erroneous parameter
>> belongs and in this case we are freeing the parameter info before actual error.
>> To fix, we need to save the filename value before freeing parameter list.
>>
>> Please find the attached patch to fix this issue.
>>
>>
>
> Couldn't we also handle this by postponing FreeConfigVariables until
> after the if (error) block?
Wouldn't doing that way can lead to bigger memory leak, if error level is ERROR. Though in current fix also it can
leakmemory but it will be just for ErrorConfFile_save. I think some similar case can happen for 'pre_value' in code
currentlyas well, that's why I have fixed it in a similar way in patch.
> Also, what's the point of this:
>
> + snprintf(ConfigAutoFileName,sizeof(ConfigAutoFileName),"%s",
> PG_AUTOCONF_FILENAME);
>
> Why copy PG_AUTOCONF_FILENAME into another buffer? Why not just pass
> PG_AUTOCONF_FILENAME directly to ParseConfigFile?
Initially, I think we were doing concat of folder and file name for
Autofile, that's why
the code was written that way, but currently it can be changed to way you are
suggesting.
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
On Mon, Jan 6, 2014 at 9:48 AM, Amit Kapila <amit.kapila16@gmail.com> wrote: >> Couldn't we also handle this by postponing FreeConfigVariables until >> after the if (error) block? > > Wouldn't doing that way can lead to bigger memory leak, if error level > is ERROR. Though in current fix also it can leak memory but it will be > just for ErrorConfFile_save. I think some similar case can happen for > 'pre_value' in code currently as well, that's why I have fixed it in a > similar way in patch. I was assuming that error-recovery would reset the containing memory context, but I'm not sure what memory context we're executing in at this point. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Tue, Jan 7, 2014 at 12:52 AM, Robert Haas <robertmhaas@gmail.com> wrote: > On Mon, Jan 6, 2014 at 9:48 AM, Amit Kapila <amit.kapila16@gmail.com> wrote: >>> Couldn't we also handle this by postponing FreeConfigVariables until >>> after the if (error) block? >> >> Wouldn't doing that way can lead to bigger memory leak, if error level >> is ERROR. Though in current fix also it can leak memory but it will be >> just for ErrorConfFile_save. I think some similar case can happen for >> 'pre_value' in code currently as well, that's why I have fixed it in a >> similar way in patch. > > I was assuming that error-recovery would reset the containing memory > context, but I'm not sure what memory context we're executing in at > this point. This function is called from multiple places and based on when it would get called the memory context varies. During Startup, it gets called in Postmaster context and if some backend runs pg_reload_conf(), then it will get called from other background processes (WAL Writer, Checpointer, etc..) in their respective contexts (for WAL Writer, the context will be WAL Writer, ..). In current code, the only time it can go to error path with elevel as ERROR is during Postmaster startup (context == PGC_POSTMASTER), at which it will anyway upgrade ERROR to FATAL, so it should not be a problem to move function FreeConfigVariables() after error block check. However in future, if someone added any more ERROR (the chances of which seems to be quite less), it can cause leak, may be thats why original code has been written that way. If you think it's better to fix by moving FreeConfigVariables() after error block check, then I can update the patch by doing so and incorporate other change (directly use PG_AUTOCONF_FILENAME) suggested by you as well? With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
On Mon, Jan 6, 2014 at 11:37 PM, Amit Kapila <amit.kapila16@gmail.com> wrote: > On Tue, Jan 7, 2014 at 12:52 AM, Robert Haas <robertmhaas@gmail.com> wrote: >> On Mon, Jan 6, 2014 at 9:48 AM, Amit Kapila <amit.kapila16@gmail.com> wrote: >>>> Couldn't we also handle this by postponing FreeConfigVariables until >>>> after the if (error) block? >>> >>> Wouldn't doing that way can lead to bigger memory leak, if error level >>> is ERROR. Though in current fix also it can leak memory but it will be >>> just for ErrorConfFile_save. I think some similar case can happen for >>> 'pre_value' in code currently as well, that's why I have fixed it in a >>> similar way in patch. >> >> I was assuming that error-recovery would reset the containing memory >> context, but I'm not sure what memory context we're executing in at >> this point. > > This function is called from multiple places and based on when it would > get called the memory context varies. During Startup, it gets called in > Postmaster context and if some backend runs pg_reload_conf(), then > it will get called from other background processes (WAL Writer, > Checpointer, etc..) in their respective contexts (for WAL Writer, the > context will be WAL Writer, ..). > > In current code, the only time it can go to error path with elevel as > ERROR is during Postmaster startup > (context == PGC_POSTMASTER), at which it will anyway upgrade > ERROR to FATAL, so it should not be a problem to move > function FreeConfigVariables() after error block check. However > in future, if someone added any more ERROR (the chances of which > seems to be quite less), it can cause leak, may be thats why original > code has been written that way. > > If you think it's better to fix by moving FreeConfigVariables() after error > block check, then I can update the patch by doing so and incorporate other > change (directly use PG_AUTOCONF_FILENAME) suggested by you > as well? Yeah, let's do it that way. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Thu, Jan 9, 2014 at 12:26 AM, Robert Haas <robertmhaas@gmail.com> wrote: > On Mon, Jan 6, 2014 at 11:37 PM, Amit Kapila <amit.kapila16@gmail.com> wrote: >> On Tue, Jan 7, 2014 at 12:52 AM, Robert Haas <robertmhaas@gmail.com> wrote: >>> On Mon, Jan 6, 2014 at 9:48 AM, Amit Kapila <amit.kapila16@gmail.com> wrote: >>>>> Couldn't we also handle this by postponing FreeConfigVariables until >>>>> after the if (error) block? >>>> >>>> Wouldn't doing that way can lead to bigger memory leak, if error level >>>> is ERROR. Though in current fix also it can leak memory but it will be >>>> just for ErrorConfFile_save. I think some similar case can happen for >>>> 'pre_value' in code currently as well, that's why I have fixed it in a >>>> similar way in patch. >>> >>> I was assuming that error-recovery would reset the containing memory >>> context, but I'm not sure what memory context we're executing in at >>> this point. >> >> >> In current code, the only time it can go to error path with elevel as >> ERROR is during Postmaster startup >> (context == PGC_POSTMASTER), at which it will anyway upgrade >> ERROR to FATAL, so it should not be a problem to move >> function FreeConfigVariables() after error block check. However >> in future, if someone added any more ERROR (the chances of which >> seems to be quite less), it can cause leak, may be thats why original >> code has been written that way. >> >> If you think it's better to fix by moving FreeConfigVariables() after error >> block check, then I can update the patch by doing so and incorporate other >> change (directly use PG_AUTOCONF_FILENAME) suggested by you >> as well? > > Yeah, let's do it that way. Okay, done. Attached patch fixes both the display of wrong file name and usage of PG_AUTOCONF_FILENAME. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com
Attachment
On Thu, Jan 9, 2014 at 11:06 PM, Amit Kapila <amit.kapila16@gmail.com> wrote: > On Thu, Jan 9, 2014 at 12:26 AM, Robert Haas <robertmhaas@gmail.com> wrote: >> On Mon, Jan 6, 2014 at 11:37 PM, Amit Kapila <amit.kapila16@gmail.com> wrote: >>> On Tue, Jan 7, 2014 at 12:52 AM, Robert Haas <robertmhaas@gmail.com> wrote: >>>> On Mon, Jan 6, 2014 at 9:48 AM, Amit Kapila <amit.kapila16@gmail.com> wrote: >>>>>> Couldn't we also handle this by postponing FreeConfigVariables until >>>>>> after the if (error) block? >>>>> >>>>> Wouldn't doing that way can lead to bigger memory leak, if error level >>>>> is ERROR. Though in current fix also it can leak memory but it will be >>>>> just for ErrorConfFile_save. I think some similar case can happen for >>>>> 'pre_value' in code currently as well, that's why I have fixed it in a >>>>> similar way in patch. >>>> >>>> I was assuming that error-recovery would reset the containing memory >>>> context, but I'm not sure what memory context we're executing in at >>>> this point. >>> >>> >>> In current code, the only time it can go to error path with elevel as >>> ERROR is during Postmaster startup >>> (context == PGC_POSTMASTER), at which it will anyway upgrade >>> ERROR to FATAL, so it should not be a problem to move >>> function FreeConfigVariables() after error block check. However >>> in future, if someone added any more ERROR (the chances of which >>> seems to be quite less), it can cause leak, may be thats why original >>> code has been written that way. >>> >>> If you think it's better to fix by moving FreeConfigVariables() after error >>> block check, then I can update the patch by doing so and incorporate other >>> change (directly use PG_AUTOCONF_FILENAME) suggested by you >>> as well? >> >> Yeah, let's do it that way. > > Okay, done. Attached patch fixes both the display of wrong file name and > usage of PG_AUTOCONF_FILENAME. Committed with a comment change. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company