Thread: Converting postgresql.conf parameters to kilobytes
Hi, I was toying around with idea of converting all the memory related parameters in postgresql.conf to kilobytes for simplicity and uniformity. Attached is a proof of concept patch that converts shared_buffers to kilobytes using assign_hook. It compiled all-right but I experienced a strange behavior. At the time of initdb, it demanded 69MB of shared memory. I had to bump up SHMMAX from 32MB to 128MB to get initdb through. Certainly I did something wrong somewhere but I don't know what and where. This is linux 2.6.4. The postgresql.conf is properly altered and shows 8000(Though the description around is no longer in sync.) I also don't know where to put the assign_hook. I have put in guc.c for the time being. Only other int hook I found was assign_max_stack_depth which is in postgres.c Any comments/pointers? Regards,Shridhar
Shridhar Daithankar <shridhar@frodo.hserus.net> writes:
> I was toying around with idea of converting all the memory related
> parameters in postgresql.conf to kilobytes for simplicity and
> uniformity.
Why is that a good idea?
regards, tom lane
> Hi, > > I was toying around with idea of converting all the memory related > parameters > in postgresql.conf to kilobytes for simplicity and uniformity. > > Attached is a proof of concept patch that converts shared_buffers to > kilobytes > using assign_hook. > > It compiled all-right but I experienced a strange behavior. At the time of > initdb, it demanded 69MB of shared memory. I had to bump up SHMMAX from > 32MB > to 128MB to get initdb through. Certainly I did something wrong somewhere > but > I don't know what and where. This is linux 2.6.4. > > The postgresql.conf is properly altered and shows 8000(Though the > description > around is no longer in sync.) > > I also don't know where to put the assign_hook. I have put in guc.c for > the > time being. Only other int hook I found was assign_max_stack_depth which > is > in postgres.c > > Any comments/pointers? > > Regards, > Shridhar Changing the format of numbers is tricky. I wouldn't change it, I'd allow for a suffix, i.e. 1024K or 1M Additionally, the configuration patch I submited and I think Bruce has already or will merge, allows for a configuration function. A specific keyword can be handled by a handler function. This is how I got "include" to work, but I would modify *all* the number handlers in GUC to accept the suffix rather than explicitly change anything that currently works. It is more consistent, more readble i.e. 128 vs 128M.
On Sunday 30 May 2004 21:33, Tom Lane wrote: > Shridhar Daithankar <shridhar@frodo.hserus.net> writes: > > I was toying around with idea of converting all the memory related > > parameters in postgresql.conf to kilobytes for simplicity and > > uniformity. > > Why is that a good idea? Right now following are measured in pages wal_buffers shared_buffers effective_cachesize Following are in kbs work_mem maintenance_work_mem max_stack_depth while rest of the memory parameters are in kb. I thought being uniform would be good. Besides it will make it independent of page size as well. Shridhar
Shridhar Daithankar wrote: > On Sunday 30 May 2004 21:33, Tom Lane wrote: > > Shridhar Daithankar <shridhar@frodo.hserus.net> writes: > > > I was toying around with idea of converting all the memory related > > > parameters in postgresql.conf to kilobytes for simplicity and > > > uniformity. > > > > Why is that a good idea? > > Right now following are measured in pages > wal_buffers > shared_buffers > effective_cachesize > > Following are in kbs > work_mem > maintenance_work_mem > max_stack_depth > > while rest of the memory parameters are in kb. I thought being uniform would > be good. Besides it will make it independent of page size as well. Sounds good to me. How are you handling cases where the value has to be a multiple of page size --- rounding or throwing an error? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Monday 31 May 2004 18:41, Bruce Momjian wrote: > Shridhar Daithankar wrote: > > On Sunday 30 May 2004 21:33, Tom Lane wrote: > > > Shridhar Daithankar <shridhar@frodo.hserus.net> writes: > > > > I was toying around with idea of converting all the memory related > > > > parameters in postgresql.conf to kilobytes for simplicity and > > > > uniformity. > > > > > > Why is that a good idea? > > > > Right now following are measured in pages > > wal_buffers > > shared_buffers > > effective_cachesize > > > > Following are in kbs > > work_mem > > maintenance_work_mem > > max_stack_depth > > > > while rest of the memory parameters are in kb. I thought being uniform > > would be good. Besides it will make it independent of page size as well. > > Sounds good to me. How are you handling cases where the value has to be > a multiple of page size --- rounding or throwing an error? The patch rounds it down owing to assignment of possible real number to integer. but I didn't imagine of this requirement, to be honest. This seems to be better behavior than throwing an error. Shridhar
Shridhar Daithankar <shridhar@frodo.hserus.net> writes:
> Right now following are measured in pages
> wal_buffers
> shared_buffers
> effective_cachesize
> while rest of the memory parameters are in kb. I thought being uniform would
> be good. Besides it will make it independent of page size as well.
It would probably be reasonable to change effective_cache_size, since we
really do not know what the kernel's unit of buffering is (except on
Linux, where we *do* know that it ain't 8K ;-)). Personally I'd opt for
measuring it in MB not KB, though; that would be a much more convenient
unit on modern machines. We could easily make it a float for anyone who
thinks they know the cache size to sub-MB accuracy.
As for the others, I'll side with Emerson: "a foolish consistency is the
hobgoblin of little minds". We know very well what the unit of
allocation of those is, and it's pages. There's no advantage to using
KB except making it harder to work out what's really happening. We
could measure max_connections in KB too if we had a mind to: there's
a very definite shared-mem cost per connection slot. Or the FSM
parameters, or checkpoint_segments, or max_locks_per_transaction.
The fact that they have quantifiable space costs doesn't mean that space
is the most useful way to measure them.
BTW, were you intending to convert KB to NBuffers by charging exactly 8K
per buffer, or were you intending to allow for the additional shmem
costs such as buffer headers, per-buffer LWLocks, etc? If not the
latter, then what are you really measuring? For sure it's not shared
memory size --- charging an artificial number isn't going to help anyone
who's trying to pick shared_buffers to arrive at a particular actual
shmem size. But if it is the latter then it'll become even more
impossible to tell what's really happening, and we'll be forced to
invent some way of reading out how many buffers really got allocated.
So I disagree with the premise. Measuring these things in KB is not an
improvement.
regards, tom lane
Tom Lane wrote: > Shridhar Daithankar <shridhar@frodo.hserus.net> writes: > > Right now following are measured in pages > > wal_buffers > > shared_buffers > > effective_cachesize > > while rest of the memory parameters are in kb. I thought being uniform would > > be good. Besides it will make it independent of page size as well. > > It would probably be reasonable to change effective_cache_size, since we > really do not know what the kernel's unit of buffering is (except on > Linux, where we *do* know that it ain't 8K ;-)). Personally I'd opt for > measuring it in MB not KB, though; that would be a much more convenient > unit on modern machines. We could easily make it a float for anyone who > thinks they know the cache size to sub-MB accuracy. I thought the idea was that you could put 'm', 'k', or maybe 'p' after the value to specify the units. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Tom Lane wrote: > So I disagree with the premise. Measuring these things in KB is not an > improvement. I agree, although I think changing effective_cache_size to be measured in KB/MB is worth doing. -Neil
Neil Conway <neilc@samurai.com> writes: > Tom Lane wrote: > > So I disagree with the premise. Measuring these things in KB is not an > > improvement. > > I agree, although I think changing effective_cache_size to be measured in KB/MB > is worth doing. I have to say as a user the parameters that are measured in postgres blocks are really annoying and confusing. Really really annoying and confusing. If someone's playing with this I would suggest they should work something like dd parameters and take a unit. So you could specify effective_cache=500k or effective_cache=1M or effective_cache=1G or whatever unit you wanted. And this should be true for _all_ parameters that measure space. Consistency makes it much much easier for people to learn a new system. -- greg
On Monday 31 May 2004 22:00, Tom Lane wrote: > Shridhar Daithankar <shridhar@frodo.hserus.net> writes: > > Right now following are measured in pages > > wal_buffers > > shared_buffers > > effective_cachesize > > while rest of the memory parameters are in kb. I thought being uniform > > would be good. Besides it will make it independent of page size as well. > > It would probably be reasonable to change effective_cache_size, since we > really do not know what the kernel's unit of buffering is (except on > Linux, where we *do* know that it ain't 8K ;-)). Personally I'd opt for > measuring it in MB not KB, though; that would be a much more convenient > unit on modern machines. We could easily make it a float for anyone who > thinks they know the cache size to sub-MB accuracy. I have no problems with MB. Only thing I want to see is a bit more user friendly and consistent configuration. Initially I thought of bytes as oracle does but said 'let's be little more practical' and put KB..:-) MB is perfectly OK. It would just change the multiplier. And I don't think specifying a float is such a good idea. It is just so counter-intuitive. I mean how many people would care for fraction of an MB wasted? > As for the others, I'll side with Emerson: "a foolish consistency is the > hobgoblin of little minds". We know very well what the unit of > allocation of those is, and it's pages. There's no advantage to using > KB except making it harder to work out what's really happening. We > could measure max_connections in KB too if we had a mind to: there's > a very definite shared-mem cost per connection slot. Or the FSM > parameters, or checkpoint_segments, or max_locks_per_transaction. > The fact that they have quantifiable space costs doesn't mean that space > is the most useful way to measure them. Agreed. This is not to change things left and right. It is only to put some consistency in place. > BTW, were you intending to convert KB to NBuffers by charging exactly 8K > per buffer, or were you intending to allow for the additional shmem > costs such as buffer headers, per-buffer LWLocks, etc? If not the > latter, then what are you really measuring? For sure it's not shared > memory size --- charging an artificial number isn't going to help anyone > who's trying to pick shared_buffers to arrive at a particular actual > shmem size. But if it is the latter then it'll become even more > impossible to tell what's really happening, and we'll be forced to > invent some way of reading out how many buffers really got allocated. Well, for the purpose, this is beyond what I am trying to do. As of now there is a int value sitting in postgresql.conf which is in page blocks. Now there will be a conversion before it is used anytime so that it is bit more user friendly. The change should be skin deep so as to be low impact. As far putting a prefix such as K or M, I don't know much work that would be. Does that mean we need to convert shared_buffers to a string parameter and parse it? pgsql@mahawksoft.com(I would gladly write a real name but alas) said yesterday that there is a patch pending with Bruce for such a framework. I don't know what and how it does. For simplicity, I would convert all memory parameters to either KB or MB and state so in postgresql.conf. No floats no suffixes. This is my opinion of course. Any suggestions are always welcome.. Actually I need to find out few more things about it. It is not as simple as adding a assign_hook. When I tried to initdb with changes, it demanded 64MB of shared buffers which I (now) think that somewhere NBuffers are used before postgresql.conf is parsed. So 8192*8000=64MB. But this is just guesswork. Haven't looked in it there. If this seems reasonably OK, then I would spend some more time on it. We would need quite some documentation update then. > So I disagree with the premise. Measuring these things in KB is not an > improvement. As I said, KBs or MBs is not the issue. Not having it in terms of pagesize is. Shridhar
On Tuesday 01 June 2004 14:12, Shridhar Daithankar wrote:
> Actually I need to find out few more things about it. It is not as simple
> as adding a assign_hook. When I tried to initdb with changes, it demanded
> 64MB of shared buffers which I (now) think that somewhere NBuffers are used
> before postgresql.conf is parsed. So 8192*8000=64MB. But this is just
> guesswork. Haven't looked in it there.
Found it. Following is the code that is causing problem.
guc.c:2998
----------- if (conf->assign_hook) if (!(*conf->assign_hook) (newval, changeVal, source))
{ ereport(elevel, (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid value for parameter \"%s\": %d", name,
newval))); return false; }
if (changeVal || makeDefault) { if (changeVal) {
*conf->variable = newval; conf->gen.source = source; }
-----------
So even if assign_hook is executed, the value of variable is overwritten in
next step which nullifies any factoring/change in value done in assign hook.
I find this as a convention at many other place at guc.c. Call assign_hook and
the overwrite the value. So is assign_hook called only to validate the value?
How do I modify the value of the variable without getting specific?
I tried
if (changeVal && !(conf->assign_hook))
and it worked. However that is just for int variables. I am not sure if that
is a design decision. What should I do?
Regards,Shridhar
Shridhar, Tom, As long as we're messing around with PostgreSQL.conf, I propose that we comment the file much more thouroughly -- in the style of Apache's httpd.conf and our own pg_hba.conf (though maybe not quite as long as hba). Someone proposed this for 7.4 and we ran out of time, and as I've thought about the idea over the last 6 months I've come to like it. I'm happy to do the work for this ... it should be relatively easy if everyone's Doc patches are up to date. I can't see any drawback to it; is there something I'm missing? -- -Josh BerkusAglio Database SolutionsSan Francisco
Josh Berkus <josh@agliodbs.com> writes:
> As long as we're messing around with PostgreSQL.conf, I propose that we
> comment the file much more thouroughly -- in the style of Apache's httpd.conf
> and our own pg_hba.conf (though maybe not quite as long as hba).
ISTM we had decided that putting vast amounts of documentation into the
file comments was exactly the thing *not* to do. It duplicates the SGML
documentation, thereby doubling the maintenance effort, to very little
purpose. pg_hba's comments in particular had gotten quite out of hand
at one point, and have been pruned back severely. Let's not follow the
same wheel of fate for postgresql.conf.
regards, tom lane
Tom Lane wrote: > ISTM we had decided that putting vast amounts of documentation into the > file comments was exactly the thing *not* to do. It duplicates the SGML > documentation, thereby doubling the maintenance effort, to very little > purpose. I agree. If people really think that adding more comments to pg_hba.conf is a good idea, I think the best way to do that is to automatically extract that information from the main SGML docs. -Neil
Neil Conway wrote: > Tom Lane wrote: > > ISTM we had decided that putting vast amounts of documentation into the > > file comments was exactly the thing *not* to do. It duplicates the SGML > > documentation, thereby doubling the maintenance effort, to very little > > purpose. > > I agree. If people really think that adding more comments to > pg_hba.conf is a good idea, I think the best way to do that is to > automatically extract that information from the main SGML docs. And if folks want more info in postgresql.conf, we can perhaps pull them from the docs too, or at least the descriptions we added. Also, I it seems postgres --describe-config isn't working. It outputs nothing here. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Hi,
Any updates/opinions? Should we convert assign hooks to perform actual
assignment and custom validation instead of just custom validation? It is
clear from README that it is for validation purposes only..
Or Shall i look for some place else to perform conversion?
Shridhar
On Tuesday 01 June 2004 18:01, Shridhar Daithankar wrote:
> On Tuesday 01 June 2004 14:12, Shridhar Daithankar wrote:
> > Actually I need to find out few more things about it. It is not as simple
> > as adding a assign_hook. When I tried to initdb with changes, it demanded
> > 64MB of shared buffers which I (now) think that somewhere NBuffers are
> > used before postgresql.conf is parsed. So 8192*8000=64MB. But this is
> > just guesswork. Haven't looked in it there.
>
> Found it. Following is the code that is causing problem.
>
> guc.c:2998
> -----------
> if (conf->assign_hook)
> if (!(*conf->assign_hook) (newval, changeVal, source))
> {
> ereport(elevel,
> (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> errmsg("invalid value for parameter \"%s\": %d",
> name, newval)));
> return false;
> }
>
> if (changeVal || makeDefault)
> {
> if (changeVal)
> {
> *conf->variable = newval;
> conf->gen.source = source;
> }
> -----------
>
> So even if assign_hook is executed, the value of variable is overwritten in
> next step which nullifies any factoring/change in value done in assign
> hook.
>
> I find this as a convention at many other place at guc.c. Call assign_hook
> and the overwrite the value. So is assign_hook called only to validate the
> value? How do I modify the value of the variable without getting specific?
>
> I tried
>
> if (changeVal && !(conf->assign_hook))
Shridhar Daithankar <shridhar@frodo.hserus.net> writes:
> Any updates/opinions? Should we convert assign hooks to perform actual
> assignment and custom validation instead of just custom validation? It is
> clear from README that it is for validation purposes only..
As it should be. Assign hooks have no business altering the
user-supplied value.
We do have provisions for letting string assign hooks do that, but the
intended use of this was just for trivial display adjustments like
case-normalizing time zone names.
I remain unalterably opposed to the notion of measuring shared_buffers
in KB, but if you think you can get such a thing in over my objections,
the way to do it is to decouple the GUC parameter from NBuffers. The
GUC setting is whatever it is; you can reject the value if it's too
far out of range, but you do not editorialize upon it. What you do is
compute the derived value for NBuffers and assign that in the assign
hook.
regards, tom lane
Shridhar Daithankar <shridhar@frodo.hserus.net> writes:
>> I remain unalterably opposed to the notion of measuring shared_buffers
>> in KB, but if you think you can get such a thing in over my objections,
> Are you OK with MBs? I am fine with anything.
No, I'm not. shared_buffers should be measured in buffers (ie, pages).
Anything else is obscurantism. Not to mention highly likely to confuse
people who are used to how it's been set in the past.
regards, tom lane
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Also, I it seems postgres --describe-config isn't working. It outputs
> nothing here.
Yeah, same here. I'll take a look --- I may have side-swiped that during
recent hacking in main.c.
regards, tom lane
On Wednesday 02 June 2004 20:16, Tom Lane wrote: > Shridhar Daithankar <shridhar@frodo.hserus.net> writes: > > Any updates/opinions? Should we convert assign hooks to perform actual > > assignment and custom validation instead of just custom validation? It is > > clear from README that it is for validation purposes only.. > > As it should be. Assign hooks have no business altering the > user-supplied value. OK > I remain unalterably opposed to the notion of measuring shared_buffers > in KB, but if you think you can get such a thing in over my objections, Are you OK with MBs? I am fine with anything. > the way to do it is to decouple the GUC parameter from NBuffers. The > GUC setting is whatever it is; you can reject the value if it's too > far out of range, but you do not editorialize upon it. What you do is > compute the derived value for NBuffers and assign that in the assign > hook. That means removing NBuffers from declaration for config structure and substituting a dummy variable for it? If you think this is good and acceptable enough, I will proceed making changes that way. Shall I take this as guideline for other parameters as well? Shridhar
On Wednesday 02 June 2004 20:59, Tom Lane wrote: > Frank Wiles <frank@wiles.org> writes: > > This may be an unreasonable suggestion, but how about allowing both? > > I've seen several configuration systems do the following: > > > > shared_buffers = 10000 ( shared_buffers in pages ) > > shared_buffers = 100M ( 100 MBs of shared_buffers ) > > shared_buffers = 2048K ( 2MBs of shared_buffers ) > > I could live with that. I'm not sure how painful it would be to wedge > into GUC though; and I have a feeling that it is exactly *not* what > Shridhar would think is simpler than the present behavior ;-) Usability POV, I could live with that. That would mean converting the GUC from int to string though. Shridhar
Frank Wiles <frank@wiles.org> writes:
> This may be an unreasonable suggestion, but how about allowing both?
> I've seen several configuration systems do the following:
> shared_buffers = 10000 ( shared_buffers in pages )
> shared_buffers = 100M ( 100 MBs of shared_buffers )
> shared_buffers = 2048K ( 2MBs of shared_buffers )
I could live with that. I'm not sure how painful it would be to wedge
into GUC though; and I have a feeling that it is exactly *not* what
Shridhar would think is simpler than the present behavior ;-)
regards, tom lane
On Wed, 02 Jun 2004 11:05:43 -0400 Tom Lane <tgl@sss.pgh.pa.us> wrote: > Shridhar Daithankar <shridhar@frodo.hserus.net> writes: > >> I remain unalterably opposed to the notion of measuring > >shared_buffers> in KB, but if you think you can get such a thing in > >over my objections, > > > Are you OK with MBs? I am fine with anything. > > No, I'm not. shared_buffers should be measured in buffers (ie, > pages). Anything else is obscurantism. Not to mention highly likely > to confuse people who are used to how it's been set in the past. This may be an unreasonable suggestion, but how about allowing both? I've seen several configuration systems do the following: shared_buffers = 10000 ( shared_buffers in pages ) shared_buffers = 100M ( 100 MBs of shared_buffers ) shared_buffers= 2048K ( 2MBs of shared_buffers ) Using something like this would leave the old functionality in tact, allow users to use what they like, and shouldn't introducethat much complexity into the code. --------------------------------- Frank Wiles <frank@wiles.org> http://frank.wiles.org---------------------------------
In article <15702.1085933038@sss.pgh.pa.us>, Tom Lane <tgl@sss.pgh.pa.us> writes: > Shridhar Daithankar <shridhar@frodo.hserus.net> writes: >> I was toying around with idea of converting all the memory related >> parameters in postgresql.conf to kilobytes for simplicity and >> uniformity. > Why is that a good idea? Two reasons: 1. Some values are in KB, some in 8 KB 2. I find it easier to calculate in KB I'd like to see the following: * If the value is purely numeric, treat it as before (to ensure compatibility with older versions) * If the value is numeric with a prefix of [KMG], interpret it as KB, MB, or GB, respectively
Frank Wiles wrote: > shared_buffers = 10000 ( shared_buffers in pages ) > shared_buffers = 100M ( 100 MBs of shared_buffers ) > shared_buffers = 2048K ( 2MBs of shared_buffers ) I don't know if this is pedantic or just obsessive-compulsive, but I think it should be MB and KB (or more properly, kB) instead of just M and K, to distinguish from kilopages or megapages. mk
On Fri, 4 Jun 2004 13:10:02 +0300 Marko Karppinen <marko@karppinen.fi> wrote: > Frank Wiles wrote: > > shared_buffers = 10000 ( shared_buffers in pages ) > > shared_buffers = 100M ( 100 MBs of shared_buffers ) > > shared_buffers = 2048K ( 2MBs of shared_buffers ) > > I don't know if this is pedantic or just obsessive-compulsive, > but I think it should be MB and KB (or more properly, kB) > instead of just M and K, to distinguish from kilopages > or megapages. I can't see why anyone would have a problem with that. That extra keystroke isn't going to give anyone carpal tunnel! :) --------------------------------- Frank Wiles <frank@wiles.org> http://frank.wiles.org---------------------------------
Marko Karppinen <marko@karppinen.fi> writes: > Frank Wiles wrote: > > shared_buffers = 10000 ( shared_buffers in pages ) > > shared_buffers = 100M ( 100 MBs of shared_buffers ) > > shared_buffers = 2048K ( 2MBs of shared_buffers ) > > I don't know if this is pedantic or just obsessive-compulsive, > but I think it should be MB and KB (or more properly, kB) > instead of just M and K, to distinguish from kilopages > or megapages. Well, that's not how other tools handle it. I would suggest being consistent with the rest of the more user friendly unix toolset that does this. GNU dd for example uses these suffixes: > BLOCKS and BYTES may be followed by the following multiplicative suffixes: > xM M, c 1, w 2, b 512, kB 1000, K 1024, MB 1000*1000, M 1024*1024, > GB 1000*1000*1000, G 1024*1024*1024, and so on for T, P, E, Z, Y. (Though in this case the base unit is unambiguous because of the parameter. But other examples are the output of ls -lh, the -c and -n parameters of head, etc) -- greg