Thread: Re: [HACKERS] PGXLOG variable worthwhile?
Dave Page wrote: > Which in this case is what puzzles me. We are only talking about a > simple GUC variable after all - I don't know for sure, but I'm guessing > it's not a huge effort to add one? Can we get agreement on that? A GUC for pg_xlog location? Much cleaner than -X, doesn't have the problems of possible accidental use, and does allow pg_xlog moving without symlinks, which some people don't like? If I can get a few 'yes' votes I will add it to TODO and do it for 7.4. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
On Tue, 17 Sep 2002, Bruce Momjian wrote: > Dave Page wrote: > > Which in this case is what puzzles me. We are only talking about a > > simple GUC variable after all - I don't know for sure, but I'm guessing > > it's not a huge effort to add one? > > Can we get agreement on that? A GUC for pg_xlog location? Much cleaner > than -X, doesn't have the problems of possible accidental use, and does > allow pg_xlog moving without symlinks, which some people don't like? > > If I can get a few 'yes' votes I will add it to TODO and do it for 7.4. GUC instead of -X or PGXLOG : yes. However, how is that going to work if tablespaces are introduced in 7.4. Surely the same mechanism for tablespaces would be used for pg_xlog. As the tablespace mechanism hasn't been determined yet, as far as I know, wouldn't it be best to see what happens there before creating the TODO item for the log? -- Nigel J. Andrews Director --- Logictree Systems Limited Computer Consultants
"Nigel J. Andrews" wrote: <snip> > However, how is that going to work if tablespaces are introduced in 7.4. Surely > the same mechanism for tablespaces would be used for pg_xlog. As the tablespace > mechanism hasn't been determined yet, as far as I know, wouldn't it be best to > see what happens there before creating the TODO item for the log? It's a Yes from me of course. Would a TODO list entry of something like "Add a GUC xlog_path variable" be broad enough that people keep it in mind when tablespaces are created, but it doesn't get forgotten about by not being on the list? :-) Regards and best wishes, Justin Clift > -- > Nigel J. Andrews > Director > > --- > Logictree Systems Limited > Computer Consultants > > ---------------------------(end of broadcast)--------------------------- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/users-lounge/docs/faq.html -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi
Nigel J. Andrews wrote:
> On Tue, 17 Sep 2002, Bruce Momjian wrote:
>
> > Dave Page wrote:
> > > Which in this case is what puzzles me. We are only talking about a
> > > simple GUC variable after all - I don't know for sure, but I'm guessing
> > > it's not a huge effort to add one?
> >
> > Can we get agreement on that? A GUC for pg_xlog location? Much cleaner
> > than -X, doesn't have the problems of possible accidental use, and does
> > allow pg_xlog moving without symlinks, which some people don't like?
> >
> > If I can get a few 'yes' votes I will add it to TODO and do it for 7.4.
>
> GUC instead of -X or PGXLOG : yes.
>
> However, how is that going to work if tablespaces are introduced in 7.4. Surely
> the same mechanism for tablespaces would be used for pg_xlog. As the tablespace
> mechanism hasn't been determined yet, as far as I know, wouldn't it be best to
> see what happens there before creating the TODO item for the log?
Good point. How about:
Allow pg_xlog to be moved without symlinks
That is vague enough. Added to TODO.
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
Bruce Momjian wrote: > > Dave Page wrote: > > Which in this case is what puzzles me. We are only talking about a > > simple GUC variable after all - I don't know for sure, but I'm guessing > > it's not a huge effort to add one? > > Can we get agreement on that? A GUC for pg_xlog location? Much cleaner > than -X, doesn't have the problems of possible accidental use, and does > allow pg_xlog moving without symlinks, which some people don't like? > > If I can get a few 'yes' votes I will add it to TODO and do it for 7.4. 'yes' - make it one more GUC and done Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
"Nigel J. Andrews" wrote: > However, how is that going to work if tablespaces are introduced in 7.4. Surely > the same mechanism for tablespaces would be used for pg_xlog. As the tablespace > mechanism hasn't been determined yet, as far as I know, wouldn't it be best to > see what happens there before creating the TODO item for the log? No, tablespaces would have to be something DB specific, while the Xlog is instance wide (instance == one postmaster == installation == whatever you name that level). My vision is that we start off with two tablespaces per database, "default" and "default_idx", which are subdirectories inside the database directory. All (non-index-)objects created without explicitly saying what tablespace they belong to automatically belong to default. Indexes ... bla. The tablespace catalog will have a column telling the physical location of that directory. Moving it around will not be *that* easy, I guess, because the UPDATE of that entry has to go hand in hand with the move of all files in that damned directory. But that's another thing to sort out later, IMHO. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Jan Wieck wrote: > "Nigel J. Andrews" wrote: > > However, how is that going to work if tablespaces are introduced in 7.4. Surely > > the same mechanism for tablespaces would be used for pg_xlog. As the tablespace > > mechanism hasn't been determined yet, as far as I know, wouldn't it be best to > > see what happens there before creating the TODO item for the log? > > No, tablespaces would have to be something DB specific, while the Xlog > is instance wide (instance == one postmaster == installation == whatever > you name that level). > > My vision is that we start off with two tablespaces per database, > "default" and "default_idx", which are subdirectories inside the > database directory. All (non-index-)objects created without explicitly > saying what tablespace they belong to automatically belong to default. > Indexes ... bla. > > The tablespace catalog will have a column telling the physical location > of that directory. Moving it around will not be *that* easy, I guess, > because the UPDATE of that entry has to go hand in hand with the move of > all files in that damned directory. But that's another thing to sort out > later, IMHO. Yes, the nifty trick was to use a lstat() from pg_dump to learn if it is a symlink and if so, where it points to. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
On Tue, 17 Sep 2002, Bruce Momjian wrote: > Dave Page wrote: > > Which in this case is what puzzles me. We are only talking about a > > simple GUC variable after all - I don't know for sure, but I'm guessing > > it's not a huge effort to add one? > > Can we get agreement on that? A GUC for pg_xlog location? Much cleaner > than -X, doesn't have the problems of possible accidental use, and does > allow pg_xlog moving without symlinks, which some people don't like? > > If I can get a few 'yes' votes I will add it to TODO and do it for 7.4. Personally, I like the ability to define such at a command line level ... *especially* as it pertains to pointing to various directories ... I am against pulling the -X functionality out ... if you don't like it, don't use it ... add the GUC variable option to the mix, but don't take away functionality ... Hell, take a look at what you are saying above: because someone might forget to set -X, let's get rid of it in favor of a setting in a file that someone might forget to edit? Either format has the possibility of an error ... if you are so incompetent as to make that sort of mistake on a production server, it won't matter if its a GUC variable, environment variable or commnd line argument, you will still make that mistake ...
Marc G. Fournier wrote: > On Tue, 17 Sep 2002, Bruce Momjian wrote: > > > Dave Page wrote: > > > Which in this case is what puzzles me. We are only talking about a > > > simple GUC variable after all - I don't know for sure, but I'm guessing > > > it's not a huge effort to add one? > > > > Can we get agreement on that? A GUC for pg_xlog location? Much cleaner > > than -X, doesn't have the problems of possible accidental use, and does > > allow pg_xlog moving without symlinks, which some people don't like? > > > > If I can get a few 'yes' votes I will add it to TODO and do it for 7.4. > > Personally, I like the ability to define such at a command line level ... > *especially* as it pertains to pointing to various directories ... I am > against pulling the -X functionality out ... if you don't like it, don't > use it ... add the GUC variable option to the mix, but don't take away > functionality ... > > Hell, take a look at what you are saying above: because someone might > forget to set -X, let's get rid of it in favor of a setting in a file that > someone might forget to edit? > > Either format has the possibility of an error ... if you are so > incompetent as to make that sort of mistake on a production server, it > won't matter if its a GUC variable, environment variable or commnd line > argument, you will still make that mistake ... Sorry, I don't see the logic here. Using postgresql.conf, you set it once and it remains set until you change it again. With -X, you have to use it every time. I think that's where the votes came from. You argued that -X and GUC make sense, but why add -X when can get it done at once in postgresql.conf. Also, consider changing the location does require moving the WAL files, so you already have this extra step. Adding to postgresql.conf is easy. I don't think you can just point it at a random empty directory on startup. Our goal was to reduce params to postmaster/postgres in favor of GUC, not add to them. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
On Wed, 18 Sep 2002, Bruce Momjian wrote:
> Sorry, I don't see the logic here. Using postgresql.conf, you set it
> once and it remains set until you change it again. With -X, you have to
> use it every time. I think that's where the votes came from.
Ah, so you are saying that you type out your full command line each and
every time you start up the server? I know, in my case, I have a shell
script setup that I edit my changes in so that I don't have to remember
...
> You argued that -X and GUC make sense, but why add -X when can get it
> done at once in postgresql.conf. Also, consider changing the location
> does require moving the WAL files, so you already have this extra step.
> Adding to postgresql.conf is easy. I don't think you can just point it
> at a random empty directory on startup. Our goal was to reduce params
> to postmaster/postgres in favor of GUC, not add to them.
I don't disagree that editing postgresql.conf is easy, but its not
something that ppl would naturally thing of ... if I want to move a
directory with most servers I run, I will generally do a man to find out
what command options are required to do this change, and, if none are
provided, just create a god-forsaken symlink ...
The man page for postmaster should have something in it like:
-X <directory> Specifies an alternate location for WAL files. Superseded
by setting xlog_path in postmaster.conf
Hell, if you are going to remove -X because its 'easier to do it in
postmaster.conf', you should be looking at removing *all* command line
args that are better represented in the postmaster.conf file ...
The only time that *I* use the postmaster.conf file is when I'm playing
with the various scan'ng options ... why?
mars# ps aux | grep -- B
pgsql 133 0.0 0.0 77064 1512 con- S Mon10PM 3:21.15 /usr/local/bin/postmaster -B 8192 -N 512 -o -S 4096 -i
-p5432 -D/v1/pgsql (postgres)
pgsql 144 0.0 0.0 1097300 1372 ?? Is Mon10PM 0:06.04 /usr/local/pgsql/bin/postmaster -B 131072 -N 2048 -i
-p5433 -D/usr/local/pgsql/5433 -S (postgres)
its nice to be able to do a simple ps to find out which process is which,
and pointing where ... other then -D, I don't believe there is one option
in there that I couldn't have set in the postmaster.conf file, but, then,
to find out the various settings, I'd have to do ps to figure out where
the database files are stored, and then go look at the postmaster.conf
file to figure out what each are set to ...
I have one server that has 10 instances running right now:
jupiter# ps ax | grep -- -B
373 ?? Ss 0:55.31 /usr/local/pgsql721/bin/postmaster -B 10240 -N 512 -i -p 5432 -D/v1/pgsql/5432 -S (postgres)
383 ?? Ss 0:11.78 /usr/local/pgsql/bin/postmaster -B 64 -N 16 -i -p 5434 -D/v1/pgsql/5434 -S (postgres)
394 ?? Ss 0:17.82 /usr/local/pgsql/bin/postmaster -B 1024 -N 256 -i -p 5437 -D/v1/pgsql/5437 -S (postgres)
405 ?? Ss 0:16.46 /usr/local/pgsql/bin/postmaster -B 256 -N 128 -i -p 5440 -D/v1/pgsql/5440 -S (postgres)
416 ?? Ss 0:10.93 /usr/local/pgsql/bin/postmaster -B 256 -N 128 -i -p 5449 -D/v1/pgsql/5449 -S (postgres)
427 ?? Ss 0:16.30 /usr/local/pgsql/bin/postmaster -B 2048 -N 256 -i -p 5443 -D/v1/pgsql/5443 -S (postgres)
438 ?? Ss 0:10.60 /usr/local/pgsql721/bin/postmaster -B 1024 -N 512 -i -p 5446 -D/v1/pgsql/5446 -S (postgres)
88515 ?? Ss 0:10.05 /usr/local/pgsql/bin/postmaster -B 64 -N 16 -i -p 5433 -D/v1/pgsql/5433 -S (postgres)
13029 pi S+ 0:00.00 grep -- -B
445 con- S 0:10.59 /usr/local/pgsql/mb/bin/postmaster -B 256 -N 128 -i -p 5448 -D/v1/pgsql/openacs4 (postgres)
460 con- S 0:10.40 /usr/local/pgsql/bin/postmaster -B 64 -N 16 -i -p 5436 -D/v1/pgsql/electrichands (postgres)
All the information for each are right there in front of me ... I don't
have to go through 10 postmaster.conf files to figure out anything ...
the GUC value should override the command line option, agreed ... but the
ability to use the command line should not be removed just because some
ppl aren't competent enough to adjust their startup scripts if they change
their system ...
Marc G. Fournier wrote: > On Wed, 18 Sep 2002, Bruce Momjian wrote: > > > Sorry, I don't see the logic here. Using postgresql.conf, you set it > > once and it remains set until you change it again. With -X, you have to > > use it every time. I think that's where the votes came from. > > Ah, so you are saying that you type out your full command line each and > every time you start up the server? I know, in my case, I have a shell > script setup that I edit my changes in so that I don't have to remember > ... Yep, but your central place for changes should be postgresql.conf, not the command line. If we tried go get every GUC param on the command line it would be unusable. > > You argued that -X and GUC make sense, but why add -X when can get it > > done at once in postgresql.conf. Also, consider changing the location > > does require moving the WAL files, so you already have this extra step. > > Adding to postgresql.conf is easy. I don't think you can just point it > > at a random empty directory on startup. Our goal was to reduce params > > to postmaster/postgres in favor of GUC, not add to them. > > I don't disagree that editing postgresql.conf is easy, but its not > something that ppl would naturally thing of ... if I want to move a > directory with most servers I run, I will generally do a man to find out > what command options are required to do this change, and, if none are > provided, just create a god-forsaken symlink ... > > The man page for postmaster should have something in it like: > > -X <directory> Specifies an alternate location for WAL files. Superseded > by setting xlog_path in postmaster.conf > > Hell, if you are going to remove -X because its 'easier to do it in > postmaster.conf', you should be looking at removing *all* command line > args that are better represented in the postmaster.conf file ... Well, those other options are things you may want to change frequently. The xlog directory isn't going to be moving around, we hope. We have the flags there only so they can be easily adjusted for testing, I think, and in fact there has been discussion about removing more of them. > its nice to be able to do a simple ps to find out which process is which, > and pointing where ... other then -D, I don't believe there is one option > in there that I couldn't have set in the postmaster.conf file, but, then, > to find out the various settings, I'd have to do ps to figure out where > the database files are stored, and then go look at the postmaster.conf > file to figure out what each are set to ... Yea, but you aren't going to be needing to know the xlog directory that way, will you? Fact is, xlog is seldom moved, and symlinks do it fine now. The GUC was a compromise for people who didn't like symlinks. If we are getting pushback from GUC we may as well just drop the GUC idea and stick with symlinks. I think that's how the vote went last time and it seems to be heading in that direction again. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
"Marc G. Fournier" <scrappy@hub.org> writes:
> On Wed, 18 Sep 2002, Bruce Momjian wrote:
>> Sorry, I don't see the logic here. Using postgresql.conf, you set it
>> once and it remains set until you change it again. With -X, you have to
>> use it every time. I think that's where the votes came from.
> Ah, so you are saying that you type out your full command line each and
> every time you start up the server?
Let's put it this way: would you be in favor of adding a
--please-don't-wipe-my-database-directory
switch to the postmaster? And if you forget to specify that every time
you start the postmaster, we do an instant "rm -rf $PGDATA"?
Doesn't seem like a good idea, does it?
Well, specifying the XLOG location on the command line or as an
environment variable is just about as deadly as the above loaded-gun-
pointed-at-foot scenario. You start the postmaster with the wrong
context, even once, it's sayonara to your data integrity.
The point of insisting that the XLOG location be recorded *inside*
the data directory is to prevent simple admin errors from being
catastrophic. Do you remember when we regularly saw trouble reports
from people who'd corrupted their database indexes by starting the
postmaster with different LOCALE environments at different times? We
fixed that by forcing the locale collation order to be specified inside
the database directory (in pg_control, but the details are not important
here), rather than allowing it to be taken from postmaster environment.
If we allow XLOG location to be determined by a postmaster switch or
environment variable, then we *will* be opening the door for people
to shoot themselves in the foot just like they used to do with locale.
I learned something from those problems, and I do not intend to make
the same mistake again.
regards, tom lane
On Wed, 18 Sep 2002, Tom Lane wrote: > "Marc G. Fournier" <scrappy@hub.org> writes: > > On Wed, 18 Sep 2002, Bruce Momjian wrote: > >> Sorry, I don't see the logic here. Using postgresql.conf, you set it > >> once and it remains set until you change it again. With -X, you have to > >> use it every time. I think that's where the votes came from. > > > Ah, so you are saying that you type out your full command line each and > > every time you start up the server? > > Let's put it this way: would you be in favor of adding a > --please-don't-wipe-my-database-directory > switch to the postmaster? And if you forget to specify that every time > you start the postmaster, we do an instant "rm -rf $PGDATA"? > > Doesn't seem like a good idea, does it? > > Well, specifying the XLOG location on the command line or as an > environment variable is just about as deadly as the above loaded-gun- > pointed-at-foot scenario. You start the postmaster with the wrong > context, even once, it's sayonara to your data integrity. > > The point of insisting that the XLOG location be recorded *inside* > the data directory is to prevent simple admin errors from being > catastrophic. Do you remember when we regularly saw trouble reports > from people who'd corrupted their database indexes by starting the > postmaster with different LOCALE environments at different times? We > fixed that by forcing the locale collation order to be specified inside > the database directory (in pg_control, but the details are not important > here), rather than allowing it to be taken from postmaster environment. > > If we allow XLOG location to be determined by a postmaster switch or > environment variable, then we *will* be opening the door for people > to shoot themselves in the foot just like they used to do with locale. > > I learned something from those problems, and I do not intend to make > the same mistake again. Except that you are ... you are assuming that someone is going to edit their postmaster.conf file correctly ... if you want to avoid making the same mistake again, there should be some sort of 'tag' that associates the files in the XLOG directory with the data directories themselves, regardless of *how* the XLOG directory is referenced ... something that links them at a level that an administrator *can't* make a mistake about ... all forcing the use of the postmaster.conf file is doing is reducing options, it isn't making sure that the XLOG directory pointed to is apporopraite for the data directory itself ...