Thread: Renaming of pg_xlog and pg_clog
Hi all, I am relaunching $subject as 10 development will begin soon. As far as I know, there is agreement that we can do something here. Among the different proposals I have found: - pg_clog renamed to pg_commit_status, pg_xact or pg_commit - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal Another idea from Stephen (https://www.postgresql.org/message-id/20160826003715.GG4028@tamriel.snowman.net) would be to put everything that is temporary and not WAL-logged into a single place to facilitate the filtering work of backup tools. A straight renaming would be a simple patch (including pg_upgrade part), and if we actually do it for 10.0 it would be good to do it now instead of in 3 months. I don't mind writing a patch for it. Now, one of the things discussed as well was that we may want to still keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is to not break the existing tools. Personally, I'd prefer a hard break. That would not be complicated to fix for any tool maintainers. Thoughts? -- Michael
On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote: > Now, one of the things discussed as well was that we may want to still > keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is > to not break the existing tools. Personally, I'd prefer a hard break. > That would not be complicated to fix for any tool maintainers. I agree on a hard break, unless we get pushback from users, and even then, they can create the symlinks themselves. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
On Fri, Aug 26, 2016 at 11:55 AM, Bruce Momjian <bruce@momjian.us> wrote: > On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote: >> Now, one of the things discussed as well was that we may want to still >> keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is >> to not break the existing tools. Personally, I'd prefer a hard break. >> That would not be complicated to fix for any tool maintainers. > > I agree on a hard break, unless we get pushback from users, and even > then, they can create the symlinks themselves. I strongly prefer symlink approach not to break many existing tools and scripts. Regards, -- Fujii Masao
* Bruce Momjian (bruce@momjian.us) wrote: > On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote: > > Now, one of the things discussed as well was that we may want to still > > keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is > > to not break the existing tools. Personally, I'd prefer a hard break. > > That would not be complicated to fix for any tool maintainers. > > I agree on a hard break, unless we get pushback from users, and even > then, they can create the symlinks themselves. I'm in favor of the hard break generally, but I'd like David's input on it, to be sure we're not missing something. Thanks! Stephen
Re: Fujii Masao 2016-08-26 <CAHGQGwHK2yimfLvG_WQ1Vrq2h+CMzgv5u6OEmxr-cbJRO+WKWQ@mail.gmail.com> > > I agree on a hard break, unless we get pushback from users, and even > > then, they can create the symlinks themselves. > > I strongly prefer symlink approach not to break many existing tools > and scripts. Symlinks might actually be worse than removing the directories altogether. If your backup tool fails because the pg_xlog directory is gone, you'll hopefully notice, but if you end up with a backup that consists merely of a copy of a symlink named pg_xlog, you might not notice. Christoph
On Fri, Aug 26, 2016 at 12:44 PM, Christoph Berg <myon@debian.org> wrote:
Re: Fujii Masao 2016-08-26 <CAHGQGwHK2yimfLvG_WQ1Vrq2h+CMzgv5u6OEmxr-cbJRO+WKWQ@mail. gmail.com>
> > I agree on a hard break, unless we get pushback from users, and even
> > then, they can create the symlinks themselves.
>
> I strongly prefer symlink approach not to break many existing tools
> and scripts.
Symlinks might actually be worse than removing the directories
altogether. If your backup tool fails because the pg_xlog directory is
gone, you'll hopefully notice, but if you end up with a backup that
consists merely of a copy of a symlink named pg_xlog, you might not
notice.
+1. It's *much* better to cause a clean break. That way people will notice it, and can fix it (and it will be an easy fix).
An unclean break might leave people with things that look like they work, but don't. That's a lot more dangerous.
Same reason I'm also +1 for Stephens suggestion to put all things that should not be in a base backup into the same directory. That may break things now, but it will simplify things down the road. And doing it at the same time as renaming these things makes a lot of sense, because it causes breakage that tool-builders *have* to look at, and then they will hopefully also notice the other change.
On 26 August 2016 at 04:39, Michael Paquier <michael.paquier@gmail.com> wrote: > Hi all, > > I am relaunching $subject as 10 development will begin soon. As far as > I know, there is agreement that we can do something here. Among the > different proposals I have found: > - pg_clog renamed to pg_commit_status, pg_xact or pg_commit > - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal Don't mean to be a party pooper, but what discussion and agreement are we referring to here? If we are going to suggest doing something we really should summarize the reason for doing it rather than assume it is self evident, cos it certainly isn't. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Fri, Aug 26, 2016 at 8:31 PM, Simon Riggs <simon@2ndquadrant.com> wrote: > On 26 August 2016 at 04:39, Michael Paquier <michael.paquier@gmail.com> wrote: >> Hi all, >> >> I am relaunching $subject as 10 development will begin soon. As far as >> I know, there is agreement that we can do something here. Among the >> different proposals I have found: >> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit >> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal > > Don't mean to be a party pooper, but what discussion and agreement are > we referring to here? > > If we are going to suggest doing something we really should summarize > the reason for doing it rather than assume it is self evident, cos it > certainly isn't. This thread was the previous one on the matter: http://www.postgresql.org/message-id/CAASwCXcVGma9KgEu-ESC6u928mW67noZvnawbPUSW7R7AN9UVg@mail.gmail.com In short, with the current names, sometimes users think that pg_xlog and pg_clog are just logs. And so it is fine to delete them to free up space, corrupting their cluster, because they are just *logs*. Personally I have seen that, and based on the thread I am not the only one. -- Michael
Hi, On Fri, 2016-08-26 at 21:12 +0900, Michael Paquier wrote: > In short, with the current names, sometimes users think that pg_xlog > and pg_clog are just logs. And so it is fine to delete them to free up > space, corrupting their cluster, because they are just *logs*. ...and we also have "pg_logical", that includes a "log" keyword already... Regards, -- Devrim GÜNDÜZ EnterpriseDB: http://www.enterprisedb.com PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer Twitter: @DevrimGunduz , @DevrimGunduzTR
On 26-08-2016 09:25, Devrim Gündüz wrote: > On Fri, 2016-08-26 at 21:12 +0900, Michael Paquier wrote: >> In short, with the current names, sometimes users think that pg_xlog >> and pg_clog are just logs. And so it is fine to delete them to free up >> space, corrupting their cluster, because they are just *logs*. > > ...and we also have "pg_logical", that includes a "log" keyword already... > "clog" and "xlog" is almost "log"; "logical" is not. I don't imagine people confusing "logical" meaning "log". -- Euler Taveira Timbira - http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte24x7 e Treinamento
Euler Taveira <euler@timbira.com.br> writes:
> On 26-08-2016 09:25, Devrim Gündüz wrote:
>> ...and we also have "pg_logical", that includes a "log" keyword already...
> "clog" and "xlog" is almost "log"; "logical" is not. I don't imagine
> people confusing "logical" meaning "log".
Well, I dunno; people with a weak grasp of English might have an issue
there. But I never liked that directory name anyway; my problem with
it is I read "pg_logical" and think "logical what?". Naming things using
a disconnected adjective is not good, especially one with as many
potential applications as that one has. If we're up for renaming things
under PGDATA, that one is high on my hit list.
regards, tom lane
* Tom Lane (tgl@sss.pgh.pa.us) wrote: > Euler Taveira <euler@timbira.com.br> writes: > > On 26-08-2016 09:25, Devrim Gündüz wrote: > >> ...and we also have "pg_logical", that includes a "log" keyword already... > > > "clog" and "xlog" is almost "log"; "logical" is not. I don't imagine > > people confusing "logical" meaning "log". > > Well, I dunno; people with a weak grasp of English might have an issue > there. But I never liked that directory name anyway; my problem with > it is I read "pg_logical" and think "logical what?". Naming things using > a disconnected adjective is not good, especially one with as many > potential applications as that one has. If we're up for renaming things > under PGDATA, that one is high on my hit list. iirc, pg_logical also has both temporary and non-temporary data in it too. That complicates things for backup utilities that are trying to exclude all temporary files. If we actually move all the temp files into their own directory (or tree), then we're changing what's in pg_logical anyway, so renaming it seems like a good idea. Thanks! Stephen
On 08/26/2016 03:48 AM, Magnus Hagander wrote: > > Same reason I'm also +1 for Stephens suggestion to put all things that > should not be in a base backup into the same directory. That may break > things now, but it will simplify things down the road. And doing it at > the same time as renaming these things makes a lot of sense, because it > causes breakage that tool-builders *have* to look at, and then they will > hopefully also notice the other change. If this is done this fall, developers will have at least a year to fix their utilities. JD -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone appreciates your honesty, until you are honest with them. Unless otherwise stated, opinions are my own.
On 08/25/2016 07:39 PM, Michael Paquier wrote: > Hi all, > > I am relaunching $subject as 10 development will begin soon. As far as > I know, there is agreement that we can do something here. Among the > different proposals I have found: > - pg_clog renamed to pg_commit_status, pg_xact or pg_commit > - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal I think the use of the pg_ prefix is redundant. Just a directory called: wal will do (for example). In reference to pg_xlog specifically, it should be wal. It is the Write Ahead Log, not the Journal (although I recognize they can be synonymous). All the documentation talks about Write Ahead Log. Sincerely, JD -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone appreciates your honesty, until you are honest with them. Unless otherwise stated, opinions are my own.
<p dir="ltr"><p dir="ltr">On Aug 26, 2016 5:13 PM, "Joshua D. Drake" <<a href="mailto:jd@commandprompt.com">jd@commandprompt.com</a>>wrote:<br /> ><br /> > On 08/25/2016 07:39 PM, MichaelPaquier wrote:<br /> >><br /> >> Hi all,<br /> >><br /> >> I am relaunching $subject as 10development will begin soon. As far as<br /> >> I know, there is agreement that we can do something here. Among the<br/> >> different proposals I have found:<br /> >> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit<br/> >> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal<br /> ><br /> ><br /> > I think theuse of the pg_ prefix is redundant. Just a directory called: wal will do (for example).<br /> ><br /> > In referenceto pg_xlog specifically, it should be wal. It is the Write Ahead Log, not the Journal (although I recognize theycan be synonymous). All the documentation talks about Write Ahead Log.<br /> ><br /><p dir="ltr">Yes, let's avoidinventing a *third* name for it, please. It's already bad enough that we have both wal and xlog. <p dir="ltr">/Magnus<br />
On 8/25/16 10:39 PM, Michael Paquier wrote: > I am relaunching $subject as 10 development will begin soon. As far as > I know, there is agreement that we can do something here. Among the > different proposals I have found: > - pg_clog renamed to pg_commit_status, pg_xact or pg_commit > - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal If we're going to do some renaming, then I suggest we do a mini-file-system structure under $PGDATA, like $PGDATA/etc $PGDATA/log $PGDATA/run (lock files etc.) $PGDATA/tmp $PGDATA/var The names of all the things under "var" could still be refined, but it's much less likely that users will confuse data with configuration or plain logs under that scheme. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
<p dir="ltr"><p dir="ltr">On Aug 26, 2016 5:54 PM, "Peter Eisentraut" <<a href="mailto:peter.eisentraut@2ndquadrant.com">peter.eisentraut@2ndquadrant.com</a>>wrote:<br /> ><br /> > On 8/25/1610:39 PM, Michael Paquier wrote:<br /> > > I am relaunching $subject as 10 development will begin soon. As faras<br /> > > I know, there is agreement that we can do something here. Among the<br /> > > different proposalsI have found:<br /> > > - pg_clog renamed to pg_commit_status, pg_xact or pg_commit<br /> > > - pg_xlogrenamed to pg_xjournal, pg_wal or pg_journal<br /> ><br /> > If we're going to do some renaming, then I suggestwe do a<br /> > mini-file-system structure under $PGDATA, like<br /> ><br /> > $PGDATA/etc<br /> > $PGDATA/log<br/> > $PGDATA/run (lock files etc.)<br /> > $PGDATA/tmp<br /> > $PGDATA/var<br /> ><br /> > Thenames of all the things under "var" could still be refined, but it's<br /> > much less likely that users will confusedata with configuration or<br /> > plain logs under that scheme<p dir="ltr">Interesting idea. I worry a bit thatthis might encourage distributions to split it up into different places though, and I'm not sure we want to encouragethat.. <p dir="ltr">/Magnus
Magnus Hagander <magnus@hagander.net> writes:
> On Aug 26, 2016 5:54 PM, "Peter Eisentraut" <
> peter.eisentraut@2ndquadrant.com> wrote:
>> If we're going to do some renaming, then I suggest we do a
>> mini-file-system structure under $PGDATA, like
>>
>> $PGDATA/etc
>> $PGDATA/log
>> $PGDATA/run (lock files etc.)
>> $PGDATA/tmp
>> $PGDATA/var
>>
>> The names of all the things under "var" could still be refined, but it's
>> much less likely that users will confuse data with configuration or
>> plain logs under that scheme
> Interesting idea. I worry a bit that this might encourage distributions to
> split it up into different places though, and I'm not sure we want to
> encourage that..
Yeah, I'm afraid that these names are not as well standardized as Peter
probably wishes they were. Also, I'd just as soon not move/rename things
that don't really need it. If, for example, we decide to move
postgresql.conf to etc/postgresql.conf, that is going to break a metric
ton of stuff that doesn't need to get broken AFAICS.
regards, tom lane
On 08/26/2016 09:28 AM, Tom Lane wrote: > Magnus Hagander <magnus@hagander.net> writes: >> On Aug 26, 2016 5:54 PM, "Peter Eisentraut" < >> peter.eisentraut@2ndquadrant.com> wrote: >>> If we're going to do some renaming, then I suggest we do a >>> mini-file-system structure under $PGDATA, like >>> >>> $PGDATA/etc >>> $PGDATA/log >>> $PGDATA/run (lock files etc.) >>> $PGDATA/tmp >>> $PGDATA/var >>> >>> The names of all the things under "var" could still be refined, but it's >>> much less likely that users will confuse data with configuration or >>> plain logs under that scheme > >> Interesting idea. I worry a bit that this might encourage distributions to >> split it up into different places though, and I'm not sure we want to >> encourage that.. > > Yeah, I'm afraid that these names are not as well standardized as Peter > probably wishes they were. Also, I'd just as soon not move/rename things > that don't really need it. If, for example, we decide to move > postgresql.conf to etc/postgresql.conf, that is going to break a metric > ton of stuff that doesn't need to get broken AFAICS. I am not so sure that is accurate. Yes, Windows is an outlying but any Unix person is going to easily understand etc log run. Further as Linux is by far our most run platform (outside of possibly Windows) it will feel right at home for the largest growing user base. Personally I really like the idea of: $PGDATA/etc/postgresql.conf $PGDATA/log/Thu.log $PGDATA/run/postmaster.pid $PGDATA/var/base/16758 Although... wouldn't run be under var? JD > > regards, tom lane > > -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone appreciates your honesty, until you are honest with them.
On 26 August 2016 at 18:28, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Also, I'd just as soon not move/rename things that don't really need it. +1 Let's leave everything exactly as it is now... but put a small README in each directory to explain why files in it shouldn't be deleted to make space. That helps the few people who made such mistakes, but doesn't cause massive change as a result. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 8/26/16 11:58 AM, Magnus Hagander wrote: > $PGDATA/etc >> $PGDATA/log >> $PGDATA/run (lock files etc.) >> $PGDATA/tmp >> $PGDATA/var >> >> The names of all the things under "var" could still be refined, but it's >> much less likely that users will confuse data with configuration or >> plain logs under that scheme > > Interesting idea. I worry a bit that this might encourage distributions > to split it up into different places though, and I'm not sure we want to > encourage that.. They already do this. This would just formalize where we think appropriate boundaries between the pieces are. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2016-08-26 11:53:21 -0400, Peter Eisentraut wrote: > On 8/25/16 10:39 PM, Michael Paquier wrote: > > I am relaunching $subject as 10 development will begin soon. As far as > > I know, there is agreement that we can do something here. Among the > > different proposals I have found: > > - pg_clog renamed to pg_commit_status, pg_xact or pg_commit > > - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal > > If we're going to do some renaming, then I suggest we do a > mini-file-system structure under $PGDATA, like > > $PGDATA/etc > $PGDATA/log > $PGDATA/run (lock files etc.) > $PGDATA/tmp > $PGDATA/var > > The names of all the things under "var" could still be refined, but it's > much less likely that users will confuse data with configuration or > plain logs under that scheme. Splitting of ephemeral data seems to have a benefit, the rest seems more like rather noisy busywork to me.
On 8/26/16 12:28 PM, Tom Lane wrote: > Also, I'd just as soon not move/rename things > that don't really need it. I'm just as happy with not changing anything. But if we're going to rename stuff, let's at least think about something slightly more comprehensive. Any rename is going to break a bunch of stuff. But if we break it in a way that reduces the need for future discussion or changes, it would at least be a small win in the long run. > If, for example, we decide to move > postgresql.conf to etc/postgresql.conf, that is going to break a metric > ton of stuff that doesn't need to get broken AFAICS. Sure, I'd be content with leaving those in the top-level instead of say "etc". -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 8/26/16 4:02 PM, Joshua D. Drake wrote: > Although... wouldn't run be under var? Traditionally yes, but things are changing in this area, if you consider the top-level file system of a modern Linux distribution. One reason is that "run" is/can be blown away at reboot. This wouldn't be an entirely useless feature for postgres: Can you tell otherwise which of the various pid/lock/opts files you can delete if you have killed the processes and want to eliminate any left-over state? Is postmaster.opts a configuration file or a state file? -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2016-08-26 17:11:00 -0400, Peter Eisentraut wrote: > On 8/26/16 12:28 PM, Tom Lane wrote: > > Also, I'd just as soon not move/rename things > > that don't really need it. > > I'm just as happy with not changing anything. But if we're going to > rename stuff, let's at least think about something slightly more > comprehensive. Any rename is going to break a bunch of stuff. But if > we break it in a way that reduces the need for future discussion or > changes, it would at least be a small win in the long run. I do think there's an order of magnitude between the impact between moving some and moving everything. And that's going to impact cost/benefit calculations. Moving e.g. all ephemeral files into a (possibly configurable) directory is going to hardly impact anyone. Renaming pg_logical into something different (FWIW, it was originally named differently...) will hopefully impact nobody, excepting some out of date file exclusion lists possibly. But moving config files, and even pg_xlog (which we document to be symlinkable somewhere else) imo is different. The other thing is that the likelihood of getting anywhere by doing radical one-off redesigns is approximately 0.
On 8/26/16 5:20 PM, Andres Freund wrote: > I do think there's an order of magnitude between the impact between > moving some and moving everything. And that's going to impact > cost/benefit calculations. > > Moving e.g. all ephemeral files into a (possibly configurable) directory > is going to hardly impact anyone. Renaming pg_logical into something > different (FWIW, it was originally named differently...) will hopefully > impact nobody, excepting some out of date file exclusion lists possibly. > > But moving config files, and even pg_xlog (which we document to be > symlinkable somewhere else) imo is different. I agree with all that. But the subject line is specifically about moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog, then that is noted. But if we were to move it, we can think about a good place to move it to. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote: > I agree with all that. But the subject line is specifically about > moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog, > then that is noted. But if we were to move it, we can think about a > good place to move it to. I think it's probably worth moving pg_xlog, because the benefit also includes preventing a few users from shooting themselves somewhere vital. That's imo much less the case for some of the other moves. But I still don't think think a largescale reorganization is a good idea, it'll just stall and nothing will happen.
On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres@anarazel.de> wrote: > On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote: >> I agree with all that. But the subject line is specifically about >> moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog, >> then that is noted. But if we were to move it, we can think about a >> good place to move it to. > > I think it's probably worth moving pg_xlog, because the benefit also > includes preventing a few users from shooting themselves somewhere > vital. That's imo much less the case for some of the other moves. But I > still don't think think a largescale reorganization is a good idea, > it'll just stall and nothing will happen. OK, so let's focus only on the renaming mentioned in $subject. So far as I can see on this thread, here are the opinions of people who clearly gave one: - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on David's input), Magnus - Rename them, hard break not OK: Fujii-san (perhaps do nothing?) - Do nothing: Simon (add a README), Tom, Peter E As far as I can see, there is a consensus to not rename pg_xlog to pg_journal and avoid using a third meaning, but instead use pg_wal. I guess that now the other renaming would be pg_clog -> pg_xact. Other opinions? Forgot you here? -- Michael
On 27/08/16 20:33, Michael Paquier wrote: > On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres@anarazel.de> wrote: >> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote: >>> I agree with all that. But the subject line is specifically about >>> moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog, >>> then that is noted. But if we were to move it, we can think about a >>> good place to move it to. >> I think it's probably worth moving pg_xlog, because the benefit also >> includes preventing a few users from shooting themselves somewhere >> vital. That's imo much less the case for some of the other moves. But I >> still don't think think a largescale reorganization is a good idea, >> it'll just stall and nothing will happen. > OK, so let's focus only on the renaming mentioned in $subject. So far > as I can see on this thread, here are the opinions of people who > clearly gave one: > - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on > David's input), Magnus > - Rename them, hard break not OK: Fujii-san (perhaps do nothing?) > - Do nothing: Simon (add a README), Tom, Peter E > > As far as I can see, there is a consensus to not rename pg_xlog to > pg_journal and avoid using a third meaning, but instead use pg_wal. I > guess that now the other renaming would be pg_clog -> pg_xact. Other > opinions? Forgot you here? I think if there are going to be things in pg that break software - for good reasons, like making future usage easier at the cost an initial sharp pain - then to do so in version '10.0.0' is very appropriate! IMHO And better to do so in 10.0.0 (especially if closely related), rather than 10.1.0 (or whatever the next version after that is named). So, if other things might cause breakages, do so IN 10.0.0 - rather than hold back - assuming that there won't be hundreds or more major breakages!!! Cheers, Gavin
Michael Paquier <michael.paquier@gmail.com> writes:
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input), Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E
Hm, if you read me as voting against renaming pg_xlog, that wasn't
the conclusion I meant to convey. I'm against moving/renaming the
configuration files, because I think that will break a lot of users'
scripts and habits without really buying much. But I'm for consolidating
all the files that should not be copied by backup tools into one
subdirectory, and I think that while we're doing that it would be sensible
to rename pg_xlog and pg_clog to something that doesn't sound like it's
scratch data. I'm on the fence about whether pg_logical ought to get
renamed.
> As far as I can see, there is a consensus to not rename pg_xlog to
> pg_journal and avoid using a third meaning, but instead use pg_wal.
Yeah, +1 for pg_wal, we do not need yet another name for that.
> I guess that now the other renaming would be pg_clog -> pg_xact.
We already have pg_subtrans, so it seems like pg_trans is an obvious
suggestion. I'm not sure whether the other precedent of pg_multixact
is a stronger one than that.
regards, tom lane
Michael Paquier wrote: > On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres@anarazel.de> wrote: > > On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote: > >> I agree with all that. But the subject line is specifically about > >> moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog, > >> then that is noted. But if we were to move it, we can think about a > >> good place to move it to. > > > > I think it's probably worth moving pg_xlog, because the benefit also > > includes preventing a few users from shooting themselves somewhere > > vital. That's imo much less the case for some of the other moves. But I > > still don't think think a largescale reorganization is a good idea, > > it'll just stall and nothing will happen. > > OK, so let's focus only on the renaming mentioned in $subject. So far > as I can see on this thread, here are the opinions of people who > clearly gave one: > - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on > David's input), Magnus > - Rename them, hard break not OK: Fujii-san (perhaps do nothing?) > - Do nothing: Simon (add a README), Tom, Peter E I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
> to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.
I think that would make sense if we were to relocate *everything* under
PGDATA into some FHS-like subdirectory structure. But I'm against moving
the config files for previously-stated reasons, and I doubt it makes sense
to adopt an FHS-like structure only in part.
regards, tom lane
On 8/26/16 4:08 PM, Andres Freund wrote: > Splitting of ephemeral data seems to have a benefit, the rest seems more > like rather noisy busywork to me. People accidentally blowing away pg_clog or pg_xlog is a pretty common occurrence, and I don't think there's all that many tools that reference them. I think it's well worth renaming them. I actually like the idea of re-organizing everything too. It would be easy to include a tool that puts symlinks in place to all the original locations if you need that. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461
<p dir="ltr"><p dir="ltr">On 29 Aug 2016 12:10 PM, "Jim Nasby" <<a href="mailto:Jim.Nasby@bluetreble.com">Jim.Nasby@bluetreble.com</a>>wrote:<br /> ><br /> > On 8/26/16 4:08 PM, AndresFreund wrote:<br /> >><br /> >> Splitting of ephemeral data seems to have a benefit, the rest seems more<br/> >> like rather noisy busywork to me.<br /> ><br /> ><br /> > People accidentally blowing away pg_clogor pg_xlog is a pretty common occurrence, and I don't think there's all that many tools that reference them. I thinkit's well worth renaming them.<p dir="ltr">Yeah. I've seen it in BIG production users who really should have known better.<p dir="ltr">People won't see a README in amongst 5000 xlog segments while freaking out about the sever being down.<pdir="ltr">I don't care if it comes as part of some greater reorg or not but I'll be really annoyed if scope creeplands up killing the original proposal to just rename these dirs. I think that a simple rename should be done first.Then if some greater reorg is to be done it can be done shortly after. The only people that'll upset are folks trackingearly 10.0 dev and they'll be aware it's coming.
On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer <craig.ringer@2ndquadrant.com> wrote: > I don't care if it comes as part of some greater reorg or not but I'll be > really annoyed if scope creep lands up killing the original proposal to just > rename these dirs. I think that a simple rename should be done first. Then > if some greater reorg is to be done it can be done shortly after. The only > people that'll upset are folks tracking early 10.0 dev and they'll be aware > it's coming. Okay, so let's do it. Attached are two patches: - 0001 renames pg_clog to pg_trans. I have let clog.c with its current name, as well as its structures. That's the mechanical patch, the ony interesting part being in pg_upgrade. - 0002 renames pg_xlog to pg_wal. -- Michael
Attachment
On Mon, Aug 29, 2016 at 2:45 AM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:
-- On 8/26/16 4:08 PM, Andres Freund wrote:Splitting of ephemeral data seems to have a benefit, the rest seems more
like rather noisy busywork to me.
People accidentally blowing away pg_clog or pg_xlog is a pretty common occurrence, and I don't think there's all that many tools that reference them. I think it's well worth renaming them.
Pretty sure every single backup tool or script out there is referencing pg_xlog. If it's not, then it's broken...
pg_clog is a different story of course.
On 29 August 2016 at 14:30, Michael Paquier <michael.paquier@gmail.com> wrote:
> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
> <craig.ringer@2ndquadrant.com> wrote:
>> I don't care if it comes as part of some greater reorg or not but I'll be
>> really annoyed if scope creep lands up killing the original proposal to just
>> rename these dirs. I think that a simple rename should be done first. Then
>> if some greater reorg is to be done it can be done shortly after. The only
>> people that'll upset are folks tracking early 10.0 dev and they'll be aware
>> it's coming.
>
> Okay, so let's do it. Attached are two patches:
> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current
> name, as well as its structures. That's the mechanical patch, the ony
> interesting part being in pg_upgrade.
> - 0002 renames pg_xlog to pg_wal.
Is there any expectation that a 10.0 pg_basebackup should work on a
9.x server, or fail to work gracefully? There doesn't look to be any
version specific handling of the rename there.
Otherwise looks good.
Doesn't upset 'make check' or the TAP recovery suite.
Works:
src/test/regress/tmp_check/data $ ls
base pg_commit_ts pg_hba.conf pg_logical pg_notify
pg_serial pg_stat pg_subtrans pg_trans PG_VERSION
postgresql.auto.conf postmaster.opts
global pg_dynshmem pg_ident.conf pg_multixact pg_replslot
pg_snapshots pg_stat_tmp pg_tblspc pg_twophase pg_wal
postgresql.conf
It leaves pg_xlogfilename etc with original names as it IMO should.
There's no need to break more than we have to and it's still the xlog.
The documentation on backup/restore might benefit from a note saying
that pg_wal was named pg_xlog prior to 10.0 so tools intended to work
on older versions should check PG_VERSION. Though in practice most
people who write new tools will target 10.0+ and people maintaining
older tools will know, so it's not a big deal.
I don't know if the renaming in XLogFileRead of XLOG_FROM_PG_XLOG =>
XLOG_FROM_PG_WAL is really necessary, but tend to think it's good
since that define explicitly refers to the directory name, not
transaction logs in general.
The patch does not update the translations. I wonder if it's worth
doing so to save translators the hassle by sed'ing the following
lines:
- errhint("The database server
will regularly poll the pg_xlog subdirectory to check for files placed
there.")));
+ errhint("The database server
will regularly poll the pg_wal subdirectory to check for files placed
there.")));
- (errmsg("could not open directory
\"%s\": %m", "pg_xlog")));
+ (errmsg("could not open directory
\"%s\": %m", "pg_wal")));
with
sed -i 's/\<pg_xlog\>/pg_wal/g' src/backend/po/*.po
?
src/backend/access/transam/README should probably have a note about the rename.
Looks like changes in pg_upgrade for clog are a bit more than the
mechanical changes elsewhere, but seem sensible to me. I haven't yet
done a test pg_upgrade run.
--
-- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Remote DBA, Training
&Services
On 27/08/16 18:50, Tom Lane wrote: > Michael Paquier <michael.paquier@gmail.com> writes: >> OK, so let's focus only on the renaming mentioned in $subject. So far >> as I can see on this thread, here are the opinions of people who >> clearly gave one: >> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on >> David's input), Magnus >> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?) >> - Do nothing: Simon (add a README), Tom, Peter E > > I'm against moving/renaming the > configuration files, because I think that will break a lot of users' > scripts and habits without really buying much. I don't agree with this part, mainly because there is significant number of installations (everything that uses debian/ubuntu scripts) that already don't have configuration files inside the data directory. On the other matters: +1 for renaming pg_xlog to pg_wal and pg_clog to pg_xact/trans (don't care really which one) And +1 for renaming pg_logical to something more reasonable. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Mon, Aug 29, 2016 at 5:28 PM, Craig Ringer <craig.ringer@2ndquadrant.com> wrote: > On 29 August 2016 at 14:30, Michael Paquier <michael.paquier@gmail.com> wrote: >> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer >> <craig.ringer@2ndquadrant.com> wrote: >>> I don't care if it comes as part of some greater reorg or not but I'll be >>> really annoyed if scope creep lands up killing the original proposal to just >>> rename these dirs. I think that a simple rename should be done first. Then >>> if some greater reorg is to be done it can be done shortly after. The only >>> people that'll upset are folks tracking early 10.0 dev and they'll be aware >>> it's coming. >> >> Okay, so let's do it. Attached are two patches: >> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current >> name, as well as its structures. That's the mechanical patch, the ony >> interesting part being in pg_upgrade. >> - 0002 renames pg_xlog to pg_wal. > > Is there any expectation that a 10.0 pg_basebackup should work on a > 9.x server, or fail to work gracefully? There doesn't look to be any > version specific handling of the rename there. Oops. Per the docs: pg_basebackup works with servers of the same or an older major version, down to 9.1. However, WAL streaming mode (-X stream) only works with server version 9.3 and later, and tar format mode (--format=tar) of the current version only works with server version 9.5 or later. So we need to do a bit better than what the patch proposes, but that's actually just tweaking the things with pg_xlog/pg_wal depending on the version of the target server. > The patch does not update the translations. I wonder if it's worth > doing so to save translators the hassle by sed'ing the following > lines: > sed -i 's/\<pg_xlog\>/pg_wal/g' src/backend/po/*.po > ? Yes I was wondering about that... But concluded that normally the translation updates would just do it. It is not complicated to update if need be anyway. > src/backend/access/transam/README should probably have a note about the rename. Good idea. > Looks like changes in pg_upgrade for clog are a bit more than the > mechanical changes elsewhere, but seem sensible to me. I haven't yet > done a test pg_upgrade run. I have tested that FWIW using 10devel -> 10devel, 9.5 -> 10devel, 9.4 -> 10devel. There are versions of 10devel using pg_xlog and others pg_wal if this patch is introduced. I am aware of the fact that pg_upgrade supports upgrades for the same major version though it seems to me that we would not want to support this scenario, which is why it would be good to get this patch at the beginning of the dev cycle. -- Michael
On 29 August 2016 at 20:28, Michael Paquier <michael.paquier@gmail.com> wrote: > On Mon, Aug 29, 2016 at 5:28 PM, Craig Ringer > <craig.ringer@2ndquadrant.com> wrote: >> On 29 August 2016 at 14:30, Michael Paquier <michael.paquier@gmail.com> wrote: >>> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer >>> <craig.ringer@2ndquadrant.com> wrote: >>>> I don't care if it comes as part of some greater reorg or not but I'll be >>>> really annoyed if scope creep lands up killing the original proposal to just >>>> rename these dirs. I think that a simple rename should be done first. Then >>>> if some greater reorg is to be done it can be done shortly after. The only >>>> people that'll upset are folks tracking early 10.0 dev and they'll be aware >>>> it's coming. >>> >>> Okay, so let's do it. Attached are two patches: >>> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current >>> name, as well as its structures. That's the mechanical patch, the ony >>> interesting part being in pg_upgrade. >>> - 0002 renames pg_xlog to pg_wal. >> >> Is there any expectation that a 10.0 pg_basebackup should work on a >> 9.x server, or fail to work gracefully? There doesn't look to be any >> version specific handling of the rename there. > > Oops. Per the docs: > pg_basebackup works with servers of the same or an older major > version, down to 9.1. However, WAL streaming mode (-X stream) only > works with server version 9.3 and later, and tar format mode > (--format=tar) of the current version only works with server version > 9.5 or later. > > So we need to do a bit better than what the patch proposes, but that's > actually just tweaking the things with pg_xlog/pg_wal depending on the > version of the target server. Cool. I'll mark as waiting on author pending that. It'll be good to get this footgun put away. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Remote DBA, Training &Services
Craig Ringer wrote: > People won't see a README in amongst 5000 xlog segments while > freaking out about the sever being down. Maybe they're more likely to google "pg_xlog". BTW, renaming it will not help with respect to that, as there's a pretty good corpus on knowledge linked to that particular keyword. The first google results of "pg_xlog" are, for me: - Solving pg_xlog out of disk space problem on Postgres - PostgreSQL: Documentation: 9.1: WAL Internals - Why is my pg_xlog directory so huge? - PostgreSQL - Database Soup: Don't delete pg_xlog ... That's spot-on. For each person deleting stuff in pg_xlog and then regretting it, how many look it up in the above results and avoid making the mistake? Will they find comparable results if the directory is "wal" ? Aside from that, we might also question how much of the excuse "pg_xlog looked like it was just deletable logs" is a lie made up after the fact, because anybody wrecking a database is not against deflecting a bit of the blame to the software, that's human. The truth being that they took the gamble that postgres will somehow recover from the deletion, at the risk of possibly loosing the latest transactions. That doesn't necessarily look so bad when current transactions are failing anyway for lack of disk space, users are yelling at you, and you're frantically searching for anything to delete in the FS to come back online quickly. Personally I'm quite skeptical of the *name* of the directory playing that much of a role in this scenario. Best regards, -- Daniel Vérité PostgreSQL-powered mailer: http://www.manitou-mail.org Twitter: @DanielVerite
On 08/27/2016 11:11 AM, Tom Lane wrote: > Alvaro Herrera <alvherre@2ndquadrant.com> writes: >> I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog >> to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that. > > I think that would make sense if we were to relocate *everything* under > PGDATA into some FHS-like subdirectory structure. But I'm against moving > the config files for previously-stated reasons, and I doubt it makes sense > to adopt an FHS-like structure only in part. I think that is a very reasonable suggestion. Also as a note to the idea that we make break things for external user space; the next version being v10 is the exact time to do that. JD > > regards, tom lane > > -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone appreciates your honesty, until you are honest with them.
On 08/29/2016 12:04 AM, Magnus Hagander wrote: > On Mon, Aug 29, 2016 at 2:45 AM, Jim Nasby <Jim.Nasby@bluetreble.com > <mailto:Jim.Nasby@bluetreble.com>> wrote: > > On 8/26/16 4:08 PM, Andres Freund wrote: > > Splitting of ephemeral data seems to have a benefit, the rest > seems more > like rather noisy busywork to me. > > > People accidentally blowing away pg_clog or pg_xlog is a pretty > common occurrence, and I don't think there's all that many tools > that reference them. I think it's well worth renaming them. > > > Pretty sure every single backup tool or script out there is referencing > pg_xlog. If it's not, then it's broken... No, not really. Consider a filesytem backup using archiving and base backups. It doesn't care one lick about pg_xlog. And I guarantee you that there are tons of people running a backup like that considering the same script would work all the way back to 8.2 (.1?). Sincerely, JD -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone appreciates your honesty, until you are honest with them.
On 08/29/2016 06:42 AM, Daniel Verite wrote: > > Aside from that, we might also question how much of the excuse > "pg_xlog looked like it was just deletable logs" is a lie made up > after the fact, because anybody wrecking a database is not against > deflecting a bit of the blame to the software, that's human. > > The truth being that they took the gamble that postgres > will somehow recover from the deletion, at the risk of possibly > loosing the latest transactions. That doesn't necessarily look > so bad when current transactions are failing anyway for lack of disk > space, users are yelling at you, and you're frantically searching for > anything to delete in the FS to come back online quickly. > Personally I'm quite skeptical of the *name* of the directory > playing that much of a role in this scenario. Oh it makes perfect sense. User who isn't a postgres/database person installs X software. X software uses PostgreSQL and you have read on the intertubes about how Postgres is awesome. So you install it, get everything up and running from the README.md on Github. You walk away. A year later, after it becomes crucial to whatever it is you do the system crashes. You have no idea what is going on, you just set this up on some recycled server or VM. You log in, see that all the space and you find that you are using a ton of disk space. You look around for anything you can delete. You find a directory called pg_xlog, it says log, junior ignorant, don't want to be a sysadmin 101 says, "delete logs". Boom, crushed server. Let us not forget that by far the number of PostgreSQL users we have, have never done ANYTHING with postgres except make it so something like Drupal can connect to it. JD > > Best regards, > -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone appreciates your honesty, until you are honest with them.
"Joshua D. Drake" <jd@commandprompt.com> writes:
> Also as a note to the idea that we make break things for external user
> space; the next version being v10 is the exact time to do that.
Let's please drop this meme that "v10 is a great time to break things".
We don't want this to be any worse than any other major-version upgrade.
If we throw thirty different major incompatibilities in at once, we're
going to be hearing about how painful it was for the next decade, even if
any one of them individually would have been manageable. Or, if we make
the pain factor too high, users will simply not upgrade, and we'll be
faced with demands that we support 9.6 forever.
regards, tom lane
On 8/27/16 4:33 AM, Michael Paquier wrote: > On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres@anarazel.de> wrote: >> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote: >>> I agree with all that. But the subject line is specifically about >>> moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog, >>> then that is noted. But if we were to move it, we can think about a >>> good place to move it to. >> >> I think it's probably worth moving pg_xlog, because the benefit also >> includes preventing a few users from shooting themselves somewhere >> vital. That's imo much less the case for some of the other moves. But I >> still don't think think a largescale reorganization is a good idea, >> it'll just stall and nothing will happen. > > OK, so let's focus only on the renaming mentioned in $subject. So far > as I can see on this thread, here are the opinions of people who > clearly gave one: > - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on > David's input), Magnus > - Rename them, hard break not OK: Fujii-san (perhaps do nothing?) > - Do nothing: Simon (add a README), Tom, Peter E > > As far as I can see, there is a consensus to not rename pg_xlog to > pg_journal and avoid using a third meaning, but instead use pg_wal. I > guess that now the other renaming would be pg_clog -> pg_xact. Other > opinions? Forgot you here? I'm definitely +1 for a hard break (internal links are a pain). Ideally I'd would like to see: pg_xlog -> pg_wal pg_clog -> pg_xact pg_logical -> pg_replgcl pg_logical is a special case because it contains both ephemeral and persistent files. I'd like to see the temporary files in pg_tmp/pg_replgcl (or whatever it gets renamed to) but that means that pg_tmp must be on the same device to get atomic renames. It would be enough if pg_replgcl had a pgsql_tmp subdirectory so temp files are excluded from backups with the current logic. I'm also in favor of leaving configuration files where they are because these are the files that are most likely to be checked/manipulated in various user scripts (other than pg_xlog) of course. As Stephen mentioned I would like to see purely ephemeral data moved into its own directory. Maybe $PGDATA/pg_tmp? pg_subtrans -> pg_tmp/pg_subxact pg_stat_tamp -> pg_tmp/pg_stat pg_dynshmem -> pg_tmp/pg_dynshmem (or pg_shmem?) pg_notify -> pg_tmp/pg_notify pg_replslot -> pg_tmp/pg_repslot pg_serial -> pg_tmp/pg_serial pg_snapshots -> pg_tmp/pg_snapshot It would be helpful if we had consistent temp directory naming everywhere so it's clear what to exclude when it is not practical to put files in the root pg_tmp. As such I would rename pgsql_tmp to pg_tmp in base and pg_tblspc/*/*/. -- -David david@pgmasters.net
Hi, On 2016-08-29 11:18:38 -0400, David Steele wrote: > pg_logical -> pg_replgcl That seems considerably worse. > pg_replslot -> pg_tmp/pg_repslot That's most certainly not ephemeral. Just because something isn't generally appropriate on a standby, doesn't, by far, mean it's ephemeral. Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes:
> On 2016-08-29 11:18:38 -0400, David Steele wrote:
>> pg_replslot -> pg_tmp/pg_repslot
> That's most certainly not ephemeral. Just because something isn't
> generally appropriate on a standby, doesn't, by far, mean it's ephemeral.
Do we need to account for both of those concepts explicitly?
The original point here was to simplify figuring out what needed to be
copied in base backups. I'm not sure whether we really need a marker
for what's going to be flushed at restart.
regards, tom lane
On 8/29/16 11:35 AM, Andres Freund wrote: > On 2016-08-29 11:18:38 -0400, David Steele wrote: >> pg_logical -> pg_replgcl > > That seems considerably worse. Fair enough. I was just trying to throw something out there to get rid of the "log" in logical. >> pg_replslot -> pg_tmp/pg_repslot > > That's most certainly not ephemeral. Just because something isn't > generally appropriate on a standby, doesn't, by far, mean it's ephemeral. Yes, ephemeral was a poor choice of words. I really meant "can be excluded from backups". -- -David david@pgmasters.net
On 2016-08-29 12:07:51 -0400, David Steele wrote: > >> pg_replslot -> pg_tmp/pg_repslot > > > > That's most certainly not ephemeral. Just because something isn't > > generally appropriate on a standby, doesn't, by far, mean it's ephemeral. > > Yes, ephemeral was a poor choice of words. I really meant "can be > excluded from backups". Then it most certainly can't be called pg_tmp, that'll just invite people to rm -rf it. And then they'll be screwed.
On 08/29/2016 08:07 AM, Tom Lane wrote: > "Joshua D. Drake" <jd@commandprompt.com> writes: >> Also as a note to the idea that we make break things for external user >> space; the next version being v10 is the exact time to do that. > > Let's please drop this meme that "v10 is a great time to break things". > We don't want this to be any worse than any other major-version upgrade. The moment you break backward compatibility, it will be worse. We are talking about breaking backward compatibility. So let's just accept it as it is, there is a mentality about a major jump. A major jump (9.6->10) is *the* perfect time to make world changing, changes. > If we throw thirty different major incompatibilities in at once, we're > going to be hearing about how painful it was for the next decade, even if > any one of them individually would have been manageable. Or, if we make > the pain factor too high, users will simply not upgrade, and we'll be > faced with demands that we support 9.6 forever. Whiners always find a reason to whine. Let's be on two feet here. I am not saying we should jump to using json notation for our next release. I am simply stating that any largish (even if it is a small patch) changes to expected behavior should be done with care. We have a window because no matter how much you yell, I yell, Magnus yells, or anybody else yells; the telling story will be, "10.0 is a huge jump from 9.6". Most people *WOULD NOT CARE* if we only changed one thing. They care that we jumped 4 releases. That type of communication implies BIG CHANGES. Whether you like it or not. Whether I like it or not. Sincerely, JD -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone appreciates your honesty, until you are honest with them. Unless otherwise stated, opinions are my own.
On Sat, Aug 27, 2016 at 5:33 PM, Michael Paquier <michael.paquier@gmail.com> wrote: > On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres@anarazel.de> wrote: >> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote: >>> I agree with all that. But the subject line is specifically about >>> moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog, >>> then that is noted. But if we were to move it, we can think about a >>> good place to move it to. >> >> I think it's probably worth moving pg_xlog, because the benefit also >> includes preventing a few users from shooting themselves somewhere >> vital. That's imo much less the case for some of the other moves. But I >> still don't think think a largescale reorganization is a good idea, >> it'll just stall and nothing will happen. > > OK, so let's focus only on the renaming mentioned in $subject. So far > as I can see on this thread, here are the opinions of people who > clearly gave one: > - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on > David's input), Magnus > - Rename them, hard break not OK: Fujii-san (perhaps do nothing?) > - Do nothing: Simon (add a README), Tom, Peter E I vote for "do nothing". First of all, I can't have much hope for that renaming the directories really prevents "careless" users from wrongly deleting the important files. Regards, -- Fujii Masao
On Sat, Aug 27, 2016 at 2:03 PM, Michael Paquier <michael.paquier@gmail.com> wrote: > - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on > David's input), Magnus I'm in favor of this. But I don't like Peter's proposal to use a more complicated structure. As for the new names, how about pg_wal and pg_xact? I don't think "pg_trans" is as good; is it transactional or transient or transport or transitive or what? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Joshua D. Drake wrote: > You log in, see that all the space and you find that you are using a > ton of disk space. You look around for anything you can delete. You > find a directory called pg_xlog, it says log, junior ignorant, don't > want to be a sysadmin 101 says, "delete logs". Yes, it happens. I don't deny the problem, but I'm wondering about the wishful thinking we're possibly falling into here concerning the solution. Let's imagine that pg_xlog is named wal instead. How does that help our user with the disk space problem? Does that point to a path of resolution? I don't see it. What do we think that user's next move will be? After all, WAL means Write Ahead *Log*. On the other hand, by decommissioning pg_xlog as a name, that makes it obsolete in all presentations, tutorials, docs that refer to that directory, and there are many of them. There are years of confusion ahead with questions like "where is that pg_xlog directory that I'm supposed to monitor, or move into a different partition, etc...", for a benefit that remains to be seen. Best regards, -- Daniel Vérité PostgreSQL-powered mailer: http://www.manitou-mail.org Twitter: @DanielVerite
On 08/29/2016 10:00 AM, Daniel Verite wrote: > Let's imagine that pg_xlog is named wal instead. > How does that help our user with the disk space problem? > Does that point to a path of resolution? I don't see it. > What do we think that user's next move will be? > After all, WAL means Write Ahead *Log*. If, they look it up (which they would likely have to) they will see it isn't removable. :D That is the point I am making. If it is called xlog the brain says "logs". If it says wal the brain says, "What is wal?" At that point they have to look it up and then if they still delete it; well that is what emergency rates are for :P Sincerely, JD -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone appreciates your honesty, until you are honest with them. Unless otherwise stated, opinions are my own.
On Fri, Aug 26, 2016 at 05:14:36PM +0200, Magnus Hagander wrote: > On Aug 26, 2016 5:13 PM, "Joshua D. Drake" <jd@commandprompt.com> wrote: > > > > On 08/25/2016 07:39 PM, Michael Paquier wrote: > >> > >> Hi all, > >> > >> I am relaunching $subject as 10 development will begin soon. As far as > >> I know, there is agreement that we can do something here. Among the > >> different proposals I have found: > >> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit > >> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal > > > > > > I think the use of the pg_ prefix is redundant. Just a directory called: wal > will do (for example). > > > > In reference to pg_xlog specifically, it should be wal. It is the Write Ahead > Log, not the Journal (although I recognize they can be synonymous). All the > documentation talks about Write Ahead Log. > > > > Yes, let's avoid inventing a *third* name for it, please. It's already bad > enough that we have both wal and xlog. As far as removing 'pg_', consider that many users place pg_xlog on a different device, so using "wal" would not be clear it was related to Postgres. Perhaps we can have initdb --xlogdir use pg_wal, and maybe document this suggestion. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
On 8/27/16 1:11 PM, Tom Lane wrote: > Alvaro Herrera <alvherre@2ndquadrant.com> writes: >> I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog >> to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that. > > I think that would make sense if we were to relocate *everything* under > PGDATA into some FHS-like subdirectory structure. But I'm against moving > the config files for previously-stated reasons, and I doubt it makes sense > to adopt an FHS-like structure only in part. What if we left symlinks for the config files? Or perhaps even better, provide a tool that will create them for people that actually need them. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461
On 2016-08-29 19:27:29 -0500, Jim Nasby wrote: > On 8/27/16 1:11 PM, Tom Lane wrote: > > Alvaro Herrera <alvherre@2ndquadrant.com> writes: > > > I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog > > > to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that. > > > > I think that would make sense if we were to relocate *everything* under > > PGDATA into some FHS-like subdirectory structure. But I'm against moving > > the config files for previously-stated reasons, and I doubt it makes sense > > to adopt an FHS-like structure only in part. > > What if we left symlinks for the config files? Or perhaps even better, > provide a tool that will create them for people that actually need > them. See the thread around http://archives.postgresql.org/message-id/20160826104446.n3cif4m7modslkrs%40msg.df7cb.de
On 8/29/16 12:54 PM, Robert Haas wrote: > As for the new names, how about pg_wal and > pg_xact? I don't think "pg_trans" is as good; is it transactional or > transient or transport or transitive or what? pg_transaction_status? (or pg_xact_status if you want to keep it short) -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 8/28/16 8:45 PM, Jim Nasby wrote: > People accidentally blowing away pg_clog or pg_xlog is a pretty common > occurrence, and I don't think there's all that many tools that reference > them. I think it's well worth renaming them. I think a related problem is that the default log directory is called "pg_log", which is very similar to those other two. There is no quick way to tell their meaning apart. I would consider changing the default from "pg_log" to "log". Then we'd also be at the point where we can say, everything starting with "pg_" is internal, don't touch it. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer <craig.ringer@2ndquadrant.com> wrote: > Cool. I'll mark as waiting on author pending that. > > It'll be good to get this footgun put away. OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch stack as that's the one making no discussion it seems: a lot of people like pg_wal. I have added as well handling for the renaming in pg_basebackup by using PQserverVersion. One thing to note is that a connection needs to be made to the target server *before* creating the soft link of pg_xlog/pg_wal because we need to know the version of the target server. pg_upgrade is handled as well. And that's all in 0001. Patch 0002 does pg_clog -> pg_trans, "trans" standing for "transaction". Switching to pg_trans_status or pg_xact_status is not that complicated to change anyway... -- Michael
Attachment
On 8/29/16 7:34 PM, Andres Freund wrote: >> What if we left symlinks for the config files? Or perhaps even better, >> > provide a tool that will create them for people that actually need >> > them. > See the thread around > http://archives.postgresql.org/message-id/20160826104446.n3cif4m7modslkrs%40msg.df7cb.de Right, but I was referring only to the config files. AFAIK it should be safe for those to be symlinks. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461
On Tue, Aug 30, 2016 at 11:04 AM, Michael Paquier <michael.paquier@gmail.com> wrote: > On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer > <craig.ringer@2ndquadrant.com> wrote: >> Cool. I'll mark as waiting on author pending that. >> >> It'll be good to get this footgun put away. > > OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch > stack as that's the one making no discussion it seems: a lot of people > like pg_wal. I have added as well handling for the renaming in > pg_basebackup by using PQserverVersion. One thing to note is that a > connection needs to be made to the target server *before* creating the > soft link of pg_xlog/pg_wal because we need to know the version of the > target server. pg_upgrade is handled as well. And that's all in 0001. > > Patch 0002 does pg_clog -> pg_trans, "trans" standing for > "transaction". Switching to pg_trans_status or pg_xact_status is not > that complicated to change anyway... Any input to offer for those patches? If there is nothing happening, I guess that the best move is just to move it to next CF. At least I can see that the flow would be to get those renamings done. -- Michael
On Thu, Sep 29, 2016 at 9:17 PM, Michael Paquier <michael.paquier@gmail.com> wrote: > On Tue, Aug 30, 2016 at 11:04 AM, Michael Paquier > <michael.paquier@gmail.com> wrote: >> On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer >> <craig.ringer@2ndquadrant.com> wrote: >>> Cool. I'll mark as waiting on author pending that. >>> >>> It'll be good to get this footgun put away. >> >> OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch >> stack as that's the one making no discussion it seems: a lot of people >> like pg_wal. I have added as well handling for the renaming in >> pg_basebackup by using PQserverVersion. One thing to note is that a >> connection needs to be made to the target server *before* creating the >> soft link of pg_xlog/pg_wal because we need to know the version of the >> target server. pg_upgrade is handled as well. And that's all in 0001. >> >> Patch 0002 does pg_clog -> pg_trans, "trans" standing for >> "transaction". Switching to pg_trans_status or pg_xact_status is not >> that complicated to change anyway... > > Any input to offer for those patches? If there is nothing happening, I > guess that the best move is just to move it to next CF. At least I can > see that the flow would be to get those renamings done. +1 for pg_xlog -> pg_wal. Of the existing suggestions, would like to add my vote for the following renames, matching pg_multixact: pg_clog -> pg_xact pg_subtrans -> pg_subxact If longer names are on the table, I would consider expanding all three of those: pg_clog -> pg_transaction pg_subtrans -> pg_subtransaction pg_multixact -> pg_multitransaction They sound eminently non-deletable. -- Thomas Munro http://www.enterprisedb.com
On Fri, Sep 30, 2016 at 1:32 PM, Thomas Munro <thomas.munro@enterprisedb.com> wrote: > +1 for pg_xlog -> pg_wal. This is clearly the winner here and... > Of the existing suggestions, would like to add my vote for the > following renames, matching pg_multixact: > > pg_clog -> pg_xact > pg_subtrans -> pg_subxact > > If longer names are on the table, I would consider expanding all three of those: > > pg_clog -> pg_transaction This one sounds like the consensus by what I am reading upthread. > pg_subtrans -> pg_subtransaction > pg_multixact -> pg_multitransaction > > They sound eminently non-deletable. Well the original complain was that users tend to remove the entries that include *log. I am not sure if it is worth worrying about doing that much. Anyway, those patches have rotten because of the commits of yesterday for initdb and basebackup.c (stuff I wrote and/or reviewed actually). So attached is a rebased set. 0001 does the work for pg_wal, 0002 for pg_transaction. As there have been no reviews at code level, I am moving that to the next CF. -- Michael
Attachment
On Fri, Sep 30, 2016 at 1:45 AM, Michael Paquier <michael.paquier@gmail.com> wrote: > As there have been no reviews at code level, I am moving that to the next CF. Code review of 0001: I think the tests for PQserverVersion(conn) / 100 >= 1000 are strange. I submit that either PQserverVersion(conn) >= 100000 or PQserverVersion(conn) / 10000 >= 10 is an easier-to-understand test. I vote for the first style. So in pg_basebackup.c I'd do: #define MINIMUM_VERSION_FOR_PG_WAL 100000 ... int version = PQserverVersion(conn); ... if (version >= MINIMUM_VERSION_FOR_PG_WAL) /* do whatever */ Also, for this sort of thing: + if (!conn) + /* Error message already written in GetConnection() */ + exit(1); ...because of the comment, I'd add braces around this. Tom changed the error-reporting framework for pg_upgrade in f002ed2b8e45286fe73a36e119cba2016ea0de19, so you'll need to do some rebasing over that. I haven't checked what the new conventions are, but obviously you'll want to try to be consistent with them. Other than those minor nitpicks, I think this looks OK. I'm not sure we have consensus on 0002, but 0001 seems to be popular with far more people than not. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: Michael Paquier 2016-09-30 <CAB7nPqR=SZNo_=B1ukwCiNn7aWDcw_dV0z-LG4ys9WF1N4a=uQ@mail.gmail.com>
"pg_trans" is used in two places:
> -pg_clog records the commit status for each transaction that has been assigned
> +pg_trans records the commit status for each transaction that has been assigned
> - /* copy old commit logs to new data dir */
> - copy_subdir_files("pg_clog");
> + /*
> + * Copy old commit logs to new data dir. pg_clog has been renamed to
> + * pg_trans in post-10 clusters.
(Fwiw, I'd prefer something shorter than the imho clumsy
pg_transaction. "pg_xact" sounded very fine for me, it also combines
nicely with pg_subxact and the existing pg_multixact.)
Christoph
Robert Haas <robertmhaas@gmail.com> writes:
> I think the tests for PQserverVersion(conn) / 100 >= 1000 are strange.
> I submit that either PQserverVersion(conn) >= 100000 or
> PQserverVersion(conn) / 10000 >= 10 is an easier-to-understand test.
> I vote for the first style.
+1, that's the way most existing tests of this sort are written.
(Right at the moment, there's kind of a lot of zeroes there, but
once we get to version 11 it'll be less bad.)
regards, tom lane
On Mon, Oct 3, 2016 at 10:00 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Sep 30, 2016 at 1:45 AM, Michael Paquier
> <michael.paquier@gmail.com> wrote:
>> As there have been no reviews at code level, I am moving that to the next CF.
>
> Code review of 0001:
>
> I think the tests for PQserverVersion(conn) / 100 >= 1000 are strange.
> I submit that either PQserverVersion(conn) >= 100000 or
> PQserverVersion(conn) / 10000 >= 10 is an easier-to-understand test.
> I vote for the first style. So in pg_basebackup.c I'd do:
>
> #define MINIMUM_VERSION_FOR_PG_WAL 100000
> ...
> int version = PQserverVersion(conn);
> ...
> if (version >= MINIMUM_VERSION_FOR_PG_WAL)
> /* do whatever */
I have adopted the existing style of BaseBackup() in last versions,
but I don't mind as well doing the way you suggest, without changing
the stuff in BaseBackup() though, we'd still want a pg_basebackup
compiled with 10.1 to work with 10.2 or newer minor versions of 10.
> Also, for this sort of thing:
>
> + if (!conn)
> + /* Error message already written in GetConnection() */
> + exit(1);
>
> ...because of the comment, I'd add braces around this.
Fixed. That was done without brackets on HEAD, but you are right that
is not PG-like. (There are other places in src/bin/pg_basebackup doing
the same thing, but I have not modified them to not create more diff
noise).
> Tom changed the error-reporting framework for pg_upgrade in
> f002ed2b8e45286fe73a36e119cba2016ea0de19, so you'll need to do some
> rebasing over that. I haven't checked what the new conventions are,
> but obviously you'll want to try to be consistent with them.
strerror(errno) needs to be used instead of getErrorText() in the
context of this patch.
> Other than those minor nitpicks, I think this looks OK. I'm not sure
> we have consensus on 0002, but 0001 seems to be popular with far more
> people than not.
Yes, pg_wal is fine as new name in replacement of pg_xlog. Now for the
pg_clog...
Re: Michael Paquier 2016-09-30
<CAB7nPqR=SZNo_=B1ukwCiNn7aWDcw_dV0z-LG4ys9WF1N4a=uQ@mail.gmail.com>
> "pg_trans" is used in two places:
> -pg_clog records the commit status for each transaction that has been assigned
> +pg_trans records the commit status for each transaction that has been assigned
> - /* copy old commit logs to new data dir */
> - copy_subdir_files("pg_clog");
> + /*
> + * Copy old commit logs to new data dir. pg_clog has been renamed to
> + * pg_trans in post-10 clusters.
> (Fwiw, I'd prefer something shorter than the imho clumsy
> pg_transaction. "pg_xact" sounded very fine for me, it also combines
> nicely with pg_subxact and the existing pg_multixact.)
Thanks, fixed those.
So this is still open for votes. Here are the candidates and who
voiced for what:
- pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
- pg_xact: David S, Robert
- pg_trans: Tom
- pg_transaction_status: Peter E.
Attached is a new patch set. I have re-done tests with pg_upgrade with
9.5 -> HEAD and 9.6 -> HEAD, and pg_basebackup for symlink handling.
Actually in the latter case I have found a bug in 0001 introduced by
bc34223b because fsync_pgdata() is not aware of the version of the
server it is trying to fsync(). I think that the better way to address
that is to extend fsync_pgdata in file_utils.c with the server version
to fsync() like attached.
--
Michael
Attachment
On 10/4/16 1:48 AM, Michael Paquier wrote: > So this is still open for votes. Here are the candidates and who > voiced for what: > - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that. > - pg_xact: David S, Robert > - pg_trans: Tom > - pg_transaction_status: Peter E. Christoph also prefers pg_xact [1]. -- -David david@pgmasters.net [1] https://www.postgresql.org/message-id/20161003141959.qhvd6roesj4kpgww@msg.df7cb.de
On Tue, Oct 4, 2016 at 1:48 AM, Michael Paquier <michael.paquier@gmail.com> wrote: > Yes, pg_wal is fine as new name in replacement of pg_xlog. Now for the > pg_clog... Of course the irony here is that "WAL" stands for "Write Ahead Log". So we're renaming a directly that has "log" in the name (pg_xlog) to a directory that has an "l" in the name which stands for "log" (pg_wal). Whee! -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 10/4/16 1:48 AM, Michael Paquier wrote: > So this is still open for votes. Here are the candidates and who > voiced for what: > - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that. > - pg_xact: David S, Robert > - pg_trans: Tom > - pg_transaction_status: Peter E. I think this would all be a lot easier if we just moved all the pg_* directories one directory down. We did this with "global" many years ago and it worked out well. We didn't actually have to change the file names, the top-level directory became cleaner, and no one was messing around with the files anymore. Adjusting external tools also becomes easier. There is so much lore out there about clog and xlog; it would be annoying to break with that. I can see a reason for not doing that with pg_xlog, because that is a bit of an external interface, with initdb -X and such, and pg_wal is a pretty good alternative name. But no one deals with these other directories directly, so just move them out of the way. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 10/4/16 1:48 AM, Michael Paquier wrote:
>> So this is still open for votes. Here are the candidates and who
>> voiced for what:
>> - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
>> - pg_xact: David S, Robert
>> - pg_trans: Tom
>> - pg_transaction_status: Peter E.
> I think this would all be a lot easier if we just moved all the pg_*
> directories one directory down.
The main problem we're trying to fix here is people thinking that
something with "log" in the name contains discardable data. Just
relocating the directory without renaming it won't improve that.
The relocation aspect was meant to address another problem, which
is making it easier to distinguish which parts of the tree should
be copied by tools like pg_basebackup. But that's not what Michael
is asking for votes on.
regards, tom lane
* David Steele (david@pgmasters.net) wrote: > On 10/4/16 1:48 AM, Michael Paquier wrote: > > > So this is still open for votes. Here are the candidates and who > > voiced for what: > > - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that. > > - pg_xact: David S, Robert > > - pg_trans: Tom > > - pg_transaction_status: Peter E. > > Christoph also prefers pg_xact [1]. Since we're asking for votes, I also prefer pg_xact over the other options listed here. Thanks! Stephen
On Thu, Oct 13, 2016 at 12:31 AM, Stephen Frost <sfrost@snowman.net> wrote: > * David Steele (david@pgmasters.net) wrote: >> On 10/4/16 1:48 AM, Michael Paquier wrote: >> >> > So this is still open for votes. Here are the candidates and who >> > voiced for what: >> > - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that. >> > - pg_xact: David S, Robert >> > - pg_trans: Tom >> > - pg_transaction_status: Peter E. >> >> Christoph also prefers pg_xact [1]. > > Since we're asking for votes, I also prefer pg_xact over the other > options listed here. OK. I can live with that as well. Attached are three patches. The pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the pg_clog -> pg_xact move. Only one survivor to be chosen among the last two ones. -- Michael
Attachment
On 10/12/16 11:22 AM, Tom Lane wrote: > The main problem we're trying to fix here is people thinking that > something with "log" in the name contains discardable data. Just > relocating the directory without renaming it won't improve that. I think it would help if we moved it to something like "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it out of sight. We have a tool called pg_xlogdump in the standard installation. initdb has an option --xlogdir, pg_basebackup has --xlog and others. Renaming the xlog directory would make this all a bit confusing, unless we're prepared to rename the programs and options as well. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
* Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote: > On 10/12/16 11:22 AM, Tom Lane wrote: > > The main problem we're trying to fix here is people thinking that > > something with "log" in the name contains discardable data. Just > > relocating the directory without renaming it won't improve that. > > I think it would help if we moved it to something like > "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it > out of sight. I disagree that this will materially help with the issue. > We have a tool called pg_xlogdump in the standard installation. initdb > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming > the xlog directory would make this all a bit confusing, unless we're > prepared to rename the programs and options as well. pg_xlogdump is not a user-facing tool, frankly, so I don't believe we should be terribly concerned about either leaving it named as-is or renaming it. I agree that we should consider adding alternative names to the options for initdb and pg_basebackup. Thanks! Stephen
Re: Stephen Frost 2016-10-14 <20161014113523.GZ13284@tamriel.snowman.net> > > I think it would help if we moved it to something like > > "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it > > out of sight. > > I disagree that this will materially help with the issue. And internal/base and internal/global and internal/pg_... because these shouldn't be touched by the users either. I don't think this would lead anywhere. Christoph
* Christoph Berg (myon@debian.org) wrote: > Re: Stephen Frost 2016-10-14 <20161014113523.GZ13284@tamriel.snowman.net> > > > I think it would help if we moved it to something like > > > "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it > > > out of sight. > > > > I disagree that this will materially help with the issue. > > And internal/base and internal/global and internal/pg_... because > these shouldn't be touched by the users either. > > I don't think this would lead anywhere. It'd probably be easier to move the things that are *not* PG internal (eg: config files, et al) *out* of the data directory and into somewhere sensible, like /etc ... Oh, wait ... Thanks! Stephen
On 10/14/16 9:06 AM, Stephen Frost wrote: >> > And internal/base and internal/global and internal/pg_... because >> > these shouldn't be touched by the users either. >> > >> > I don't think this would lead anywhere. > It'd probably be easier to move the things that are *not* PG internal > (eg: config files, et al) *out* of the data directory and into somewhere > sensible, like /etc ... I do think it would be an improvement to segregate things users are expected to touch (*.conf and pg_log are what come to mind) from everything else, which could easily be done by moving everything else to an internal/ directory. I agree that's not much of an improvement for pg_[cx]log, but we could create internal/ as well as rename some things. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532) mobile: 512-569-9461
* Jim Nasby (Jim.Nasby@BlueTreble.com) wrote: > On 10/14/16 9:06 AM, Stephen Frost wrote: > >>> And internal/base and internal/global and internal/pg_... because > >>> these shouldn't be touched by the users either. > >>> > >>> I don't think this would lead anywhere. > >It'd probably be easier to move the things that are *not* PG internal > >(eg: config files, et al) *out* of the data directory and into somewhere > >sensible, like /etc ... > > I do think it would be an improvement to segregate things users are > expected to touch (*.conf and pg_log are what come to mind) from > everything else, which could easily be done by moving everything > else to an internal/ directory. I agree that's not much of an > improvement for pg_[cx]log, but we could create internal/ as well as > rename some things. Apologis, I should have made it clear what I was getting at. The Debian/Ubuntu packaging already does segregate out the *.conf and pg_log into other locations based on the FHS. On a Debian-based system, there really shouldn't be anything that the user is expected to touch under /var/lib (where the PG data directory lives). Thanks! Stephen
Jim Nasby <Jim.Nasby@BlueTreble.com> writes:
> On 10/14/16 9:06 AM, Stephen Frost wrote:
>> It'd probably be easier to move the things that are *not* PG internal
>> (eg: config files, et al) *out* of the data directory and into somewhere
>> sensible, like /etc ...
> I do think it would be an improvement to segregate things users are
> expected to touch (*.conf and pg_log are what come to mind) from
> everything else, which could easily be done by moving everything else to
> an internal/ directory. I agree that's not much of an improvement for
> pg_[cx]log, but we could create internal/ as well as rename some things.
I can't get excited about that at all. It will just get in the way of
the actually useful end goal we had for directory restructuring, which
was to separate data to be copied by pg_basebackup from data not to be
copied by it. And in reality, people who don't understand that the
contents of PGDATA should be treated as "hands off unless documented
otherwise" are not going to be any less likely to shoot themselves in
the foot just because there's a directory named "internal" or
"here_be_dragons" or "keep_out_this_means_you" in the way.
I'd also suggest that an extra level of directory search to get at
database data files is not free.
regards, tom lane
On Fri, Oct 14, 2016 at 9:56 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Jim Nasby <Jim.Nasby@BlueTreble.com> writes:
> On 10/14/16 9:06 AM, Stephen Frost wrote:
>> It'd probably be easier to move the things that are *not* PG internal
>> (eg: config files, et al) *out* of the data directory and into somewhere
>> sensible, like /etc ...
> I do think it would be an improvement to segregate things users are
> expected to touch (*.conf and pg_log are what come to mind) from
> everything else, which could easily be done by moving everything else to
> an internal/ directory. I agree that's not much of an improvement for
> pg_[cx]log, but we could create internal/ as well as rename some things.
I can't get excited about that at all. It will just get in the way of
the actually useful end goal we had for directory restructuring, which
was to separate data to be copied by pg_basebackup from data not to be
copied by it. And in reality, people who don't understand that the
contents of PGDATA should be treated as "hands off unless documented
otherwise" are not going to be any less likely to shoot themselves in
the foot just because there's a directory named "internal" or
"here_be_dragons" or "keep_out_this_means_you" in the way.
I don't think moving files that are already supposed to be internal into a directory called internal is going to help much, an din particular not for xlog. If we don't rename it, the problem will remain.
And for the "don't touch" part, if anything we should move the config files into a subdirectory of PGDATA and not the other way around.
--
On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost <sfrost@snowman.net> wrote:
* Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> On 10/12/16 11:22 AM, Tom Lane wrote:
> > The main problem we're trying to fix here is people thinking that
> > something with "log" in the name contains discardable data. Just
> > relocating the directory without renaming it won't improve that.
>
> I think it would help if we moved it to something like
> "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it
> out of sight.
I disagree that this will materially help with the issue.
> We have a tool called pg_xlogdump in the standard installation. initdb
> has an option --xlogdir, pg_basebackup has --xlog and others. Renaming
> the xlog directory would make this all a bit confusing, unless we're
> prepared to rename the programs and options as well.
pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
should be terribly concerned about either leaving it named as-is or
renaming it. I agree that we should consider adding alternative names
to the options for initdb and pg_basebackup.
Ugh. Changing the names of those options are probably going to break a *lot* of things, a lot more than changing the names of the directories. Is it really worth doing that? Especially if we are even more clear that people should not be touching those directories in the first place.
Those are just the tip of the iceberg. We do use the term xlog in a lot of places (almost as many as wal, but that's a different problem)
14.10.2016, 07:38, Peter Eisentraut kirjoitti: > On 10/12/16 11:22 AM, Tom Lane wrote: >> The main problem we're trying to fix here is people thinking that >> something with "log" in the name contains discardable data. Just >> relocating the directory without renaming it won't improve that. > > I think it would help if we moved it to something like > "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it > out of sight. > > We have a tool called pg_xlogdump in the standard installation. initdb > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming > the xlog directory would make this all a bit confusing, unless we're > prepared to rename the programs and options as well. pg_receivexlog should probably be renamed, seeing how we have pg_recvlogical perhaps pg_recvwal would work? The --xlog, -x, --xlog-method and -X flags for pg_basebackup are a bit confusing as it is. Perhaps they can be kept around as deprecated aliases for a new --wal stream/fetch switch: I don't think we need new --wal and --wal-method switches. pg_resetxlog should probably be called something different than just plain pg_resetwal to make it obvious that running it will cause data loss. pg_xlogdump is a developer tool, users shouldn't care; it's hard enough to use as it is as it doesn't do anything useful when you try to point it to a recycled wal file. All in all, I think this is a good opportunity to clarify what the tools are actually supposed to do and who should be running them. As an author of a backup tool utilizing some of these tools & options I don't think renaming commands and/or arguments is a big deal -- we have to deal with a bunch of changes for every new major version anyway. / Oskari
Re: Stephen Frost 2016-10-14 <20161014113523.GZ13284@tamriel.snowman.net> > > We have a tool called pg_xlogdump in the standard installation. initdb > > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming > > the xlog directory would make this all a bit confusing, unless we're > > prepared to rename the programs and options as well. > > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we > should be terribly concerned about either leaving it named as-is or > renaming it. I agree that we should consider adding alternative names > to the options for initdb and pg_basebackup. There's actually another instance of "rename so people shoot their feet less often" here: pg_resetxlog, which is a user-facing tool. Folks on #postgresql have repeatedly been joking that it should rather be named pg_eatmydata, given the number of "howtos" on the web that make use of it. Maybe this is the chance to find a less innocent-sounding name. (Or add a mandatory --rewind-my-data switch.) Christoph
* Magnus Hagander (magnus@hagander.net) wrote: > On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost <sfrost@snowman.net> wrote: > > * Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote: > > > On 10/12/16 11:22 AM, Tom Lane wrote: > > > > The main problem we're trying to fix here is people thinking that > > > > something with "log" in the name contains discardable data. Just > > > > relocating the directory without renaming it won't improve that. > > > > > > I think it would help if we moved it to something like > > > "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it > > > out of sight. > > > > I disagree that this will materially help with the issue. > > > > > We have a tool called pg_xlogdump in the standard installation. initdb > > > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming > > > the xlog directory would make this all a bit confusing, unless we're > > > prepared to rename the programs and options as well. > > > > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we > > should be terribly concerned about either leaving it named as-is or > > renaming it. I agree that we should consider adding alternative names > > to the options for initdb and pg_basebackup. > > Ugh. Changing the names of those options are probably going to break a > *lot* of things, a lot more than changing the names of the directories. Is > it really worth doing that? Especially if we are even more clear that > people should not be touching those directories in the first place. I would point out that I said "adding alternative names", not "change/remove the existing options." That said, while I like the general idea of "make everything referring to transaction log/write-ahead xlog/xlog/wal/whatever refer in the same way", I'm not sure that it's a really pressing issue or that changing the name of the directory should drive those other changes. > Those are just the tip of the iceberg. We do use the term xlog in a lot of > places (almost as many as wal, but that's a different problem) True, and yes, we should consider the places that already talk about "wal", but, overall, I believe we can take this one step at a time and don't have to do everything all at once simply because we're changing the directory name. We're not changing what it *is*, after all. Thanks! Stephen
* Christoph Berg (myon@debian.org) wrote: > Re: Stephen Frost 2016-10-14 <20161014113523.GZ13284@tamriel.snowman.net> > > > We have a tool called pg_xlogdump in the standard installation. initdb > > > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming > > > the xlog directory would make this all a bit confusing, unless we're > > > prepared to rename the programs and options as well. > > > > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we > > should be terribly concerned about either leaving it named as-is or > > renaming it. I agree that we should consider adding alternative names > > to the options for initdb and pg_basebackup. > > There's actually another instance of "rename so people shoot their > feet less often" here: pg_resetxlog, which is a user-facing tool. That is *not* a user-facing tool. > Folks on #postgresql have repeatedly been joking that it should rather > be named pg_eatmydata, given the number of "howtos" on the web that > make use of it. Maybe this is the chance to find a less > innocent-sounding name. (Or add a mandatory --rewind-my-data switch.) Not sure on the best approach here, but I agree that it's a problem that people think pg_resetxlog is some kind of recovery tool (hint: it's not). Thanks! Stephen
On Fri, Oct 14, 2016 at 11:48 AM, Stephen Frost <sfrost@snowman.net> wrote:
-- * Magnus Hagander (magnus@hagander.net) wrote:
> On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost <sfrost@snowman.net> wrote:
> > * Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> > > On 10/12/16 11:22 AM, Tom Lane wrote:
> > > > The main problem we're trying to fix here is people thinking that
> > > > something with "log" in the name contains discardable data. Just
> > > > relocating the directory without renaming it won't improve that.
> > >
> > > I think it would help if we moved it to something like
> > > "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it
> > > out of sight.
> >
> > I disagree that this will materially help with the issue.
> >
> > > We have a tool called pg_xlogdump in the standard installation. initdb
> > > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming
> > > the xlog directory would make this all a bit confusing, unless we're
> > > prepared to rename the programs and options as well.
> >
> > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > should be terribly concerned about either leaving it named as-is or
> > renaming it. I agree that we should consider adding alternative names
> > to the options for initdb and pg_basebackup.
>
> Ugh. Changing the names of those options are probably going to break a
> *lot* of things, a lot more than changing the names of the directories. Is
> it really worth doing that? Especially if we are even more clear that
> people should not be touching those directories in the first place.
I would point out that I said "adding alternative names", not
"change/remove the existing options."
Oh, I missed that. Thanks for clarifying :)
That said, while I like the general idea of "make everything referring
to transaction log/write-ahead xlog/xlog/wal/whatever refer in the same
way", I'm not sure that it's a really pressing issue or that changing
the name of the directory should drive those other changes.
Agreed.
> Those are just the tip of the iceberg. We do use the term xlog in a lot of
> places (almost as many as wal, but that's a different problem)
True, and yes, we should consider the places that already talk about
"wal", but, overall, I believe we can take this one step at a time and
don't have to do everything all at once simply because we're changing
the directory name. We're not changing what it *is*, after all.
Yeah, I think if we aim to fix everything at once, we'll get nothing done. That old saying of not letting perfection getting in the way of progress applies.
On Fri, Oct 14, 2016 at 07:19:02PM +0200, Christoph Berg wrote: > Re: Stephen Frost 2016-10-14 <20161014113523.GZ13284@tamriel.snowman.net> > > > We have a tool called pg_xlogdump in the standard installation. initdb > > > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming > > > the xlog directory would make this all a bit confusing, unless we're > > > prepared to rename the programs and options as well. > > > > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we > > should be terribly concerned about either leaving it named as-is or > > renaming it. I agree that we should consider adding alternative names > > to the options for initdb and pg_basebackup. > > There's actually another instance of "rename so people shoot their > feet less often" here: pg_resetxlog, which is a user-facing tool. > Folks on #postgresql have repeatedly been joking that it should rather > be named pg_eatmydata, given the number of "howtos" on the web that > make use of it. Maybe this is the chance to find a less > innocent-sounding name. (Or add a mandatory --rewind-my-data switch.) I wonder how many of the uses of pg_resetxlog were caused by mistakenly removing the pg_xlog directory. My point is renaming pg_xlog might reduce mistake use of pg_resetxlog. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
Re: Bruce Momjian 2016-10-19 <20161018220909.GA11661@momjian.us> > > There's actually another instance of "rename so people shoot their > > feet less often" here: pg_resetxlog, which is a user-facing tool. > > Folks on #postgresql have repeatedly been joking that it should rather > > be named pg_eatmydata, given the number of "howtos" on the web that > > make use of it. Maybe this is the chance to find a less > > innocent-sounding name. (Or add a mandatory --rewind-my-data switch.) > > I wonder how many of the uses of pg_resetxlog were caused by mistakenly > removing the pg_xlog directory. My point is renaming pg_xlog might > reduce mistake use of pg_resetxlog. I don't think there's much of a connection. There are people who clean up disk space by removing everything that sounds like log files. For those people renaming the directories makes sense so they don't have that idea. There are other people who have random database startup problems, ask google, and end up with applying some pg_resetxlog advice (that doesn't necessarily fit their problem). For those people renaming pg_resetxlog to something that sounds more like "it will break your data, use as last resort only" might make sense. (Though I don't have a good suggestion, and the cost of renaming utilities is higher than renaming some internal directory names.) The question would now be if there's people who used pg_resetxlog because they thought it freed up disk space, and if renaming either would have prevented that. I'm less sure about that. (tl;dr: rename pg_xlog yes, rename pg_resetxlog only if we have a good alternative.) Christoph
On Wed, Oct 19, 2016 at 7:05 AM, Christoph Berg <myon@debian.org> wrote: > (tl;dr: rename pg_xlog yes, rename pg_resetxlog only if we have a good > alternative.) I'm amused by the idea of a TL;DR in parentheses at the very bottom of the email, but maybe I'm just easily amused. One idea would be to rename pg_resetxlog to pg_resetwal. I think that's actually an improvement. The "x" in "xlog" is not a clear abbreviation for "write-ahead", especially when we also use "x" in other contexts to mean "transaction" - think "xid". Given our current practices, we could justifiably rename pg_clog to pg_xlog -- after all, that's where we log the status of the xacts. Ugh. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier <michael.paquier@gmail.com> wrote: > OK. I can live with that as well. Attached are three patches. The > pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the > pg_clog -> pg_xact move. Only one survivor to be chosen among the last > two ones. Committed 0001. To be honest, I don't really like either pg_transaction or pg_xact. Neither name captures the fact that what we're really tracking here is the transaction *status*. pg_xact is slightly worse because it's a poor abbreviation for transaction, but I think the argument against even pg_transaction is similar to the one Tom previously levied against pg_logical - viz. "logical what?". The transaction themselves are not stored in the directory, just the commit status. The fact that commit status is saved is the source of the "c" in "clog". -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> One idea would be to rename pg_resetxlog to pg_resetwal. I think
> that's actually an improvement.
This would fit in as part of a general plan to s/xlog/wal/g throughout
our user-visible names and documentation. Which seems like a good idea
to me; there's no need to expose two different terms for the same thing.
(I don't feel a great need to unify the terminology in the code, though.
Just in stuff users see.)
regards, tom lane
* Robert Haas (robertmhaas@gmail.com) wrote: > On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier > <michael.paquier@gmail.com> wrote: > > OK. I can live with that as well. Attached are three patches. The > > pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the > > pg_clog -> pg_xact move. Only one survivor to be chosen among the last > > two ones. > > Committed 0001. Glad to see that happen, finally. > To be honest, I don't really like either pg_transaction or pg_xact. > Neither name captures the fact that what we're really tracking here is > the transaction *status*. pg_xact is slightly worse because it's a > poor abbreviation for transaction, but I think the argument against > even pg_transaction is similar to the one Tom previously levied > against pg_logical - viz. "logical what?". The transaction themselves > are not stored in the directory, just the commit status. The fact > that commit status is saved is the source of the "c" in "clog". This really needs to move forward also. When it comes to the name, I tend to think of 'pg_xact' as saying "this is where we persist info we need to keep about transactions." Today that's just the commit status info, but I could imagine that there might, someday, be other things that go in there. "pg_multixact" is an example of something quite similar but does have more than just one "thing." Also, using "pg_xact" and then "pg_subxact" and "pg_multixact" bring them all under one consistent naming scheme. Thanks! Stephen
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > One idea would be to rename pg_resetxlog to pg_resetwal. I think
> > that's actually an improvement.
>
> This would fit in as part of a general plan to s/xlog/wal/g throughout
> our user-visible names and documentation. Which seems like a good idea
> to me; there's no need to expose two different terms for the same thing.
>
> (I don't feel a great need to unify the terminology in the code, though.
> Just in stuff users see.)
+1 on the general change of xlog -> wal.
That said, I'd also like to see a --force or similar option or mechanism
put in place to reduce the risk of users trashing their system because
they think pg_resetwal is "safe." ("It's just gonna reset things to make
the database start again, should be fine.").
pg_destroydb almost seems like a better choice, though I suppose
'pg_clearwal' would be more acceptable. Doesn't have quite the same
impact though.
Not sure on the best answer here, but it's definitely foot-gun that some
users have ended up using on themselves with depressing regularity.
Thanks!
Stephen
On 10/20/2016 09:12 AM, Stephen Frost wrote:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:
> That said, I'd also like to see a --force or similar option or mechanism
> put in place to reduce the risk of users trashing their system because
> they think pg_resetwal is "safe." ("It's just gonna reset things to make
> the database start again, should be fine.").
>
> pg_destroydb almost seems like a better choice, though I suppose
> 'pg_clearwal' would be more acceptable. Doesn't have quite the same
> impact though.
pg_dropwal
Users won't *drop* things they shouldn't on purpose (usually) but they
will reset and will clear them. Destroydb isn't anymore accurate because
it doesn't destroy it. Instead it makes it so I can log in again and see
my data.
(Yes we all know the real implications with it but from a DUH user
perspective...)
Sincerely,
JD
--
Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
On Thu, Oct 20, 2016 at 12:05 PM, Stephen Frost <sfrost@snowman.net> wrote: >> To be honest, I don't really like either pg_transaction or pg_xact. > >> Neither name captures the fact that what we're really tracking here is >> the transaction *status*. pg_xact is slightly worse because it's a >> poor abbreviation for transaction, but I think the argument against >> even pg_transaction is similar to the one Tom previously levied >> against pg_logical - viz. "logical what?". The transaction themselves >> are not stored in the directory, just the commit status. The fact >> that commit status is saved is the source of the "c" in "clog". > > This really needs to move forward also. > > When it comes to the name, I tend to think of 'pg_xact' as saying "this > is where we persist info we need to keep about transactions." Today > that's just the commit status info, but I could imagine that there > might, someday, be other things that go in there. "pg_multixact" is > an example of something quite similar but does have more than just one > "thing." Also, using "pg_xact" and then "pg_subxact" and "pg_multixact" > bring them all under one consistent naming scheme. I don't dispute the fact that you tend to think of it that way, but I think it's a real stretch to say that "pg_xact" is a clear name from the point of view of the uninitiated. Now, maybe the point is to be a little bit deliberately unclear, but "xact" for "transaction" is not a lot better than "xlog" for "write-ahead log". It's just arbitrary abbreviations we made up and you either know what they mean or you don't. We could call it "pg_xkcd" and we wouldn't be removing much in the way of clarity. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost <sfrost@snowman.net> wrote:
> That said, I'd also like to see a --force or similar option or mechanism
> put in place to reduce the risk of users trashing their system because
> they think pg_resetwal is "safe." ("It's just gonna reset things to make
> the database start again, should be fine.").
You know we already have that, right?
> pg_destroydb almost seems like a better choice, though I suppose
> 'pg_clearwal' would be more acceptable. Doesn't have quite the same
> impact though.
>
> Not sure on the best answer here, but it's definitely foot-gun that some
> users have ended up using on themselves with depressing regularity.
Just to provide some perspective from the other side of this, I
handled a severity-1 escalation a few weeks ago where a user basically
called up and explained their situation and I told them based on all
of the facts and circumstances presented that running pg_resetxlog was
going to be the best way forward and they said that was pretty much
what they were expecting to hear, and did so. I think there's far too
much anti-pg_resetxlog bias on this list. The situation where you
need to use it is not common in general, but it's pretty common in
support cases that reach me. I don't think I'd be exaggerating if I
said that 25% of the customer escalations I handle personally require
the use pg_resetxlog. Of course, that's because by the time the
ticket gets to me, it's typically the case that all of the easy, neat
solutions have already been ruled out.
My experience is that users who are faced with this situation
typically understand that they are in a bad situation and when I
explain to them --- in a clear and direct way but without the sort of
hysterical exaggeration sometimes seen on this mailing list --- what
the consequences are, they are not surprised by those consequences and
they are willing to accept them because they know that the
alternatives are worse. For example, consider a customer with a
mostly read-only 20TB database. If that user runs pg_resetxlog, they
can be back up in 5 minutes and it is likely (though of course not
certain) that corruption will be minimal. If they restore from
backup, they will be down for many more hours. After pg_resetxlog,
they can restore from backup on another system or do the
dump-and-restore which I invariably recommend while they are up. For
many customers, this is a good trade-off. The decision, because of
the risks associated with it, is usually passed up from the DBA I'm
working with to some business leader. If that business leader feels
that benefits of being up immediately rather than many hours later are
sufficient to justify the risk of a possibly-corrupted database,
running pg_resetxlog is a perfectly reasonable decision.
I have not yet had one DBA or business leader call back and say "hey,
that thing where we ran pg_resetxlog? you should have discouraged us
more, because that was a disaster". Now maybe those disasters have
happened and I am just not aware of them, but I think we should all
take a deep breath and remind ourselves that the database exists in
service to the business and not the other way around. My job -- when
doing support -- is to tell you what your options are and what
consequences they may have, good and bad -- not to tell you how to run
your company. Telling users that pg_resetxlog will "destroy your
database" is over-the-top hyperbole. It's a sharp tool with risks and
benefits and it deserves to be presented exactly that way.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote: > > When it comes to the name, I tend to think of 'pg_xact' as saying "this > > is where we persist info we need to keep about transactions." Today > > that's just the commit status info, but I could imagine that there > > might, someday, be other things that go in there. "pg_multixact" is > > an example of something quite similar but does have more than just one > > "thing." Also, using "pg_xact" and then "pg_subxact" and "pg_multixact" > > bring them all under one consistent naming scheme. > > I don't dispute the fact that you tend to think of it that way, but I > think it's a real stretch to say that "pg_xact" is a clear name from > the point of view of the uninitiated. Now, maybe the point is to be a > little bit deliberately unclear, but "xact" for "transaction" is not a > lot better than "xlog" for "write-ahead log". It's just arbitrary > abbreviations we made up and you either know what they mean or you > don't. We could call it "pg_xkcd" and we wouldn't be removing much in > the way of clarity. What is your suggestion for a name? If you have none, I suggest we use "pg_xact". -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
On Thu, Oct 20, 2016 at 1:39 PM, Bruce Momjian <bruce@momjian.us> wrote: > On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote: >> > When it comes to the name, I tend to think of 'pg_xact' as saying "this >> > is where we persist info we need to keep about transactions." Today >> > that's just the commit status info, but I could imagine that there >> > might, someday, be other things that go in there. "pg_multixact" is >> > an example of something quite similar but does have more than just one >> > "thing." Also, using "pg_xact" and then "pg_subxact" and "pg_multixact" >> > bring them all under one consistent naming scheme. >> >> I don't dispute the fact that you tend to think of it that way, but I >> think it's a real stretch to say that "pg_xact" is a clear name from >> the point of view of the uninitiated. Now, maybe the point is to be a >> little bit deliberately unclear, but "xact" for "transaction" is not a >> lot better than "xlog" for "write-ahead log". It's just arbitrary >> abbreviations we made up and you either know what they mean or you >> don't. We could call it "pg_xkcd" and we wouldn't be removing much in >> the way of clarity. > > What is your suggestion for a name? If you have none, I suggest we use > "pg_xact". I'm not sure. pg_transaction_status would be clear, but it's long. Is pg_xact actually better than pg_clog? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> Is pg_xact actually better than pg_clog?
Yes, because it doesn't contain the three letters "log".
We have the two precedents "pg_subtrans" and "pg_multixact", so
unless we want to get into renaming those too, I think "pg_trans"
and "pg_xact" are really the only options worth considering.
Personally I'd go for "pg_trans", but it's only a weak preference.
regards, tom lane
On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> Is pg_xact actually better than pg_clog? > > Yes, because it doesn't contain the three letters "log". I figured somebody was going to say that. > We have the two precedents "pg_subtrans" and "pg_multixact", so > unless we want to get into renaming those too, I think "pg_trans" > and "pg_xact" are really the only options worth considering. > > Personally I'd go for "pg_trans", but it's only a weak preference. Heaven forfend we actually use enough characters to make it self-documenting. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> We have the two precedents "pg_subtrans" and "pg_multixact", so
>> unless we want to get into renaming those too, I think "pg_trans"
>> and "pg_xact" are really the only options worth considering.
>>
>> Personally I'd go for "pg_trans", but it's only a weak preference.
> Heaven forfend we actually use enough characters to make it self-documenting.
$ ls $PGDATA
PG_VERSION pg_dynshmem/ pg_notify/ pg_stat_tmp/ postgresql.auto.conf
base/ pg_hba.conf pg_replslot/ pg_subtrans/ postgresql.conf
global/ pg_ident.conf pg_serial/ pg_tblspc/ postmaster.opts
pg_clog/ pg_logical/ pg_snapshots/ pg_twophase/ postmaster.pid
pg_commit_ts/ pg_multixact/ pg_stat/ pg_wal/
I don't see one single one of those subdirectory names that I'd call
self-documenting. Are you proposing we rename them all with carpal-
tunnel-syndrome-promoting names?
There's certainly some case to be made for renaming at least one of
"pg_subtrans" and "pg_multixact" so that these three similarly-purposed
subdirectories can all have similar names. But I think on the whole
that's (a) fixing what ain't broken, and (b) making it even more unlikely
that we'll ever get to consensus on changing anything. We've managed to
agree that we need to change the names ending in "log"; let's do that
and be happy that we've removed one foot-gun from the system.
regards, tom lane
On Thu, Oct 20, 2016 at 02:02:27PM -0400, Robert Haas wrote: > On Thu, Oct 20, 2016 at 1:39 PM, Bruce Momjian <bruce@momjian.us> wrote: > > On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote: > >> > When it comes to the name, I tend to think of 'pg_xact' as saying "this > >> > is where we persist info we need to keep about transactions." Today > >> > that's just the commit status info, but I could imagine that there > >> > might, someday, be other things that go in there. "pg_multixact" is > >> > an example of something quite similar but does have more than just one > >> > "thing." Also, using "pg_xact" and then "pg_subxact" and "pg_multixact" > >> > bring them all under one consistent naming scheme. > >> > >> I don't dispute the fact that you tend to think of it that way, but I > >> think it's a real stretch to say that "pg_xact" is a clear name from > >> the point of view of the uninitiated. Now, maybe the point is to be a > >> little bit deliberately unclear, but "xact" for "transaction" is not a > >> lot better than "xlog" for "write-ahead log". It's just arbitrary > >> abbreviations we made up and you either know what they mean or you > >> don't. We could call it "pg_xkcd" and we wouldn't be removing much in > >> the way of clarity. > > > > What is your suggestion for a name? If you have none, I suggest we use > > "pg_xact". > > I'm not sure. pg_transaction_status would be clear, but it's long. > Is pg_xact actually better than pg_clog? Uh, yeah, no "log". Wasn't that the point? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
On Thu, Oct 20, 2016 at 2:23 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>> We have the two precedents "pg_subtrans" and "pg_multixact", so >>> unless we want to get into renaming those too, I think "pg_trans" >>> and "pg_xact" are really the only options worth considering. >>> >>> Personally I'd go for "pg_trans", but it's only a weak preference. > >> Heaven forfend we actually use enough characters to make it self-documenting. > > $ ls $PGDATA > PG_VERSION pg_dynshmem/ pg_notify/ pg_stat_tmp/ postgresql.auto.conf > base/ pg_hba.conf pg_replslot/ pg_subtrans/ postgresql.conf > global/ pg_ident.conf pg_serial/ pg_tblspc/ postmaster.opts > pg_clog/ pg_logical/ pg_snapshots/ pg_twophase/ postmaster.pid > pg_commit_ts/ pg_multixact/ pg_stat/ pg_wal/ > > I don't see one single one of those subdirectory names that I'd call > self-documenting. Are you proposing we rename them all with carpal- > tunnel-syndrome-promoting names? No. Are you proposing that self-documenting names are a bad thing rather than a good thing? > There's certainly some case to be made for renaming at least one of > "pg_subtrans" and "pg_multixact" so that these three similarly-purposed > subdirectories can all have similar names. But I think on the whole > that's (a) fixing what ain't broken, and (b) making it even more unlikely > that we'll ever get to consensus on changing anything. We've managed to > agree that we need to change the names ending in "log"; let's do that > and be happy that we've removed one foot-gun from the system. I agree that there is an overwhelming consensus in favor of getting "log" out of the names, but I do not agree that the only two possible alternative names are "pg_trans" and "pg_xact", which strikes me as being akin to saying that the only two options for dinner tonight are overripe peaches and lunch meats a week past the sell-by date. If I had to pick only between those two, I suppose I'd go with "pg_xact" which is clear enough to someone familiar with PostgreSQL internals, as opposed to "pg_trans" which I don't think will be clear even to those people. But I don't like either one very much. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote: > Robert Haas <robertmhaas@gmail.com> writes: > > On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > >> We have the two precedents "pg_subtrans" and "pg_multixact", so > >> unless we want to get into renaming those too, I think "pg_trans" > >> and "pg_xact" are really the only options worth considering. > >> > >> Personally I'd go for "pg_trans", but it's only a weak preference. > > > Heaven forfend we actually use enough characters to make it self-documenting. > > $ ls $PGDATA > PG_VERSION pg_dynshmem/ pg_notify/ pg_stat_tmp/ postgresql.auto.conf > base/ pg_hba.conf pg_replslot/ pg_subtrans/ postgresql.conf > global/ pg_ident.conf pg_serial/ pg_tblspc/ postmaster.opts > pg_clog/ pg_logical/ pg_snapshots/ pg_twophase/ postmaster.pid > pg_commit_ts/ pg_multixact/ pg_stat/ pg_wal/ > > I don't see one single one of those subdirectory names that I'd call > self-documenting. That's a problem we should do something about, even if we can't do it by renaming these all in one go. At the very least, we can do this for any new names. > Are you proposing we rename them all with carpal- > tunnel-syndrome-promoting names? Are you saying that people are getting carpal tunnel syndrome from hitting the tab key, which has been standard for completion in shells for decades? I'm pretty sure that doesn't actually happen. > There's certainly some case to be made for renaming at least one of > "pg_subtrans" and "pg_multixact" so that these three similarly-purposed > subdirectories can all have similar names. But I think on the whole > that's (a) fixing what ain't broken, and (b) making it even more unlikely > that we'll ever get to consensus on changing anything. We've managed to > agree that we need to change the names ending in "log"; let's do that > and be happy that we've removed one foot-gun from the system. Removing foot guns, un-sexy as it may be from a developer's perspective, is very useful work. Best, David. -- David Fetter <david(at)fetter(dot)org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
* David Fetter (david@fetter.org) wrote: > On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote: > > Robert Haas <robertmhaas@gmail.com> writes: > > > On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > >> We have the two precedents "pg_subtrans" and "pg_multixact", so > > >> unless we want to get into renaming those too, I think "pg_trans" > > >> and "pg_xact" are really the only options worth considering. > > >> > > >> Personally I'd go for "pg_trans", but it's only a weak preference. > > > > > Heaven forfend we actually use enough characters to make it self-documenting. > > > > $ ls $PGDATA > > PG_VERSION pg_dynshmem/ pg_notify/ pg_stat_tmp/ postgresql.auto.conf > > base/ pg_hba.conf pg_replslot/ pg_subtrans/ postgresql.conf > > global/ pg_ident.conf pg_serial/ pg_tblspc/ postmaster.opts > > pg_clog/ pg_logical/ pg_snapshots/ pg_twophase/ postmaster.pid > > pg_commit_ts/ pg_multixact/ pg_stat/ pg_wal/ > > > > I don't see one single one of those subdirectory names that I'd call > > self-documenting. > > That's a problem we should do something about, even if we can't do it > by renaming these all in one go. At the very least, we can do this > for any new names. I disagree that excessivly long names for files or directories are useful and should be fully self-documenting. We should name our directories with useful hints at what they are used for that remind those who are knowledgable where things live. Secondary to that is using an approach to naming which avoid implying anything about the directories to those who are *not* knowledgable, which leads us to the current discussion regarding the removal of directories with 'log' in the name. Individuals who are not knowledgable in this area are not going to get any more benefit from a directory named 'pg_trans' or 'pg_transaction_status' than one named 'pg_clog' or 'pg_xact' and therefore this whole line of reasoning is a red herring. Thanks! Stephen
Stephen Frost <sfrost@snowman.net> writes: > * David Fetter (david@fetter.org) wrote: >> On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote: >>> I don't see one single one of those subdirectory names that I'd call >>> self-documenting. >> That's a problem we should do something about, even if we can't do it >> by renaming these all in one go. At the very least, we can do this >> for any new names. > I disagree that excessivly long names for files or directories are > useful and should be fully self-documenting. I'm mostly with Stephen on this. As the names stand, they encourage people to go look at the documentation, https://www.postgresql.org/docs/devel/static/storage-file-layout.html which will provide more information than you'd ever get out of any reasonable directory name. Having said that, I still don't like "pg_logical", but I suppose renaming it would have more downsides than upsides. regards, tom lane
On Fri, Oct 21, 2016 at 12:35 AM, Robert Haas <robertmhaas@gmail.com> wrote: > On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier > <michael.paquier@gmail.com> wrote: >> OK. I can live with that as well. Attached are three patches. The >> pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the >> pg_clog -> pg_xact move. Only one survivor to be chosen among the last >> two ones. > > Committed 0001. Thanks! -- Michael
On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > I'm mostly with Stephen on this. As the names stand, they encourage > people to go look at the documentation, > https://www.postgresql.org/docs/devel/static/storage-file-layout.html > which will provide more information than you'd ever get out of any > reasonable directory name. Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which would encourage that even more strongly. But I don't think that proposal can be taken seriously. Giving things meaningful names is a good practice in almost every case. > Having said that, I still don't like "pg_logical", but I suppose > renaming it would have more downsides than upsides. Remind me what your beef is? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Fri, Oct 21, 2016 at 10:03 AM, Robert Haas <robertmhaas@gmail.com> wrote: > On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> I'm mostly with Stephen on this. As the names stand, they encourage >> people to go look at the documentation, >> https://www.postgresql.org/docs/devel/static/storage-file-layout.html >> which will provide more information than you'd ever get out of any >> reasonable directory name. > > Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which > would encourage that even more strongly. But I don't think that > proposal can be taken seriously. Giving things meaningful names is a > good practice in almost every case. Moving on with the topic of this thread... I would think that "pg_xact" is what we are moving to rename pg_clog. On top of that, after reading the thread, here are the options and binaries that could be renamed for consistency with the renaming of pg_xlog: - pg_xlogdump -> pg_waldump - pg_resetxlog -> pg_resetwal - pg_receivexlog -> pg_receivewal - initdb --xlogdir -> --waldir - pg_basebackup --xlogmethod --xlogdir -> --walmethod --waldir That's quite a number, and each change is trivial. Previous options would be better kept for backward-compatibility. Personally, I see no urge in changing all that and I am fine with just renaming the *log directories of PGDATA for this thread. But as the point has been raised, here are all things that could be done. -- Michael
On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I'm mostly with Stephen on this. As the names stand, they encourage
> people to go look at the documentation,
> https://www.postgresql.org/docs/devel/static/storage- file-layout.html
> which will provide more information than you'd ever get out of any
> reasonable directory name.
Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which
would encourage that even more strongly. But I don't think that
proposal can be taken seriously. Giving things meaningful names is a
good practice in almost every case.
Those don't have the virtue of being at least somewhat
m
nemonic like pg_xact.
I'll toss my lot in with Steven's and Tom's on this.
I have no problem continuing keeping with historical precedent and allowing mnemonic abbreviations in our directory and file names at this point.
David J.
On 10/21/16 3:12 AM, David G. Johnston wrote: > I have no problem continuing keeping with historical precedent and > allowing mnemonic abbreviations in our directory and file names at this > point. I'm still in favor of pg_xact. A search of the 9.6 docs brings up a number of hits for "xact": pg_last_xact_replay_timestamp(), pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(), pg_last_committed_xact(), pg_prepared_xacts(), etc. There are also numerous column names that have "xact" in them. It's not just an arcane developer term when it shows up in a number of our user-facing functions/columns. -- -David david@pgmasters.net
* Robert Haas (robertmhaas@gmail.com) wrote:
> On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > That said, I'd also like to see a --force or similar option or mechanism
> > put in place to reduce the risk of users trashing their system because
> > they think pg_resetwal is "safe." ("It's just gonna reset things to make
> > the database start again, should be fine.").
>
> You know we already have that, right?
Yes, but I was meaning an option which would be required to make
pg_resetxlog actually *do* anything. In other words, I'd rather have it
report some info back to the user, if it's run without the
'--really-force' or what-have-you option, and only proceed with
clearing the WAL or rewriting pg_control when '--really-force' is used.
> > pg_destroydb almost seems like a better choice, though I suppose
> > 'pg_clearwal' would be more acceptable. Doesn't have quite the same
> > impact though.
> >
> > Not sure on the best answer here, but it's definitely foot-gun that some
> > users have ended up using on themselves with depressing regularity.
>
> Just to provide some perspective from the other side of this, I
[...]
I wasn't suggesting that we remove the capability. There are certainly
use-cases for it, but, unfortunately, I've seen a number of cases where
users simply google'd an error that they got back when trying to start
PG and found someone saying "well, I got this error, but then I ran
pg_resetxlog, and now the database starts up again."
It likely doesn't help that the top links tend to be to mailing list
archives where pg_resetxlog was brought up.
Thanks!
Stephen
On Fri, Oct 21, 2016 at 9:47 AM, Stephen Frost <sfrost@snowman.net> wrote:
> * Robert Haas (robertmhaas@gmail.com) wrote:
>> On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> > That said, I'd also like to see a --force or similar option or mechanism
>> > put in place to reduce the risk of users trashing their system because
>> > they think pg_resetwal is "safe." ("It's just gonna reset things to make
>> > the database start again, should be fine.").
>>
>> You know we already have that, right?
>
> Yes, but I was meaning an option which would be required to make
> pg_resetxlog actually *do* anything. In other words, I'd rather have it
> report some info back to the user, if it's run without the
> '--really-force' or what-have-you option, and only proceed with
> clearing the WAL or rewriting pg_control when '--really-force' is used.
I don't think that the problem is that people are accidentally typing
"pg_resetxlog $PGDATA" and pressing return. They're typing that on
purpose, and if you change the sequence of characters required to get
that effect, they'll just type the new thing instead. The problem is
that they don't understand when it's appropriate to use that command
and what the consequences are.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
* Robert Haas (robertmhaas@gmail.com) wrote: > I don't think that the problem is that people are accidentally typing > "pg_resetxlog $PGDATA" and pressing return. They're typing that on > purpose, and if you change the sequence of characters required to get > that effect, they'll just type the new thing instead. The problem is > that they don't understand when it's appropriate to use that command > and what the consequences are. I agree that they don't understand, and that's why I believe that making the command name, or a required option, a bit more ominous would make them pause and realize that maybe they want to consider other options first. This is not exactly unheard of- apt-get requires an entire phrase be to be entered when you're asking it to do something extremely dangerous (remove an essential package): --------------- root@ionith:~# apt-get remove login Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: login WARNING: The following essential packages will be removed. This should NOT be done unless you know exactly what you are doing! login 0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded. After this operation, 1,212 kB disk space will be freed. You are about to do something potentially harmful. To continue type in the phrase 'Yes, do as I say!'?] --------------- My thought would be to make pg_resetxlog do something more along the lines of what pg_control does and have it, by default, just investigate the state of things and complain about problems and then have it include an option to actually reset things with an appropriately ominous name. The goal there being, primairly, to give a way for users to get information about why PG isn't starting or what it is complaining about, with some additional information about what happens if they forcibly reset the xlog, noting that it could very likely cause corruption, etc. Thanks! Stephen
David Steele wrote: > On 10/21/16 3:12 AM, David G. Johnston wrote: > > > I have no problem continuing keeping with historical precedent and > > allowing mnemonic abbreviations in our directory and file names at this > > point. > > I'm still in favor of pg_xact. A search of the 9.6 docs brings up a number > of hits for "xact": pg_last_xact_replay_timestamp(), > pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(), > pg_last_committed_xact(), pg_prepared_xacts(), etc. There are also numerous > column names that have "xact" in them. I'm +1 on pg_clog -> pg_xact. Also +1 to renaming pg_subtrans to pg_subxact. In general, +1 to short mnemonic meaningful names. -1 to overly long names, -1 to meaningless names. Regarding pg_receivexlog I think I'd propose pg_recvwal rather than pg_receivewal, because of pg_recvlogical. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > David Steele wrote: >> On 10/21/16 3:12 AM, David G. Johnston wrote: >> >> > I have no problem continuing keeping with historical precedent and >> > allowing mnemonic abbreviations in our directory and file names at this >> > point. >> >> I'm still in favor of pg_xact. A search of the 9.6 docs brings up a number >> of hits for "xact": pg_last_xact_replay_timestamp(), >> pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(), >> pg_last_committed_xact(), pg_prepared_xacts(), etc. There are also numerous >> column names that have "xact" in them. > > I'm +1 on pg_clog -> pg_xact. So let's say that's the winner then. > Also +1 to renaming pg_subtrans to pg_subxact. Nice suggestion, good naming for consistency with the rest. -- Michael
On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote: > On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera > <alvherre@2ndquadrant.com> wrote: > > David Steele wrote: > >> On 10/21/16 3:12 AM, David G. Johnston wrote: > >> > >> > I have no problem continuing keeping with historical precedent and > >> > allowing mnemonic abbreviations in our directory and file names at this > >> > point. > >> > >> I'm still in favor of pg_xact. A search of the 9.6 docs brings up a number > >> of hits for "xact": pg_last_xact_replay_timestamp(), > >> pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(), > >> pg_last_committed_xact(), pg_prepared_xacts(), etc. There are also numerous > >> column names that have "xact" in them. > > > > I'm +1 on pg_clog -> pg_xact. > > So let's say that's the winner then. > > > Also +1 to renaming pg_subtrans to pg_subxact. > > Nice suggestion, good naming for consistency with the rest. Agreed. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
On 10/22/16 6:58 PM, Bruce Momjian wrote: > On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote: >> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera >> >>> Also +1 to renaming pg_subtrans to pg_subxact. >> >> Nice suggestion, good naming for consistency with the rest. > > Agreed. +1 -- -David david@pgmasters.net
On Fri, Oct 21, 2016 at 9:03 PM, Stephen Frost <sfrost@snowman.net> wrote: > WARNING: The following essential packages will be removed. > This should NOT be done unless you know exactly what you are doing! > login > 0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded. > After this operation, 1,212 kB disk space will be freed. > You are about to do something potentially harmful. > To continue type in the phrase 'Yes, do as I say!' Another case was: glxgears -iacknowledgethatthistoolisnotabenchmark Which was required to get it to print the FPS for a while. The problem -- and also the advantage -- of this is that it's scriptable. That means people can still put it in recipe books and scripts and others can copy it without being aware what it does or even that they're doing it. I think the apt-get behaviour was specifically designed to ensure it couldn't easily be put into a script which I would have said was desirable -- except I suspect there are situations where Postgres database scripts need to do a resetxlog. I'm not sure I can think of any examples offhand but I wouldn't be too surprised. -- greg
On 10/22/2016 06:00 PM, David Steele wrote: > On 10/22/16 6:58 PM, Bruce Momjian wrote: >> On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote: >>> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera >>> >>>> Also +1 to renaming pg_subtrans to pg_subxact. >>> >>> Nice suggestion, good naming for consistency with the rest. >> >> Agreed. > > +1 +1 from me, too. -- Vik Fearing +33 6 46 75 15 36 http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On Sun, Oct 23, 2016 at 5:18 PM, Vik Fearing <vik@2ndquadrant.fr> wrote: > On 10/22/2016 06:00 PM, David Steele wrote: >> On 10/22/16 6:58 PM, Bruce Momjian wrote: >>> On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote: >>>> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera >>>> >>>>> Also +1 to renaming pg_subtrans to pg_subxact. >>>> >>>> Nice suggestion, good naming for consistency with the rest. >>> >>> Agreed. >> >> +1 > > +1 from me, too. OK, glad to see that we are reaching a conclusion to this thread. Attached are two patches. 0001 renames pg_clog to pg_xact. I found a comment in xact.c that was not yet updated. And 0002 renames pg_subtrans to pg_subxact. -- Michael
Attachment
On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote: > On Fri, Oct 21, 2016 at 9:03 PM, Stephen Frost <sfrost@snowman.net> wrote: > > WARNING: The following essential packages will be removed. > > This should NOT be done unless you know exactly what you are doing! > > login > > 0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded. > > After this operation, 1,212 kB disk space will be freed. > > You are about to do something potentially harmful. > > To continue type in the phrase 'Yes, do as I say!' > > Another case was: > glxgears -iacknowledgethatthistoolisnotabenchmark > > Which was required to get it to print the FPS for a while. The problem > -- and also the advantage -- of this is that it's scriptable. That > means people can still put it in recipe books and scripts and others > can copy it without being aware what it does or even that they're > doing it. > > I think the apt-get behaviour was specifically designed to ensure it > couldn't easily be put into a script which I would have said was > desirable -- except I suspect there are situations where Postgres > database scripts need to do a resetxlog. I'm not sure I can think of > any examples offhand but I wouldn't be too surprised. Yes, pg_upgrade has eight calls to pg_resetxlog to set various value. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
Bruce Momjian wrote: > On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote: > > I think the apt-get behaviour was specifically designed to ensure it > > couldn't easily be put into a script which I would have said was > > desirable -- except I suspect there are situations where Postgres > > database scripts need to do a resetxlog. I'm not sure I can think of > > any examples offhand but I wouldn't be too surprised. > > Yes, pg_upgrade has eight calls to pg_resetxlog to set various value. There is one difference though, which is that the really destructive use of pg_resetxlog is the one that removes pg_xlog files. The other uses that simply set flags in the control file are not as bad -- you can simply go back to what the original value was. I think we would only need the obnoxious string requirement in the most dangerous mode. Alternatively, we could separate the tool in two different executables. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Mon, Oct 24, 2016 at 11:59:42AM -0300, Alvaro Herrera wrote: > Bruce Momjian wrote: > > On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote: > > > > I think the apt-get behaviour was specifically designed to ensure it > > > couldn't easily be put into a script which I would have said was > > > desirable -- except I suspect there are situations where Postgres > > > database scripts need to do a resetxlog. I'm not sure I can think of > > > any examples offhand but I wouldn't be too surprised. > > > > Yes, pg_upgrade has eight calls to pg_resetxlog to set various value. > > There is one difference though, which is that the really destructive > use of pg_resetxlog is the one that removes pg_xlog files. The other > uses that simply set flags in the control file are not as bad -- you can > simply go back to what the original value was. I think we would only > need the obnoxious string requirement in the most dangerous mode. > > Alternatively, we could separate the tool in two different executables. OK, but we are going need to document that this special flag is only needed for certain option uses. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
Bruce Momjian wrote: > On Mon, Oct 24, 2016 at 11:59:42AM -0300, Alvaro Herrera wrote: > > There is one difference though, which is that the really destructive > > use of pg_resetxlog is the one that removes pg_xlog files. The other > > uses that simply set flags in the control file are not as bad -- you can > > simply go back to what the original value was. I think we would only > > need the obnoxious string requirement in the most dangerous mode. > OK, but we are going need to document that this special flag is only > needed for certain option uses. Perhaps, but not in too much detail; just that "users need to follow on-screen indications" or some such would be enough. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hi Craig,
This is a gentle reminder.
you assigned as reviewer to the current patch in the 11-2016 commitfest.
But you haven't shared your review yet. Please share your review about
the patch. This will help us in smoother operation of commitfest.
Please Ignore if you already shared your review.
Regards,
Hari Babu
Fujitsu Australia
On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi <kommi.haribabu@gmail.com> wrote: > Hi Craig, > > This is a gentle reminder. > > you assigned as reviewer to the current patch in the 11-2016 commitfest. > But you haven't shared your review yet. Please share your review about > the patch. This will help us in smoother operation of commitfest. > > Please Ignore if you already shared your review. I have moved this CF entry to 2017-01, the remaining, still unreviewed patch are for renaming pg_subxact and pg_clog. -- Michael
On Tue, Nov 29, 2016 at 1:35 PM, Michael Paquier <michael.paquier@gmail.com> wrote: > On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi > <kommi.haribabu@gmail.com> wrote: >> Hi Craig, >> >> This is a gentle reminder. >> >> you assigned as reviewer to the current patch in the 11-2016 commitfest. >> But you haven't shared your review yet. Please share your review about >> the patch. This will help us in smoother operation of commitfest. >> >> Please Ignore if you already shared your review. > > I have moved this CF entry to 2017-01, the remaining, still unreviewed > patch are for renaming pg_subxact and pg_clog. The second patch has rotten a little because of the backup documentation. By the way, is something going to happen in the CF? Craig, you are a reviewer of this patch. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
On Tue, Jan 17, 2017 at 4:31 PM, Michael Paquier <michael.paquier@gmail.com> wrote: > On Tue, Nov 29, 2016 at 1:35 PM, Michael Paquier > <michael.paquier@gmail.com> wrote: >> On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi >> <kommi.haribabu@gmail.com> wrote: >>> Hi Craig, >>> >>> This is a gentle reminder. >>> >>> you assigned as reviewer to the current patch in the 11-2016 commitfest. >>> But you haven't shared your review yet. Please share your review about >>> the patch. This will help us in smoother operation of commitfest. >>> >>> Please Ignore if you already shared your review. >> >> I have moved this CF entry to 2017-01, the remaining, still unreviewed >> patch are for renaming pg_subxact and pg_clog. > > The second patch has rotten a little because of the backup > documentation. By the way, is something going to happen in the CF? > Craig, you are a reviewer of this patch. Moved to CF 2017-03. -- Michael
On 1/17/17 2:31 AM, Michael Paquier wrote: > On Tue, Nov 29, 2016 at 1:35 PM, Michael Paquier > <michael.paquier@gmail.com> wrote: >> On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi >> <kommi.haribabu@gmail.com> wrote: >>> Hi Craig, >>> >>> This is a gentle reminder. >>> >>> you assigned as reviewer to the current patch in the 11-2016 commitfest. >>> But you haven't shared your review yet. Please share your review about >>> the patch. This will help us in smoother operation of commitfest. >>> >>> Please Ignore if you already shared your review. >> >> I have moved this CF entry to 2017-01, the remaining, still unreviewed >> patch are for renaming pg_subxact and pg_clog. > > The second patch has rotten a little because of the backup > documentation. By the way, is something going to happen in the CF? > Craig, you are a reviewer of this patch. This patch does not apply cleanly at cccbdde: $ git apply ../other/0001-Rename-pg_clog-to-pg_xact.patch error: doc/src/sgml/ref/pg_resetxlog.sgml: No such file or directory error: patch failed: src/backend/postmaster/autovacuum.c:2468 error: src/backend/postmaster/autovacuum.c: patch does not apply Marked "Waiting on Author". I'd really like to see the rest of the renames happen for v10. It seems like the process got stalled after the pg_wal rename. -- -David david@pgmasters.net
On Fri, Mar 17, 2017 at 12:19 AM, David Steele <david@pgmasters.net> wrote: > This patch does not apply cleanly at cccbdde: > > $ git apply ../other/0001-Rename-pg_clog-to-pg_xact.patch > error: doc/src/sgml/ref/pg_resetxlog.sgml: No such file or directory > error: patch failed: src/backend/postmaster/autovacuum.c:2468 > error: src/backend/postmaster/autovacuum.c: patch does not apply This has rotten again... > Marked "Waiting on Author". > > I'd really like to see the rest of the renames happen for v10. It seems > like the process got stalled after the pg_wal rename. Let's see what happens, attached are refreshed versions. Uncertainty is the fun part of a CF. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
On Thu, Mar 16, 2017 at 9:31 PM, Michael Paquier <michael.paquier@gmail.com> wrote: > On Fri, Mar 17, 2017 at 12:19 AM, David Steele <david@pgmasters.net> wrote: >> This patch does not apply cleanly at cccbdde: >> >> $ git apply ../other/0001-Rename-pg_clog-to-pg_xact.patch >> error: doc/src/sgml/ref/pg_resetxlog.sgml: No such file or directory >> error: patch failed: src/backend/postmaster/autovacuum.c:2468 >> error: src/backend/postmaster/autovacuum.c: patch does not apply > > This has rotten again... > >> Marked "Waiting on Author". >> >> I'd really like to see the rest of the renames happen for v10. It seems >> like the process got stalled after the pg_wal rename. > > Let's see what happens, attached are refreshed versions. Uncertainty > is the fun part of a CF. I understand that the point of renaming pg_clog to pg_xact is that pg_clog contains the dreaded letters l-o-g, which we hypothesize causes DBAs to remove it. (Alternate hypothesis: "So, that's what's clogging my database!") Renaming pg_subtrans to pg_subxact has no such redeeming properties. More, with each of these renamings, we're further separating what things are called in the code (xlog, clog, subtrans) with what they're called in the filesystem (wal, xact, subxact). So if we must rename pg_clog, OK, but can't we leave pg_subtrans alone? It's not hurting anybody. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Fri, Mar 17, 2017 at 11:17 AM, Robert Haas <robertmhaas@gmail.com> wrote: > I understand that the point of renaming pg_clog to pg_xact is that > pg_clog contains the dreaded letters l-o-g, which we hypothesize > causes DBAs to remove it. (Alternate hypothesis: "So, that's what's > clogging my database!") > > Renaming pg_subtrans to pg_subxact has no such redeeming properties. > > More, with each of these renamings, we're further separating what > things are called in the code (xlog, clog, subtrans) with what they're > called in the filesystem (wal, xact, subxact). > > So if we must rename pg_clog, OK, but can't we leave pg_subtrans > alone? It's not hurting anybody. The only argument behind the renaming of pg_subtrans is really consistency with pg_xact, because both deal with transactions. I don't personally mind if this portion of the renaming is left off, as you say anything labelled with "log" is at the origin of this thread. -- Michael
On Thu, Mar 16, 2017 at 10:21 PM, Michael Paquier <michael.paquier@gmail.com> wrote: > On Fri, Mar 17, 2017 at 11:17 AM, Robert Haas <robertmhaas@gmail.com> wrote: >> I understand that the point of renaming pg_clog to pg_xact is that >> pg_clog contains the dreaded letters l-o-g, which we hypothesize >> causes DBAs to remove it. (Alternate hypothesis: "So, that's what's >> clogging my database!") >> >> Renaming pg_subtrans to pg_subxact has no such redeeming properties. >> >> More, with each of these renamings, we're further separating what >> things are called in the code (xlog, clog, subtrans) with what they're >> called in the filesystem (wal, xact, subxact). >> >> So if we must rename pg_clog, OK, but can't we leave pg_subtrans >> alone? It's not hurting anybody. > > The only argument behind the renaming of pg_subtrans is really > consistency with pg_xact, because both deal with transactions. I don't > personally mind if this portion of the renaming is left off, as you > say anything labelled with "log" is at the origin of this thread. Fine! I've committed the pg_clog renaming, but I'd really like to draw the line here. I'm not going to commit the pg_subtrans -> pg_subxact naming and am -1 on anyone else doing so. I think that having the names of things in the code match what shows up in the file system is an awfully desirable property which we should preserve insofar as we can do so without compromising other goals. Not invalidating what users and DBAs think they know about how PostgreSQL works is a good thing, too; there are a lot more people who know something about the directory layout than will read this thread. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Fri, Mar 17, 2017 at 10:58 PM, Robert Haas <robertmhaas@gmail.com> wrote: > Fine! I've committed the pg_clog renaming, but I'd really like to > draw the line here. I'm not going to commit the pg_subtrans -> > pg_subxact naming and am -1 on anyone else doing so. I think that > having the names of things in the code match what shows up in the file > system is an awfully desirable property which we should preserve > insofar as we can do so without compromising other goals. Not > invalidating what users and DBAs think they know about how PostgreSQL > works is a good thing, too; there are a lot more people who know > something about the directory layout than will read this thread. I'm cool with that, thanks for the commit. -- Michael
On Fri, Mar 17, 2017 at 11:03 PM, Michael Paquier <michael.paquier@gmail.com> wrote: > On Fri, Mar 17, 2017 at 10:58 PM, Robert Haas <robertmhaas@gmail.com> wrote: >> Fine! I've committed the pg_clog renaming, but I'd really like to >> draw the line here. I'm not going to commit the pg_subtrans -> >> pg_subxact naming and am -1 on anyone else doing so. I think that >> having the names of things in the code match what shows up in the file >> system is an awfully desirable property which we should preserve >> insofar as we can do so without compromising other goals. Not >> invalidating what users and DBAs think they know about how PostgreSQL >> works is a good thing, too; there are a lot more people who know >> something about the directory layout than will read this thread. > > I'm cool with that, thanks for the commit. monitoring.sgml still has some references to "clog". We should update them? Regards, -- Fujii Masao