Thread: xlog file naming
Isn't the naming of the xlog files slightly bogus? We have the following sequence: 00000001000008D0000000FD 00000001000008D0000000FE 00000001000008D100000000 Ignoring that we skip FF for some obscure reason (*), these are effectively supposed to be 64-bit numbers, chunked into 16MB pieces, so shouldn't the naming be 00000001000008D0FD000000 00000001000008D0FE000000 00000001000008D100000000 Then a lot of other things would also start making more sense: The backup file 00000001000008D1000000C9.00013758.backup should really be 00000001000008D1C9013758.backup (At least conceptually. It's debatable whether we'd want to change that, since as it is, it's convenient to detect the preceding WAL file name by cutting off the end. OTOH, it's safer to actually look into the backup file for that information.) The return value of pg_start_backup that currently looks something like pg_start_backup -----------------8D1/C9013758 should really be 000008D1C9013758 (perhaps the timeline should be included?) and similarly, log output and pg_controldata output like Latest checkpoint location: 8D3/5A1BB578 should be Latest checkpoint location: 000008D35A1BB578 Then all instances of xlog location would look the same. Also, the documentation offers this example: "For example, if the starting WAL file is 0000000100001234000055CD the backup history file will be named something like 0000000100001234000055CD.007C9330.backup." Both the WAL and the backup file names used here are actually impossible, and are not helping to clarify the issue. It seems to me that this is all uselessly complicated and confused. (*) - That idea appears to come from the same aboriginal confusion about WAL "files" vs "segments" vs WAL position. Unless we support WAL segments of size 1 or 2 bytes, there shouldn't be any risk of overflowing the segment counter. PS2: While we're discussing the cleanup of xlog.c, someone daring could replace XLogRecPtr by a plain uint64 and possibly save hundres of lines of code and eliminate a lot of the above confusion.
On 12.09.2011 21:36, Peter Eisentraut wrote: > PS2: While we're discussing the cleanup of xlog.c, someone daring could > replace XLogRecPtr by a plain uint64 and possibly save hundres of lines > of code and eliminate a lot of the above confusion. One problem with that is that it would break binary backwards-compatibility on little-endian systems. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Peter Eisentraut <peter_e@gmx.net> writes: > Isn't the naming of the xlog files slightly bogus? No doubt, but by now there's enough replication-ish third-party code that knows about them that I'm afraid changing these conventions would be much more painful than it's worth. regards, tom lane
On Tue, Sep 13, 2011 at 3:36 AM, Peter Eisentraut <peter_e@gmx.net> wrote: > The return value of pg_start_backup that currently looks something like > > pg_start_backup > ----------------- > 8D1/C9013758 > > should really be > > 000008D1C9013758 > > (perhaps the timeline should be included?) This change might break some tools which depends on such a result format. Instead of that, what about introducing something like pg_current_timeline() function which literally returns the current timeline ID, and extending pg_xlogfile_name() so that it accepts not only LSN but also the timeline? This idea enables us to get the backup start WAL file name by executing "SELECT pg_xlogfile_name(pg_current_timeline(), pg_start_backup());". Furthermore, in the same way, we can get the backup end WAL file name or current WAL file name from pg_stop_backup() or pg_current_xlog_location(), respectively. > and similarly, log output and pg_controldata output like > > Latest checkpoint location: 8D3/5A1BB578 > > should be > > Latest checkpoint location: 000008D35A1BB578 LSN information is useful for at least debug purpose. So, what about leaving LSN information as it is and appending the filename in the end-of-line as follows? backup_label represents the backup start location in this way. Latest checkpoint location: 0/2000020 (file 0000000000000002) Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
On Mon, Sep 12, 2011 at 8:44 PM, Fujii Masao <masao.fujii@gmail.com> wrote: > On Tue, Sep 13, 2011 at 3:36 AM, Peter Eisentraut <peter_e@gmx.net> wrote: >> The return value of pg_start_backup that currently looks something like >> >> pg_start_backup >> ----------------- >> 8D1/C9013758 >> >> should really be >> >> 000008D1C9013758 >> >> (perhaps the timeline should be included?) > > This change might break some tools which depends on such a result format. > Instead of that, what about introducing something like pg_current_timeline() > function which literally returns the current timeline ID, until here i found it could have some use > and extending > pg_xlogfile_name() so that it accepts not only LSN but also the timeline? > This idea enables us to get the backup start WAL file name by executing > "SELECT pg_xlogfile_name(pg_current_timeline(), pg_start_backup());". > we can do "SELECT pg_xlogfile_name(pg_start_backup())" now, so how is that different? > Furthermore, in the same way, we can get the backup end WAL file name or > current WAL file name from pg_stop_backup() or pg_current_xlog_location(), > respectively. > we can do that already too -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte 24x7 y capacitación
On Tue, Sep 13, 2011 at 12:18 PM, Jaime Casanova <jaime@2ndquadrant.com> wrote: >> and extending >> pg_xlogfile_name() so that it accepts not only LSN but also the timeline? >> This idea enables us to get the backup start WAL file name by executing >> "SELECT pg_xlogfile_name(pg_current_timeline(), pg_start_backup());". > > we can do "SELECT pg_xlogfile_name(pg_start_backup())" now, so how is > that different? Yeah, no difference. My idea is obviously useless for pg_start_backup().. What I had in mind was to use the extended pg_xlogfile_name() to calculate the WAL filename from (for example) pg_controldata executed on the standby. The timeline which pg_controldata displays might be different from the current one in the master. So pg_xlogfile_name() which always use the current timeline might return a wrong WAL filename. Allowing pg_xlogfile_name() to use the specified timeline to calculate a filename would address that problem. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
Are there any TODO items here? --------------------------------------------------------------------------- On Mon, Sep 12, 2011 at 09:36:02PM +0300, Peter Eisentraut wrote: > Isn't the naming of the xlog files slightly bogus? > > We have the following sequence: > > 00000001000008D0000000FD > 00000001000008D0000000FE > 00000001000008D100000000 > > Ignoring that we skip FF for some obscure reason (*), these are > effectively supposed to be 64-bit numbers, chunked into 16MB pieces, so > shouldn't the naming be > > 00000001000008D0FD000000 > 00000001000008D0FE000000 > 00000001000008D100000000 > > Then a lot of other things would also start making more sense: > > The backup file > > 00000001000008D1000000C9.00013758.backup > > should really be > > 00000001000008D1C9013758.backup > > (At least conceptually. It's debatable whether we'd want to change > that, since as it is, it's convenient to detect the preceding WAL file > name by cutting off the end. OTOH, it's safer to actually look into the > backup file for that information.) > > The return value of pg_start_backup that currently looks something like > > pg_start_backup > ----------------- > 8D1/C9013758 > > should really be > > 000008D1C9013758 > > (perhaps the timeline should be included?) > > and similarly, log output and pg_controldata output like > > Latest checkpoint location: 8D3/5A1BB578 > > should be > > Latest checkpoint location: 000008D35A1BB578 > > Then all instances of xlog location would look the same. > > Also, the documentation offers this example: > > "For example, if the starting WAL file is 0000000100001234000055CD the > backup history file will be named something like > 0000000100001234000055CD.007C9330.backup." > > Both the WAL and the backup file names used here are actually > impossible, and are not helping to clarify the issue. > > It seems to me that this is all uselessly complicated and confused. > > > (*) - That idea appears to come from the same aboriginal confusion about > WAL "files" vs "segments" vs WAL position. Unless we support WAL > segments of size 1 or 2 bytes, there shouldn't be any risk of > overflowing the segment counter. > > > PS2: While we're discussing the cleanup of xlog.c, someone daring could > replace XLogRecPtr by a plain uint64 and possibly save hundres of lines > of code and eliminate a lot of the above confusion. > > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On Wed, Aug 15, 2012 at 8:43 PM, Bruce Momjian <bruce@momjian.us> wrote: > Are there any TODO items here? It's possible there's something we want to change here, but it's far from obvious what that thing is. Our WAL file handling is ridiculously hard to understand, but the problem with changing it is that there will then be two things people have to understand, and a lot of tools that have to be revamped. It isn't clear that it's worth going through that kind of pain for a minor improvement in clarity. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Tue, 2012-08-21 at 12:01 -0400, Robert Haas wrote: > On Wed, Aug 15, 2012 at 8:43 PM, Bruce Momjian <bruce@momjian.us> wrote: > > Are there any TODO items here? > > It's possible there's something we want to change here, but it's far > from obvious what that thing is. Our WAL file handling is > ridiculously hard to understand, but the problem with changing it is > that there will then be two things people have to understand, and a > lot of tools that have to be revamped. It isn't clear that it's worth > going through that kind of pain for a minor improvement in clarity. The idea was that since we already broke some tools, possibly silently (...FF files that they previously skipped), a more radical renaming might break those tools more obviously, and make some other things simpler/easier down the road.
Peter Eisentraut <peter_e@gmx.net> writes: > On Tue, 2012-08-21 at 12:01 -0400, Robert Haas wrote: >> It's possible there's something we want to change here, but it's far >> from obvious what that thing is. Our WAL file handling is >> ridiculously hard to understand, but the problem with changing it is >> that there will then be two things people have to understand, and a >> lot of tools that have to be revamped. It isn't clear that it's worth >> going through that kind of pain for a minor improvement in clarity. > The idea was that since we already broke some tools, possibly silently > (...FF files that they previously skipped), a more radical renaming > might break those tools more obviously, and make some other things > simpler/easier down the road. I think we already had that discussion, and the consensus was that we did not want to break WAL-related tools unnecessarily. If there were a high probability that the FF change will actually break tools in practice, the conclusion might have been different; but nobody believes that there is much risk there. regards, tom lane