Thread: Regarding WAL Format Changes
<div class="WordSection1"><p class="MsoNormal">While reading patch-1 (<b><a href="http://archives.postgresql.org/pgsql-hackers/2012-06/txtFskHiYakjO.txt"><span style="color:windowtext;font-weight:normal;text-decoration:none">1-use-uint64-got-segno.patch</span></a></b>)of WAL FormatChanges(http://archives.postgresql.org/message-id/4FDA5136.6080206@enterprisedb.com), I had few observations whichare summarized below:<p class="MsoNormal"> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">1.Function header for following functions still contains refereceto log, seg</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> a. InstallXLogFileSegment()</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> b. RemoveOldXlogFiles()</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> c. XLogFileCopy() </span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> d. XLogGetLastRemoved()</span><br /><spanstyle="font-size:10.0pt;font-family:"Arial","sans-serif""> e. UpdateLastRemovedPtr()</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> f. RemoveOldXlogFiles()</span><br /><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">2.@@ -2680,8 +2645,8 @@ InstallXLogFileSegment(uint32 *log, uint32*seg, char *tmppath,</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> LWLockRelease(ControlFileLock);</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> ereport(LOG,</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> (errcode_for_file_access(),</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- errmsg("could not link file \"%s\" to \"%s\" (initialization of log file %u, segment %u):%m",</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- tmppath, path, *log, *seg)));</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ errmsg("could not link file \"%s\"to \"%s\" (initialization of log file): %m",</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ tmppath, path)));</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> If Changed error message can containlog file and segment number, it would be more clear. That should be easily</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> deducible from segment number.</span><br /><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">3. </span> <span style="font-size:10.0pt;font-family:"Arial","sans-serif"">-RemoveOldXlogFiles(uint32log, uint32 seg, XLogRecPtr endptr)</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+RemoveOldXlogFiles(XLogSegNo segno, XLogRecPtrendptr)</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">.</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">.</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">.</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">@@-4016,8 +3953,9 @@ retry:</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> if (!(((XLogPageHeader) readBuf)->xlp_info& XLP_FIRST_IS_CONTRECORD))</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> {</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> ereport(emode_for_corrupt_record(emode,*RecPtr),</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- (errmsg("thereis no contrecord flag in log file %u, segment %u, offset %u",</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- readId, readSeg, readOff)));</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ (errmsg("there is no contrecord flag in log segment %s, offset %u",</span><br /><spanstyle="font-size:10.0pt;font-family:"Arial","sans-serif"">+ XLogFileNameP(curFileTLI, readSegNo),</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ readOff)));</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> goto next_record_is_invalid;</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> }</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> pageHeaderSize= XLogPageHeaderSize((XLogPageHeader) readBuf);</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">@@-4025,10 +3963,13 @@ retry:</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> if (contrecord->xl_rem_len == 0 ||</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> total_len!= (contrecord->xl_rem_len + gotlen))</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> {</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ char fname[MAXFNAMELEN];</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ XLogFileName(fname, curFileTLI, readSegNo);</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> ereport(emode_for_corrupt_record(emode,*RecPtr),</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- (errmsg("invalidcontrecord length %u in log file %u, segment %u, offset %u",</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ (errmsg("invalidcontrecord length %u in log segment %s, offset %u",</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> contrecord->xl_rem_len,</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- readId, readSeg, readOff)));</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ XLogFileNameP(curFileTLI, readSegNo),</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ readOff)));</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> goto next_record_is_invalid;</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> }</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> For the above 2 changed error messages, 'log segment' is usedfor filename. </span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> However all similar changeshas 'log file' for filename. There are some places</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> where 'log segment' is used and other places it is 'log file'.</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> So is there any particular reason forit?</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">4.</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">@@-533,33 +533,17 @@ pg_xlog_location_diff(PG_FUNCTION_ARGS)</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- /*</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- * Sanity check</span><br /><spanstyle="font-size:10.0pt;font-family:"Arial","sans-serif"">- */</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- if (loc1.xrecoff > XLogFileSize)</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- ereport(ERROR,</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- errmsg("xrecoff \"%X\" is outof valid range, 0..%X", loc1.xrecoff, XLogFileSize)));</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- if (loc2.xrecoff > XLogFileSize)</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- ereport(ERROR,</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- errmsg("xrecoff \"%X\" is outof valid range, 0..%X", loc2.xrecoff, XLogFileSize)));</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ bytes1 = (((uint64)loc1.xlogid) << 32L) + loc1.xrecoff;</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ bytes2 = (((uint64)loc2.xlogid)<< 32L) + loc2.xrecoff;</span><br /><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> Is there no chance that it can be out of valid range after newchanges, just a doubt?</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br /><spanstyle="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">5.</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">---a/src/backend/replication/walreceiver.c</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+++b/src/backend/replication/walreceiver.c</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">@@-69,11 +69,12 @@ walrcv_disconnect_type walrcv_disconnect = NULL;</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> /*</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> * These variables are used similarly to openLogFile/Id/Seg/Off,</span><br/><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">- * but for walreceiverto write the XLOG.</span><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">+ * but for walreceiverto write the XLOG. recvFileTLI is the TimeLineID</span><br /><br /><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Inthe above comments, there is still reference to Id/Seg/Off.</span><pclass="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><p class="MsoNormal"><spanstyle="font-size:10.0pt;font-family:"Arial","sans-serif""> </span><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">WithRegards,</span><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">AmitKapila.</span><p class="MsoNormal"> </div>
On 27.06.2012 17:14, Amit Kapila wrote:
> 1. Function header for following functions still contains referece to log,
> seg
> a. InstallXLogFileSegment()
> b. RemoveOldXlogFiles()
> c. XLogFileCopy()
> d. XLogGetLastRemoved()
> e. UpdateLastRemovedPtr()
> f. RemoveOldXlogFiles()
Thanks, fixed.
> 2. @@ -2680,8 +2645,8 @@ InstallXLogFileSegment(uint32 *log, uint32 *seg,
> char *tmppath,
> LWLockRelease(ControlFileLock);
> ereport(LOG,
> (errcode_for_file_access(),
> - errmsg("could not link file \"%s\" to
> \"%s\" (initialization of log file %u, segment %u): %m",
> - tmppath, path, *log,
> *seg)));
> + errmsg("could not link file \"%s\" to
> \"%s\" (initialization of log file): %m",
> + tmppath, path)));
> If Changed error message can contain log file and segment number, it
> would be more clear. That should be easily
> deducible from segment number.
That seems redundant. The target file name is calculated from the
segment number, and we're now using the file name instead of log+seg in
other messages too.
> 3. -RemoveOldXlogFiles(uint32 log, uint32 seg, XLogRecPtr endptr)
> +RemoveOldXlogFiles(XLogSegNo segno, XLogRecPtr endptr)
> .
> .
> .
> @@ -4016,8 +3953,9 @@ retry:
> if (!(((XLogPageHeader) readBuf)->xlp_info&
> XLP_FIRST_IS_CONTRECORD))
> {
> ereport(emode_for_corrupt_record(emode,
> *RecPtr),
> - (errmsg("there is no
> contrecord flag in log file %u, segment %u, offset %u",
> - readId,
> readSeg, readOff)));
> + (errmsg("there is no
> contrecord flag in log segment %s, offset %u",
> +
> XLogFileNameP(curFileTLI, readSegNo),
> + readOff)));
>
> goto next_record_is_invalid;
> }
> pageHeaderSize =
> XLogPageHeaderSize((XLogPageHeader) readBuf);
> @@ -4025,10 +3963,13 @@ retry:
> if (contrecord->xl_rem_len == 0 ||
> total_len != (contrecord->xl_rem_len +
> gotlen))
> {
> + char fname[MAXFNAMELEN];
> + XLogFileName(fname, curFileTLI, readSegNo);
>
> ereport(emode_for_corrupt_record(emode,
> *RecPtr),
> - (errmsg("invalid contrecord
> length %u in log file %u, segment %u, offset %u",
> + (errmsg("invalid contrecord
> length %u in log segment %s, offset %u",
>
> contrecord->xl_rem_len,
> - readId,
> readSeg, readOff)));
> +
> XLogFileNameP(curFileTLI, readSegNo),
> + readOff)));
>
> goto next_record_is_invalid;
> }
>
> For the above 2 changed error messages, 'log segment' is used for
> filename.
> However all similar changes has 'log file' for filename. There are some
> places
> where 'log segment' is used and other places it is 'log file'.
> So is there any particular reason for it?
Not really. There are several messages that use "log file %s", and also
several places that use "log segment %s" Should we make it consistent
and use either "log segment" or "log file" everywhere?
> 4. @@ -533,33 +533,17 @@ pg_xlog_location_diff(PG_FUNCTION_ARGS)
> - /*
> - * Sanity check
> - */
> - if (loc1.xrecoff> XLogFileSize)
> - ereport(ERROR,
> - (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> - errmsg("xrecoff \"%X\" is out of valid
> range, 0..%X", loc1.xrecoff, XLogFileSize)));
> - if (loc2.xrecoff> XLogFileSize)
> - ereport(ERROR,
> - (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> - errmsg("xrecoff \"%X\" is out of valid
> range, 0..%X", loc2.xrecoff, XLogFileSize)));
> + bytes1 = (((uint64)loc1.xlogid)<< 32L) + loc1.xrecoff;
> + bytes2 = (((uint64)loc2.xlogid)<< 32L) + loc2.xrecoff;
>
> Is there no chance that it can be out of valid range after new changes,
> just a doubt?
No. Not in the sense it used to be, anyway, the XLogFileSize check is no
longer relevant. Perhaps we should check for InvalidXLogRecPtr or that
the pointer doesn't point e.g in the middle of a page header. But then
again, this calculation works fine with both of those cases, so I see no
reason to make it stricter.
> 5.
> --- a/src/backend/replication/walreceiver.c
> +++ b/src/backend/replication/walreceiver.c
> @@ -69,11 +69,12 @@ walrcv_disconnect_type walrcv_disconnect = NULL;
>
> /*
> * These variables are used similarly to openLogFile/Id/Seg/Off,
> - * but for walreceiver to write the XLOG.
> + * but for walreceiver to write the XLOG. recvFileTLI is the TimeLineID
>
> In the above comments, there is still reference to Id/Seg/Off.
Thanks, fixed.
-- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Excerpts from Heikki Linnakangas's message of mié jun 27 10:56:13 -0400 2012: > On 27.06.2012 17:14, Amit Kapila wrote: > > For the above 2 changed error messages, 'log segment' is used for > > filename. > > However all similar changes has 'log file' for filename. There are some > > places > > where 'log segment' is used and other places it is 'log file'. > > So is there any particular reason for it? > > Not really. There are several messages that use "log file %s", and also > several places that use "log segment %s" Should we make it consistent > and use either "log segment" or "log file" everywhere? I think it would be better to use "log segment" for WAL segments. That way we don't cause confusion with the regular text/csv log output files. Heck, maybe even "WAL segment" instead of "log". As a translator, I can't have a single, clear explanation of what "log file" is because there are multiple meanings. It would be better not to depend on context. -- Álvaro Herrera <alvherre@commandprompt.com> The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On Wed, Jun 27, 2012 at 11:56 PM, Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> wrote: > On 27.06.2012 17:14, Amit Kapila wrote: >> >> 1. Function header for following functions still contains referece to log, >> seg >> a. InstallXLogFileSegment() >> b. RemoveOldXlogFiles() >> c. XLogFileCopy() >> d. XLogGetLastRemoved() >> e. UpdateLastRemovedPtr() >> f. RemoveOldXlogFiles() > > > Thanks, fixed. There is still reference to log/seg in the comment of InstallXLogFileSegment(). The attached patch should be applied? Regards, -- Fujii Masao
Attachment
On 27.06.2012 18:55, Fujii Masao wrote: > On Wed, Jun 27, 2012 at 11:56 PM, Heikki Linnakangas > <heikki.linnakangas@enterprisedb.com> wrote: >> On 27.06.2012 17:14, Amit Kapila wrote: >>> >>> 1. Function header for following functions still contains referece to log, >>> seg >>> a. InstallXLogFileSegment() >>> b. RemoveOldXlogFiles() >>> c. XLogFileCopy() >>> d. XLogGetLastRemoved() >>> e. UpdateLastRemovedPtr() >>> f. RemoveOldXlogFiles() >> >> >> Thanks, fixed. > > There is still reference to log/seg in the comment of InstallXLogFileSegment(). > The attached patch should be applied? Thanks, applied. Sorry for being so sloppy with this.. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:
>> So is there any particular reason for it?
> Not really. There are several messages that use "log file %s", and also
> several places that use "log segment %s" Should we make it consistent
> and use either "log segment" or "log file" everywhere?
+1 for uniformity. I think I'd vote for using "file" and eliminating
the "segment" terminology altogether, but the other direction would be
okay too, and might require fewer changes.
IIRC, in the original coding "segment" meant 16MB worth of WAL while
"file" was sometimes used to denote 4GB worth (ie, the boundaries where
we had to increment the high half of the LSN struct). Now that 4GB
boundaries are not special, there's no reason to retain the "file"
concept or terminology.
regards, tom lane
On Thu, Jun 28, 2012 at 1:13 AM, Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> wrote: > On 27.06.2012 18:55, Fujii Masao wrote: >> >> On Wed, Jun 27, 2012 at 11:56 PM, Heikki Linnakangas >> <heikki.linnakangas@enterprisedb.com> wrote: >>> >>> On 27.06.2012 17:14, Amit Kapila wrote: >>>> >>>> >>>> 1. Function header for following functions still contains referece to >>>> log, >>>> seg >>>> a. InstallXLogFileSegment() >>>> b. RemoveOldXlogFiles() >>>> c. XLogFileCopy() >>>> d. XLogGetLastRemoved() >>>> e. UpdateLastRemovedPtr() >>>> f. RemoveOldXlogFiles() >>> >>> >>> >>> Thanks, fixed. >> >> >> There is still reference to log/seg in the comment of >> InstallXLogFileSegment(). >> The attached patch should be applied? > > > Thanks, applied. Sorry for being so sloppy with this.. No problem. WAL format change you did is really nice! Regards, -- Fujii Masao
From: Heikki Linnakangas [mailto:heikki.linnakangas@enterprisedb.com]
Sent: Wednesday, June 27, 2012 8:26 PM
On 27.06.2012 17:14, Amit Kapila wrote:
>> 2. @@ -2680,8 +2645,8 @@ InstallXLogFileSegment(uint32 *log, uint32 *seg,
>> char *tmppath,
>> LWLockRelease(ControlFileLock);
>> ereport(LOG,
>> (errcode_for_file_access(),
>> - errmsg("could not link file \"%s\" to
>> \"%s\" (initialization of log file %u, segment %u): %m",
>> - tmppath, path, *log,
>> *seg)));
>> + errmsg("could not link file \"%s\" to
>> \"%s\" (initialization of log file): %m",
>> + tmppath, path)));
>> If Changed error message can contain log file and segment number, it
>> would be more clear. That should be easily
>> deducible from segment number.
>That seems redundant. The target file name is calculated from the
>segment number, and we're now using the file name instead of log+seg in
>other messages too.
errmsg("could not link file \"%s\" to \"%s\" (initialization of log file):
%m", + tmppath, path)));
In this if we try to get the meaning of second part of message
"(initialization of log file)", it was much
better previously as in this message it refers 2 files and previously it was
clear initialization of which log
file failed. So we can mention file name in second part of message
"(initialization of log file)" as well.
>> 3. -RemoveOldXlogFiles(uint32 log, uint32 seg, XLogRecPtr endptr)
>> For the above 2 changed error messages, 'log segment' is used for
>> filename.
>> However all similar changes has 'log file' for filename. There are
some
>> places
>> where 'log segment' is used and other places it is 'log file'.
>> So is there any particular reason for it?
> Not really. There are several messages that use "log file %s", and also
> several places that use "log segment %s" Should we make it consistent
> and use either "log segment" or "log file" everywhere?
'file' seems to be better option as some users may not be even aware of
segments, they would be using default values of segments and they can relate
to 'file' easily.
Also using 'WAL' instead of 'log' as suggested by Alvaro is good if others
also thinks same.
With Regards,
Amit Kapila.