Thread: WAL Record Header Size Reduction

WAL Record Header Size Reduction

From

"Simon Riggs"

Date:

25 January 2007, 12:51:42

Current WAL Header uses 32 bytes on a 64-bit CPU. It seems possible to
reduce this to 24 bytes, without reducing resilience, when
full_page_writes = off. This will reduce overall WAL volumes by around
5-15%, depending upon the application with performance gains in various
ways.

If full_page_writes = off then it this is true
xl_tot_len == xl_len + SizeOfXLogRecord
since there are no backup blocks. As a result, there is no loss in
resilience by removing this field.

xlog.h shows this definition currently:

typedef struct XLogRecord
{
pg_crc32 xl_crc; /* CRC for this record */
XLogRecPtr xl_prev; /* ptr to previous record in log */
TransactionId  xl_xid; /* xact id */
uint32 xl_tot_len; /* total len of entire record */
uint32 xl_len; /* total len of rmgr data */
uint8 xl_info; /* flag bits, see below */
RmgrId xl_rmid; /* resource manager for this record */

/* Depending on MAXALIGN, there are either 2 or 6 wasted bytes here */

I propose to rearrange the XLogRecord structure to this:

pg_crc32 xl_crc; /* CRC for this record */
uint8 xl_info; /* flag bits, see below */
RmgrId xl_rmid; /* resource manager for this record */
XLogRecPtr xl_prev; /* ptr to previous record in log */
TransactionId xl_xid; /* xact id */
uint32 xl_tot_len; /* total len of entire record */

which will occupy 24 bytes, saving 4 bytes on 32-bit and 8 bytes on
64-bit architectures, once alignment is considered.

The xl_len field would be included only if backup blocks are included
with the record. This is already marked by flags in the xl_info field,
so no new flags are required.

These changes can be mostly isolated to xlog.c, since only XLogInsert(),
ReadRecord() and pg_resetxlog need to know about the changes. The xlog
record would dynamically adjust according to whether backup blocks are
present, so it can still work as full_page_writes is switched on/off by
user or during the period between start/stop backup.

The saving is only really relevant when full_page_writes = off, so I'm
not worried about changing the xlrec header in that case anyway.

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com

Re: WAL Record Header Size Reduction

From

Heikki Linnakangas

Date:

25 January 2007, 13:05:23

Simon Riggs wrote:
> Current WAL Header uses 32 bytes on a 64-bit CPU. It seems possible to
> reduce this to 24 bytes, without reducing resilience, when
> full_page_writes = off. This will reduce overall WAL volumes by around
> 5-15%, depending upon the application with performance gains in various
> ways.

Actually, it would help even when full_page_writes=on, because even then 
most xlog records don't have backup blocks attached to them.

> xlog.h shows this definition currently:
> 
> typedef struct XLogRecord
> {
> pg_crc32 xl_crc; /* CRC for this record */
> XLogRecPtr xl_prev; /* ptr to previous record in log */
> TransactionId  xl_xid; /* xact id */
> uint32 xl_tot_len; /* total len of entire record */
> uint32 xl_len; /* total len of rmgr data */
> uint8 xl_info; /* flag bits, see below */
> RmgrId xl_rmid; /* resource manager for this record */
> 
> /* Depending on MAXALIGN, there are either 2 or 6 wasted bytes here */
> 
> I propose to rearrange the XLogRecord structure to this:

I think you got your alignment wrong:

> pg_crc32 xl_crc; /* CRC for this record */
> uint8 xl_info; /* flag bits, see below */
> RmgrId xl_rmid; /* resource manager for this record */

Because of xl_prev below which is two uint32 fields, there will be 2 
bytes of wasted space in here.

> XLogRecPtr xl_prev; /* ptr to previous record in log */
> TransactionId xl_xid; /* xact id */
> uint32 xl_tot_len; /* total len of entire record */


ISTM that we would get the effect your looking for by just moving the 
xl_tot_len field to the end, and only storing it for records with backup 
blocks:
> pg_crc32 xl_crc; /* CRC for this record */> XLogRecPtr xl_prev; /* ptr to previous record in log */> TransactionId
xl_xid;/* xact id */> uint32 xl_len; /* total len of rmgr data */> uint8 xl_info; /* flag bits, see below */> RmgrId
xl_rmid;/* resource manager for this record */>> uint32 xl_tot_len; /* total len of entire record, if backup blocks 
 
indicated in xl_info*/

--   Heikki Linnakangas  EnterpriseDB   http://www.enterprisedb.com

Re: WAL Record Header Size Reduction

From

"Simon Riggs"

Date:

25 January 2007, 13:58:29

On Thu, 2007-01-25 at 14:04 +0000, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > Current WAL Header uses 32 bytes on a 64-bit CPU. It seems possible to
> > reduce this to 24 bytes, without reducing resilience, when
> > full_page_writes = off. This will reduce overall WAL volumes by around
> > 5-15%, depending upon the application with performance gains in various
> > ways.
> 
> Actually, it would help even when full_page_writes=on, because even then 
> most xlog records don't have backup blocks attached to them.

Sure it would be active, but I meant it probably would not reduce the
overall WAL volume by very much. We'll see, I guess.

> > xlog.h shows this definition currently:
> > 
> > typedef struct XLogRecord
> > {
> > pg_crc32 xl_crc; /* CRC for this record */
> > XLogRecPtr xl_prev; /* ptr to previous record in log */
> > TransactionId  xl_xid; /* xact id */
> > uint32 xl_tot_len; /* total len of entire record */
> > uint32 xl_len; /* total len of rmgr data */
> > uint8 xl_info; /* flag bits, see below */
> > RmgrId xl_rmid; /* resource manager for this record */
> > 
> > /* Depending on MAXALIGN, there are either 2 or 6 wasted bytes here */
> > 
> > I propose to rearrange the XLogRecord structure to this:
> 
> I think you got your alignment wrong:
> 
> > pg_crc32 xl_crc; /* CRC for this record */
> > uint8 xl_info; /* flag bits, see below */
> > RmgrId xl_rmid; /* resource manager for this record */
> 
> Because of xl_prev below which is two uint32 fields, there will be 2 
> bytes of wasted space in here.

No escaping that though.

> ISTM that we would get the effect your looking for by just moving the 
> xl_tot_len field to the end, and only storing it for records with backup 
> blocks:
> 
>  > pg_crc32 xl_crc; /* CRC for this record */
>  > XLogRecPtr xl_prev; /* ptr to previous record in log */
>  > TransactionId  xl_xid; /* xact id */
>  > uint32 xl_len; /* total len of rmgr data */
>  > uint8 xl_info; /* flag bits, see below */
>  > RmgrId xl_rmid; /* resource manager for this record */
>  >> uint32 xl_tot_len; /* total len of entire record, if backup blocks 
> indicated in xl_info*/

OK. I was assuming XLogRecPtr was 8-byte aligned, but its a struct with
2 4-byte aligned elements, so I thought I had to move it to get the
benefit. Seems not...

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com

Re: WAL Record Header Size Reduction

From

Tom Lane

Date:

25 January 2007, 15:03:38

Heikki Linnakangas <heikki@enterprisedb.com> writes:
> ISTM that we would get the effect your looking for by just moving the 
> xl_tot_len field to the end, and only storing it for records with backup 
> blocks:

I like that formulation better --- seems like less
change-for-the-sake-of-change.
        regards, tom lane

Re: WAL Record Header Size Reduction

From

"Simon Riggs"

Date:

25 January 2007, 15:48:19

On Thu, 2007-01-25 at 11:03 -0500, Tom Lane wrote:
> Heikki Linnakangas <heikki@enterprisedb.com> writes:
> > ISTM that we would get the effect your looking for by just moving the 
> > xl_tot_len field to the end, and only storing it for records with backup 
> > blocks:
> 
> I like that formulation better --- seems like less
> change-for-the-sake-of-change.

Will do.

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com