Re: Improving compressibility of WAL files - Mailing list pgsql-hackers

From Aidan Van Dyk
Subject Re: Improving compressibility of WAL files
Date
Msg-id 20090108230235.GI12094@yugib.highrise.ca
Whole thread Raw
In response to Improving compressibility of WAL files  (Bruce Momjian <bruce@momjian.us>)
Responses Re: Improving compressibility of WAL files
Re: Improving compressibility of WAL files
List pgsql-hackers
* Bruce Momjian <bruce@momjian.us> [090108 16:43]:
> 
> The attached patch from Aidan Van Dyk zeros out the end of WAL files to
> improve their compressibility. (The patch was originally sent to
> 'general' which explains why it was lost until now.)
> 
> Would someone please eyeball it?;  it is useful for compressing PITR
> logs even if we find a better solution for replication streaming?

The reason I didn't push it was that people claimed it would chew up to
much WAL bandwidhh (causing a large commit latency) when the new 0's are
all written/fsynced at once...

I don't necessarily buy it, because the force_switch is usually either a
1) timeed occurance on an otherwise idle time
2) user-forced (i.e. forced checkpoint/pg_backup, so your IO is going to  be hammered anyways...

But that's why I didn't follow up on it...

There's possible a few other ways to do it, such as zero the WAL on
recycling (but not fsyncing it), and hopefully most of the zero's get
trickled out by the OS before it comes down to a single 16MB fsync, but
not many people seemed too enthused about the whole WAL compressablitly
subject...

But, the way I see things going on -hackers, I must admit, sync-rep (WAL
streaming) looks like it's a long way off and possibly not even going to
do what I want, so *I* would really like this wal zero'ing...

If anybody has any specific things with the patch ehty think needs
chaning, I'll try and accomidate, but I do note that I never
submitted it for the Commitfest...

a.

-- 
Aidan Van Dyk                                             Create like a god,
aidan@highrise.ca                                       command like a king,
http://www.highrise.ca/                                   work like a slave.

pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: Improving compressibility of WAL files
Next
From: Tom Lane
Date:
Subject: Re: Buffer pool statistics in Explain Analyze