Re: Improving compressibility of WAL files - Mailing list pgsql-hackers

From Greg Smith
Subject Re: Improving compressibility of WAL files
Date
Msg-id Pine.GSO.4.64.0901091530320.24949@westnet.com
Whole thread Raw
In response to Re: Improving compressibility of WAL files  (Simon Riggs <simon@2ndQuadrant.com>)
Responses Re: Improving compressibility of WAL files  ("Kevin Grittner" <Kevin.Grittner@wicourts.gov>)
List pgsql-hackers
On Fri, 9 Jan 2009, Simon Riggs wrote:

> Half-filled WAL files were necessary to honour archive_timeout. With
> continuous streaming all WAL files will be 100% full before we switch,
> for most purposes.

The main use case I'm concerned about losing support for is:

1) Two systems connected by a WAN with significant transmit latency
2) The secondary system runs a warm standby aimed at disaster recovery
3) Business requirements want the standby to never be more than (say) 5 
minutes behind the primary, presuming the WAN is up
4) WAN traffic is "expensive" (money==bandwidth, one of the two is scarce)

This seems a pretty common scenario in my experience.  Right now, this 
case is served quite well like this:

-archive_timeout='5 minutes'
-[pglesslog|pg_clearxlogtail] | gzip | rsync

The main concern I have with switching to a more synchronous scheme is 
that network efficiency drops as the payload breaks into smaller pieces. 
I haven't had enough time to keep up with all the sync rep advances 
recently to know for sure if there's a configuration there that's suitable 
for this case.  If that can be configured to send only in relatively large 
chunks, while still never letting things lag too far behind, then I'd 
agree completely that the case for any of these WAL cleaner utilities is 
dead--presuming said support makes it into the next release.

If that's not available, say because the only useful option sends in very 
small pieces, there may still be a need for some utility to fill in for 
this particular requirement.  Luckily there are many to choose from if it 
comes to that.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD


pgsql-hackers by date:

Previous
From: Kenneth Marshall
Date:
Subject: Re: [PATCHES] updated hash functions for postgresql v1
Next
From: Tom Lane
Date:
Subject: Re: New patch for Column-level privileges