Thread: WAL/PITR additional items
A range of WAL/PITR items still need work. I'm fairly sure I won't be doing any of these in time for 8.1, so feel free to have a go. 1. PITR archive recovery could do more to overlap recovery of files, so that recovery overall is faster. This is already possible by designing your archive command to perform this buffering for you, but it may be a good idea to build this directly into the recovery scheme. 2. PreallocXlogFiles to run from a separate process to make it effective, as discussed last year. The most natural place to put it would be the Archiver, though that is only present when archive_command is set. That means we could have the bgwriter execute it every so often, perhaps every 5 or 10 cycles to avoid lock contention. 3. If PreallocXlogFiles becomes effective, it should be possible to turn a pg_xlog directory out of space PANIC into a checkpoint to recycle outstanding WAL files, then retry the prealloc. If that helps then reset checkpoint_segments (and issue a WARNING to say it has happened) so that the out of space condition is not encountered again. 4. CRC reduction to CRC-32, plus hardware tuning. I'll stay in touch with that in case we have further information to tune the assembler. 5. WAL archive compression 6. xlog.c refactoring into two pieces: WAL writing and WAL recovery Best Regards, Simon Riggs
Should any of these be added to TODO? --------------------------------------------------------------------------- Simon Riggs wrote: > A range of WAL/PITR items still need work. I'm fairly sure I won't be > doing any of these in time for 8.1, so feel free to have a go. > > 1. PITR archive recovery could do more to overlap recovery of files, so > that recovery overall is faster. This is already possible by designing > your archive command to perform this buffering for you, but it may be a > good idea to build this directly into the recovery scheme. > > 2. PreallocXlogFiles to run from a separate process to make it > effective, as discussed last year. The most natural place to put it > would be the Archiver, though that is only present when archive_command > is set. That means we could have the bgwriter execute it every so often, > perhaps every 5 or 10 cycles to avoid lock contention. > > 3. If PreallocXlogFiles becomes effective, it should be possible to turn > a pg_xlog directory out of space PANIC into a checkpoint to recycle > outstanding WAL files, then retry the prealloc. If that helps then reset > checkpoint_segments (and issue a WARNING to say it has happened) so that > the out of space condition is not encountered again. > > 4. CRC reduction to CRC-32, plus hardware tuning. I'll stay in touch > with that in case we have further information to tune the assembler. > > 5. WAL archive compression > > 6. xlog.c refactoring into two pieces: WAL writing and WAL recovery > > Best Regards, Simon Riggs > > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Don't 'kill -9' the postmaster > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Simon Riggs <simon@2ndquadrant.com> writes:
> 2. PreallocXlogFiles to run from a separate process to make it
> effective, as discussed last year.
What?
> The most natural place to put it
> would be the Archiver, though that is only present when archive_command
> is set.
If there's any use to this, which I doubt, the archiver is certainly not
the place for it, since not everyone uses archiving.
regards, tom lane
On Wed, 20 Apr 2005, Tom Lane wrote: > Simon Riggs <simon@2ndquadrant.com> writes: > > 2. PreallocXlogFiles to run from a separate process to make it > > effective, as discussed last year. > > What? The discussion Simon is refering to came up during the 8.0 beta IIRC. The problem was that we were not allocating xlogs quickly enough under heavy workloads and there was some discussion about the bgwriter taking over this task since it could assess the need for new xlogs more often. Gavin
Gavin Sherry <swm@linuxworld.com.au> writes:
> On Wed, 20 Apr 2005, Tom Lane wrote:
>> What?
> The discussion Simon is refering to came up during the 8.0 beta IIRC. The
> problem was that we were not allocating xlogs quickly enough under
> heavy workloads and there was some discussion about the bgwriter taking
> over this task since it could assess the need for new xlogs more often.
Huh? The bgwriter already has this task, since it runs checkpoints.
It's possible that we ought to allow more "slop" in the number of
prealloc'd xlog segments --- I think that the current code is probably
too enthusiastic about deleting "extra" segments after a spike in
activity subsides. But I don't see the point of moving the
responsibility somewhere else.
regards, tom lane
On Wed, 20 Apr 2005, Tom Lane wrote: > Gavin Sherry <swm@linuxworld.com.au> writes: > > On Wed, 20 Apr 2005, Tom Lane wrote: > >> What? > > > The discussion Simon is refering to came up during the 8.0 beta IIRC. The > > problem was that we were not allocating xlogs quickly enough under > > heavy workloads and there was some discussion about the bgwriter taking > > over this task since it could assess the need for new xlogs more often. > > Huh? The bgwriter already has this task, since it runs checkpoints. > > It's possible that we ought to allow more "slop" in the number of > prealloc'd xlog segments --- I think that the current code is probably > too enthusiastic about deleting "extra" segments after a spike in > activity subsides. But I don't see the point of moving the > responsibility somewhere else. I guess I was recalling this part of the earlier thread: http://archives.postgresql.org/pgsql-hackers/2004-07/msg01088.php Thanks, Gavin
Gavin Sherry <swm@linuxworld.com.au> writes: > I guess I was recalling this part of the earlier thread: > http://archives.postgresql.org/pgsql-hackers/2004-07/msg01088.php Mmm. The question really is whether PreallocXlogFiles does anything useful at all anymore. It once would allocate multiple segments but Bruce took out that parameter on the grounds that it was useless... and I think the functionality that's left may be useless too. The code that *really* creates forward log segments these days is the logic that recycles old log segments when we are done with 'em. Once you reach a steady state, each checkpoint will cause about checkpoint_segments forward xlog segments to come into being. That's way more than what PreallocXlogFiles will ever do, and at least in steady state it's sufficient. Maybe what we need is for PreallocXlogFiles to make sure there are checkpoint_segments forward segments, not only one. Given that functionality, running it only during checkpoints would be sufficient. (It'd probably be sufficient to run it only at startup and when checkpoint_segments changes, actually...) regards, tom lane
On Wed, 2005-04-20 at 17:18 -0400, Bruce Momjian wrote: > Should any of these be added to TODO? Yes, please - though some already are. Best Regards, Simon Riggs
Simon Riggs wrote: > On Wed, 2005-04-20 at 17:18 -0400, Bruce Momjian wrote: > > Should any of these be added to TODO? > > Yes, please - though some already are. Sorry, I got lost in the discussion. What are they? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073