Thread: DeArchiver process

DeArchiver process

From
Simon Riggs
Date:
Currently, the Startup process is responsible for running
restore_command. So when the Startup process is busy or waiting, then
no new WAL files arrive.

That has these effects
* Recovery must wait while the Startup process requests next WAL file.
This reduces performance of archive recovery.
* If replication is file-based then no new files can be downloaded
while we are waiting. If the Startup process waits, it then is much
slower to catch up than it could be if it had already downloaded the
files from the archive.
* We cannot run an archive_cleanup_command, so the archive keep growing.
* Cascading from a standby that uses file based replication is not
easily possible

My solution is to create a new process called the DeArchiver. This
will run restore_command in a tight loop until the number of files
would exceed wal_keep_files, then sleep. Each time the DeArchiver
executes restore_command it will set the return code and if rc=0 the
new XLogRecPtr reached. If standby_mode = on it will continue to retry
indefinitely.

The Startup process will just read files from pg_xlog rather than from
the archive, just as it does for streaming, so this will remove the
special case code in xlog.c. (WALReciver and this process will still
need to coordinate so they are not both simultaneously active at any
point, as now).

This proposal gives a performance gain because the DeArchiver can be
restoring the next file while the Startup process is processing the
current file, so they work together using pipeline parallelism.

The DeArchiver would start when we are not in crash recovery and exit
at the end of recovery. This would then allow restore_command to be
set via reload rather than restart.

Previously, we have given greater weight to files from the archive to
files already in pg_xlog. To ensure that behaviour continues, if
restore_command is set at the Startup process will read the files in
the pg_xlog directory and remember which ones were there at startup.
That way it will be able to tell the difference between files newly
downloaded and those already in the directory. If a file is absent
from the archive we will use the file from pg_xlog.

This makes file-based and stream-based replication work in a similar
way, which is neater, and it also means all required files are
available in case of a crash, which means we can more easily get rid
of shutdown checkpoints in case of failoiver (discussed on separate
thread).

Since more files are available, it allows cascading replication to
have a sender which receives WAL data in files.

Which do we prefer "DeArchiver", "Restore process", or "WALFileReceiver".

Thoughts?

--
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: DeArchiver process

From
Robert Haas
Date:
On Wed, Nov 2, 2011 at 11:56 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> My solution is to create a new process called the DeArchiver. This
> will run restore_command in a tight loop until the number of files
> would exceed wal_keep_files, then sleep. Each time the DeArchiver
> executes restore_command it will set the return code and if rc=0 the
> new XLogRecPtr reached. If standby_mode = on it will continue to retry
> indefinitely.

Are you thinking of reusing the existing GUC wal_keep_segments (not
wal_keep_files), or creating a new one?  I'd suggest creating a new
one, so as to avoid having a GUC that does one thing on the master and
something quite different on the slave.

> Which do we prefer "DeArchiver", "Restore process", or "WALFileReceiver".

My personal preference would be restore process, since we already use
the name restore_command.

> Thoughts?

+1.  Great idea.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


Re: DeArchiver process

From
Dimitri Fontaine
Date:
Simon Riggs <simon@2ndQuadrant.com> writes:
> My solution is to create a new process called the DeArchiver. This
> will run restore_command in a tight loop until the number of files
> would exceed wal_keep_files, then sleep. Each time the DeArchiver
> executes restore_command it will set the return code and if rc=0 the
> new XLogRecPtr reached. If standby_mode = on it will continue to retry
> indefinitely.

+1

I think you mean wal_keep_segments, a GUC that we already have.

> Which do we prefer "DeArchiver", "Restore process", or "WALFileReceiver".

The only part of your proposal that I don't like is the process name,
that "deArchiver" thing.  "wal restore process" or something like that
would be better.  We already have "wal writer process" and "wal sender
process" and "wal receiver process".

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support


Re: DeArchiver process

From
Simon Riggs
Date:
On Wed, Nov 2, 2011 at 4:14 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, Nov 2, 2011 at 11:56 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> My solution is to create a new process called the DeArchiver. This
>> will run restore_command in a tight loop until the number of files
>> would exceed wal_keep_files, then sleep. Each time the DeArchiver
>> executes restore_command it will set the return code and if rc=0 the
>> new XLogRecPtr reached. If standby_mode = on it will continue to retry
>> indefinitely.
>
> Are you thinking of reusing the existing GUC wal_keep_segments (not
> wal_keep_files), or creating a new one?  I'd suggest creating a new
> one, so as to avoid having a GUC that does one thing on the master and
> something quite different on the slave.

Any standby can now become a sender node, so the meaning in that case
would be the same. That takes a little time to get your head around,
and I'm not used to it myself yet.

I guess you might say that you know for certain that a node will never
be a relay node, and so a different meaning is possible, but it seems
easier to avoid adding a new parameter just for that case.


>> Which do we prefer "DeArchiver", "Restore process", or "WALFileReceiver".
>
> My personal preference would be restore process, since we already use
> the name restore_command.

Restore process, with file called restore.c in src/backend/postmaster
(or src/backend/replication?)

We can change that again later, but its enough to be getting on with.


>> Thoughts?
>
> +1.  Great idea.

Thanks.


--
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: DeArchiver process

From
Robert Haas
Date:
On Wed, Nov 2, 2011 at 12:42 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
> Any standby can now become a sender node, so the meaning in that case
> would be the same. That takes a little time to get your head around,
> and I'm not used to it myself yet.

I think a new parameter will be more clear, even if in practice the
difference is fairly thin.

>>> Which do we prefer "DeArchiver", "Restore process", or "WALFileReceiver".
>>
>> My personal preference would be restore process, since we already use
>> the name restore_command.
>
> Restore process, with file called restore.c in src/backend/postmaster
> (or src/backend/replication?)

Yeah, that works.  I'd go for postmaster over replication, for parallelism.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


Re: DeArchiver process

From
Tom Lane
Date:
Dimitri Fontaine <dimitri@2ndQuadrant.fr> writes:
> The only part of your proposal that I don't like is the process name,
> that "deArchiver" thing.  "wal restore process" or something like that
> would be better.  We already have "wal writer process" and "wal sender
> process" and "wal receiver process".

+1, "restore" seems pretty vague in this context.
        regards, tom lane


Re: DeArchiver process

From
Simon Riggs
Date:
On Wed, Nov 2, 2011 at 5:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Dimitri Fontaine <dimitri@2ndQuadrant.fr> writes:
>> The only part of your proposal that I don't like is the process name,
>> that "deArchiver" thing.  "wal restore process" or something like that
>> would be better.  We already have "wal writer process" and "wal sender
>> process" and "wal receiver process".
>
> +1, "restore" seems pretty vague in this context.

Yeh, walrestore seems more natural than just "restore".

--
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: DeArchiver process

From
Fujii Masao
Date:
On Thu, Nov 3, 2011 at 2:52 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> On Wed, Nov 2, 2011 at 5:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Dimitri Fontaine <dimitri@2ndQuadrant.fr> writes:
>>> The only part of your proposal that I don't like is the process name,
>>> that "deArchiver" thing.  "wal restore process" or something like that
>>> would be better.  We already have "wal writer process" and "wal sender
>>> process" and "wal receiver process".
>>
>> +1, "restore" seems pretty vague in this context.
>
> Yeh, walrestore seems more natural than just "restore".

+1 with this name and whole idea.

If we introduce "walrestore" process, pg_standby seems no longer useful.
We should get rid of it?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center


Re: DeArchiver process

From
Simon Riggs
Date:
On Fri, Nov 4, 2011 at 2:36 AM, Fujii Masao <masao.fujii@gmail.com> wrote:

> If we introduce "walrestore" process, pg_standby seems no longer useful.
> We should get rid of it?

Removing things too quickly can cause problems. There's no harm done
by keeping it a while longer.

I agree it should go, just want to be absolutely clear that its no
longer needed for any use case.

--
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: DeArchiver process

From
Dimitri Fontaine
Date:
Fujii Masao <masao.fujii@gmail.com> writes:
> If we introduce "walrestore" process, pg_standby seems no longer useful.

pg_standby is one possible restore_command, right?  I had understood
that walrestore would be the process that cares for running that
command, not another implementation of it.

That said, I would really like us to provide a default restore command,
so if you had any intend of handling the restoring command in the
walrestore process, by all means, go ahead :)

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support


Re: DeArchiver process

From
Simon Riggs
Date:
On Fri, Nov 4, 2011 at 11:08 AM, Dimitri Fontaine
<dimitri@2ndquadrant.fr> wrote:
> Fujii Masao <masao.fujii@gmail.com> writes:
>> If we introduce "walrestore" process, pg_standby seems no longer useful.
>
> pg_standby is one possible restore_command, right?  I had understood
> that walrestore would be the process that cares for running that
> command, not another implementation of it.

Yes, that was the idea.

> That said, I would really like us to provide a default restore command,
> so if you had any intend of handling the restoring command in the
> walrestore process, by all means, go ahead :)

A different proposal, I think. Not no, just not here and now.

--
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


Re: DeArchiver process

From
Robert Haas
Date:
On Fri, Nov 4, 2011 at 5:15 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> On Fri, Nov 4, 2011 at 2:36 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
>> If we introduce "walrestore" process, pg_standby seems no longer useful.
>> We should get rid of it?
>
> Removing things too quickly can cause problems. There's no harm done
> by keeping it a while longer.
>
> I agree it should go, just want to be absolutely clear that its no
> longer needed for any use case.

I agree that it would be premature to remove pg_standby at this point.But how about changing the default value of
standby_modefrom "off"
 
to "on" in 9.2?  I think most new installations are probably using
that, rather than pg_standby, and changing the default would give
people a gentle push in what now seems to be the preferred direction.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company