Thread: pg_archive_bypass

pg_archive_bypass

From
Dimitri Fontaine
Date:
Hi,

I tend to consider it a bug that there's no known way under windows to
use the same trick as under Unix by using '/usr/bin/true' as your
archive command. And this Unix trick itself does feel like a hack.

Also I'd very much like to be able to recommend (even if not change the
official defaults) to setup wal_level to archive, archive_mode=on and
archive_command=pg_archive_bypass, so that the day you have a HA budget
ain't the day you're going to restart the server to enable the fault
tolerance settings…

So please find attached a very simple "let's see about it" patch to
implement an internal archive_command that just returns true and is
called pg_archive_bypass. It's missing documentation, which I'll provide
if needed (meaning there's some will to consider applying such a patch).

Regards,
--
dim

*** a/src/backend/postmaster/pgarch.c
--- b/src/backend/postmaster/pgarch.c
***************
*** 475,481 **** pgarch_ArchiverCopyLoop(void)
  /*
   * pgarch_archiveXlog
   *
!  * Invokes system(3) to copy one archive file to wherever it should go
   *
   * Returns true if successful
   */
--- 475,485 ----
  /*
   * pgarch_archiveXlog
   *
!  * Invokes system(3) to copy one archive file to wherever it should go, or
!  * just return true if set to the internal command "pg_archive_bypass". That
!  * allows windows users to easily have the "/usr/bin/true", and that allows
!  * for setting up a archiving only when you need it by changing the command
!  * and issuing a reload.
   *
   * Returns true if successful
   */
***************
*** 490,495 **** pgarch_archiveXlog(char *xlog)
--- 494,509 ----
      const char *sp;
      int            rc;

+     /*
+      * If the command is "pg_archive_bypass", just return true
+      */
+     if( strcmp(XLogArchiveCommand, "pg_archive_bypass") == 0 )
+     {
+         ereport(DEBUG3,
+                 (errmsg_internal("bypassing archive command")));
+         return true;
+     }
+
      snprintf(pathname, MAXPGPATH, XLOGDIR "/%s", xlog);

      /*

Re: pg_archive_bypass

From
Simon Riggs
Date:
On Mon, 2010-06-14 at 12:39 +0200, Dimitri Fontaine wrote:

> I tend to consider it a bug that there's no known way under windows to
> use the same trick as under Unix by using '/usr/bin/true' as your
> archive command. And this Unix trick itself does feel like a hack.
> 
> Also I'd very much like to be able to recommend (even if not change the
> official defaults) to setup wal_level to archive, archive_mode=on and
> archive_command=pg_archive_bypass, so that the day you have a HA budget
> ain't the day you're going to restart the server to enable the fault
> tolerance settings…
> 
> So please find attached a very simple "let's see about it" patch to
> implement an internal archive_command that just returns true and is
> called pg_archive_bypass. It's missing documentation, which I'll provide
> if needed (meaning there's some will to consider applying such a patch).

ISTM like a good idea to have a couple of more obvious commands provided
purely as "internal commands". If we do this on restore_command as well,
we can skip pg_archivecleanup completely, for example.

So, not for 9.0.

But I like concept for 9.1. Would need to be coded as some kind of
escape phrase, followed by other command. So not just a quick hack with
this one exception.

Something like
archive_command = 'pg_archive_internal:true'

Wouldn't rush to another patch though, needs agreement first.

-- Simon Riggs           www.2ndQuadrant.com



Re: pg_archive_bypass

From
Heikki Linnakangas
Date:
On 14/06/10 13:39, Dimitri Fontaine wrote:
> I tend to consider it a bug that there's no known way under windows to
> use the same trick as under Unix by using '/usr/bin/true' as your
> archive command. And this Unix trick itself does feel like a hack.
>
> Also I'd very much like to be able to recommend (even if not change the
> official defaults) to setup wal_level to archive, archive_mode=on and
> archive_command=pg_archive_bypass, so that the day you have a HA budget
> ain't the day you're going to restart the server to enable the fault
> tolerance settings…

That particular use case is better handled by making archive_mode 
changeable on-the-fly. Now that we have wal_level as a separate GUC, it 
shouldn't be too hard.

I like internal commands as a solution in general, but it's still in 
search of a problem..

--   Heikki Linnakangas  EnterpriseDB   http://www.enterprisedb.com


Re: pg_archive_bypass

From
Dimitri Fontaine
Date:
Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:
> On 14/06/10 13:39, Dimitri Fontaine wrote:
>> I tend to consider it a bug that there's no known way under windows to
>> use the same trick as under Unix by using '/usr/bin/true' as your
>> archive command. And this Unix trick itself does feel like a hack.
>>
>> Also I'd very much like to be able to recommend (even if not change the
>> official defaults) to setup wal_level to archive, archive_mode=on and
>> archive_command=pg_archive_bypass, so that the day you have a HA budget
>> ain't the day you're going to restart the server to enable the fault
>> tolerance settings…
>
> That particular use case is better handled by making archive_mode changeable
> on-the-fly. Now that we have wal_level as a separate GUC, it shouldn't be
> too hard.

Ok.

> I like internal commands as a solution in general, but it's still in search
> of a problem..

What about /usr/bin/true, or a simple archive where you cp in a given
location (which could happen to be a remote server thanks to unix
network file systems or windows shares), etc. It seems to me those are
existing problem that we solve poorly: let each user face the same
pitfalls (error management).

I also like Simon's approach to have options to internal commands
too. What about this syntax :
 guc_name = "pg_internal function_name arg1 %r ... argn"

Then function_name is any SQL callable function and you have the %x
substitution still happening there for you. So you can mix hard-coded
arguments (some path) and dynamic arguments. And you can script your
commands in PL/WhateverU.

It's more involved a patch, of course, but if we do that and provide
some simple commands in -core, then most installations will only use
that and stop bothering writing scripts to handle their simple
cases. Which is what I'm after.

Regards,
--
dim


Re: pg_archive_bypass

From
Greg Stark
Date:
On Mon, Jun 14, 2010 at 1:55 PM, Dimitri Fontaine
<dfontaine@hi-media.com> wrote:
> What about /usr/bin/true, or a simple archive where you cp in a given
> location (which could happen to be a remote server thanks to unix
> network file systems or windows shares), etc. It seems to me those are
> existing problem that we solve poorly: let each user face the same
> pitfalls (error management).
>

I would like to see the case where the archive is just a mounted
directory accessible through the filesystem be handled internally.
Ideally if I could just specify the archive location on both the
master and slave using the same parameter just containing a filesystem
path then I could have the same configuration on both machines except
for the actual parameter which decides whether they're a master or
slave. That would make me much more confident I've configured both
machines properly. All the nodes would have the same information and
they would be deciding for themselves whether to push or pull archives
based on whether they're the master or slave.

-- 
greg


Re: pg_archive_bypass

From
Bruce Momjian
Date:
Dimitri Fontaine wrote:
> Hi,
>
> I tend to consider it a bug that there's no known way under windows to
> use the same trick as under Unix by using '/usr/bin/true' as your
> archive command. And this Unix trick itself does feel like a hack.
>
> Also I'd very much like to be able to recommend (even if not change the
> official defaults) to setup wal_level to archive, archive_mode=on and
> archive_command=pg_archive_bypass, so that the day you have a HA budget
> ain't the day you're going to restart the server to enable the fault
> tolerance settings?
>
> So please find attached a very simple "let's see about it" patch to
> implement an internal archive_command that just returns true and is
> called pg_archive_bypass. It's missing documentation, which I'll provide
> if needed (meaning there's some will to consider applying such a patch).

Turn out 'REM' acts like /bin/true on Windows.  I have documented that
fact in the attached, applied patch.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + None of us is going to be here forever. +
Index: doc/src/sgml/config.sgml
===================================================================
RCS file: /cvsroot/pgsql/doc/src/sgml/config.sgml,v
retrieving revision 1.287
diff -c -c -r1.287 config.sgml
*** doc/src/sgml/config.sgml    29 Jun 2010 22:29:13 -0000    1.287
--- doc/src/sgml/config.sgml    30 Jun 2010 02:40:40 -0000
***************
*** 1793,1799 ****
          disabled, but the server continues to accumulate WAL segment files in
          the expectation that a command will soon be provided.  Setting
          <varname>archive_command</> to a command that does nothing but
!         return true, e.g. <literal>/bin/true</>, effectively disables
          archiving, but also breaks the chain of WAL files needed for
          archive recovery, so it should only be used in unusual circumstances.
         </para>
--- 1793,1800 ----
          disabled, but the server continues to accumulate WAL segment files in
          the expectation that a command will soon be provided.  Setting
          <varname>archive_command</> to a command that does nothing but
!         return true, e.g. <literal>/bin/true</> (<literal>REM</> on
!         Windows), effectively disables
          archiving, but also breaks the chain of WAL files needed for
          archive recovery, so it should only be used in unusual circumstances.
         </para>

Re: pg_archive_bypass

From
Dimitri Fontaine
Date:
Bruce Momjian <bruce@momjian.us> writes:
> Turn out 'REM' acts like /bin/true on Windows.  I have documented that
> fact in the attached, applied patch.

I guess that kills it for the "pg_archive_bypass" internal command, but
in a good way. Thanks!

Regards,
-- 
Dimitri Fontaine
PostgreSQL DBA, Architecte

Damn, I still baffled that we can escape the internal commands
altogether.