Thread: Blank archive_command
Hi, Should archive_command being blank when archiving is enabled result in a fatal error? This doesn't even produce a warning when restarting, just an entry in the log when it goes to archive a WAL segment, and finds the archive_command is empty. Is there a valid scenario where someone would have archiving enabled but no archive command? Naturally it will build up WAL until it is corrected, which will result in a less desirable error, and likely at a less convenient time, and to avoid it, someone either has to have checked the logs and noticed this error, or got curious as to why their WAL collection is nearly running out of shelf space. Thom
On Mon, Jan 17, 2022 at 8:14 PM Thom Brown <thom@linux.com> wrote: > > Hi, > > Should archive_command being blank when archiving is enabled result in > a fatal error? This doesn't even produce a warning when restarting, > just an entry in the log when it goes to archive a WAL segment, and > finds the archive_command is empty. > > Is there a valid scenario where someone would have archiving enabled > but no archive command? Naturally it will build up WAL until it is > corrected, which will result in a less desirable error, and likely at > a less convenient time, and to avoid it, someone either has to have > checked the logs and noticed this error, or got curious as to why > their WAL collection is nearly running out of shelf space. Yes, the .ready files under archive_status and wal files under pg_wal directory grow up with archive_mode on but no archive_command. The archiver keeps emitting "archive_mode enabled, yet archive_command is not set" warnings into server logs, maybe this is something that needs to be monitored for. The expectation is to have a good archiving configuration setup in place which updates both archive_command and archive_mode to appropriate values. The server keeps the WAL files from the point when archive_mode is enabled, but not from the point when the archive_command is set. The archive_mode needs postmaster restart whereas archive_command doesn't, if the archive_command too needed a postmaster restart, then we would have failed FATALly if archive_command was empty. But making the archive_command a needs-postmaster-restart class of parameter is not the path we go IMO because avoiding pomaster restarts in production environments is to be avoided whenever possible. An extreme scenario I can think of is if the archive_command is set to empty by a service layer code. Of course, this is something postgres doesn't need to care about. However, a reasonable thing to do is to emit a WARNING or ERROR-out when archive_command is set to null in it's check_archive_command when archive_mode is on? Regards, Bharath Rupireddy.
Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> writes:
> However, a reasonable thing to do is to
> emit a WARNING or ERROR-out when archive_command is set to null in
> it's check_archive_command when archive_mode is on?
We have been burned badly in the past by attempts to do that sort of
thing (ie, make behavior that's conditional on combinations of GUC
settings). There tends to be collateral damage along the lines of
"certain orders of operations stop working". I'm not really eager
to open that can of worms here.
regards, tom lane
On Mon, Jan 17, 2022 at 9:02 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> writes: > > However, a reasonable thing to do is to > > emit a WARNING or ERROR-out when archive_command is set to null in > > it's check_archive_command when archive_mode is on? > > We have been burned badly in the past by attempts to do that sort of > thing (ie, make behavior that's conditional on combinations of GUC > settings). There tends to be collateral damage along the lines of > "certain orders of operations stop working". I'm not really eager > to open that can of worms here. +1 to create any GUC setting dependencies. Let's leave the responsibility of setting appropriate archive_command to the archiving handlers outside postgres. FWIW, having a note in the archive_command GUC definition in the docs might help to some extent. Regards, Bharath Rupireddy.
On Mon, 17 Jan 2022 at 15:25, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote: > > On Mon, Jan 17, 2022 at 8:14 PM Thom Brown <thom@linux.com> wrote: > > > > Hi, > > > > Should archive_command being blank when archiving is enabled result in > > a fatal error? This doesn't even produce a warning when restarting, > > just an entry in the log when it goes to archive a WAL segment, and > > finds the archive_command is empty. > > > > Is there a valid scenario where someone would have archiving enabled > > but no archive command? Naturally it will build up WAL until it is > > corrected, which will result in a less desirable error, and likely at > > a less convenient time, and to avoid it, someone either has to have > > checked the logs and noticed this error, or got curious as to why > > their WAL collection is nearly running out of shelf space. > > Yes, the .ready files under archive_status and wal files under pg_wal > directory grow up with archive_mode on but no archive_command. The > archiver keeps emitting "archive_mode enabled, yet archive_command is > not set" warnings into server logs, maybe this is something that needs > to be monitored for. The expectation is to have a good archiving > configuration setup in place which updates both archive_command and > archive_mode to appropriate values. > > The server keeps the WAL files from the point when archive_mode is > enabled, but not from the point when the archive_command is set. The > archive_mode needs postmaster restart whereas archive_command doesn't, > if the archive_command too needed a postmaster restart, then we would > have failed FATALly if archive_command was empty. But making the > archive_command a needs-postmaster-restart class of parameter is not > the path we go IMO because avoiding pomaster restarts in production > environments is to be avoided whenever possible. Okay, that makes sense. Thanks. I guess people have to be careful with their settings. I was hoping there was one less footgun that could be disarmed. > > An extreme scenario I can think of is if the archive_command is set to > empty by a service layer code. Of course, this is something postgres > doesn't need to care about. However, a reasonable thing to do is to > emit a WARNING or ERROR-out when archive_command is set to null in > it's check_archive_command when archive_mode is on? > > Regards, > Bharath Rupireddy. -- Thom
On Mon, Jan 17, 2022 at 9:05 PM Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote: > > +1 to not create any GUC setting dependencies. Let's leave the > responsibility of setting appropriate archive_command to the archiving > handlers outside postgres. FWIW, having a note in the archive_command > GUC definition in the docs might help to some extent. On further search, the documentation says the following which is enough IMO: This parameter can only be set in the postgresql.conf file or on the server command line. It is ignored unless archive_mode was enabled at server start. If archive_command is an empty string (the default) while archive_mode is enabled, WAL archiving is temporarily disabled, but the server continues to accumulate WAL segment files in the expectation that a command will soon be provided. Setting archive_command to a command that does nothing but return true, e.g., /bin/true (REM on Windows), effectively disables archiving, but also breaks the chain of WAL files needed for archive recovery, so it should only be used in unusual circumstances. https://www.postgresql.org/docs/devel/runtime-config-wal.html#GUC-ARCHIVE-COMMAND Regards, Bharath Rupireddy.
On Mon, Jan 17, 2022 at 10:53 AM Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote: > This parameter can only be set in the postgresql.conf file or on the > server command line. It is ignored unless archive_mode was enabled at > server start. If archive_command is an empty string (the default) > while archive_mode is enabled, WAL archiving is temporarily disabled, > but the server continues to accumulate WAL segment files in the > expectation that a command will soon be provided. Setting > archive_command to a command that does nothing but return true, e.g., > /bin/true (REM on Windows), effectively disables archiving, but also > breaks the chain of WAL files needed for archive recovery, so it > should only be used in unusual circumstances. Yeah, the fact that this has been documented behavior for a long time is a good reason not to get too excited about the possibility of changing it. People are likely using it intentionally. It might be nice to do something about the fact that you can't change archive_mode without a server restart, though. I suspect we had a good reason for that limitation from an engineering perspective, but from a user perspective, it sucks pretty hard. -- Robert Haas EDB: http://www.enterprisedb.com
Robert Haas <robertmhaas@gmail.com> writes:
> It might be nice to do something about the fact that you can't change
> archive_mode without a server restart, though. I suspect we had a good
> reason for that limitation from an engineering perspective, but from a
> user perspective, it sucks pretty hard.
Agreed. I don't recall what the motivation for that was, but
maybe it could be fixed with some more elbow grease.
regards, tom lane