Thread: New option in pg_basebackup to exclude pg_log files during base backup
New option in pg_basebackup to exclude pg_log files during base backup
Hi all,
Following the discussion in message id - CAHGQGwFFMOr4EcugWHZpAaPYQbsEKDg66VmJ1rveJ6Z-EgaqAg@mail.gmail.com , I have developed the patch which gives option to user to exclude pg_log directory contents in pg_basebackup.
[Current situation]
During pg_basebackup, all files in pg_log directory will be copied to new backup directory.
[Design]
- Added new non-mandatory option “-S/--skip-log-dir” to pg_basebackup .
- If “skip-log-dir” is specified in pg_basebackup command, then in basebackup, exclude copying log files from standard “pg_log” directory and any other directory specified in Log_directory guc variable. (Still empty folder “pg_log”/$Log_directory will be created)
- In case, pg_log/$Log_directory is symbolic link, then an empty folder will be created
[Advantage]
It gives an option to user to avoid copying of large log files if they doesn’t wish to and hence can save memory space.
Attached the patch.
Thanks & Regards,
Vaishnavi
Fujitsu Australia
Attachment
Re: New option in pg_basebackup to exclude pg_log files during base backup
Hi all,
Following the discussion in message id - CAHGQGwFFMOr4EcugWHZpAaPYQbsEKDg66VmJ1rveJ6Z-EgaqAg@mail.gmail.com , I have developed the patch which gives option to user to exclude pg_log directory contents in pg_basebackup.
[Current situation]
During pg_basebackup, all files in pg_log directory will be copied to new backup directory.
[Design]
- Added new non-mandatory option “-S/--skip-log-dir” to pg_basebackup .
- If “skip-log-dir” is specified in pg_basebackup command, then in basebackup, exclude copying log files from standard “pg_log” directory and any other directory specified in Log_directory guc variable. (Still empty folder “pg_log”/$Log_directory will be created)
- In case, pg_log/$Log_directory is symbolic link, then an empty folder will be created
[Advantage]
It gives an option to user to avoid copying of large log files if they doesn’t wish to and hence can save memory space.
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Re: New option in pg_basebackup to exclude pg_log files during base backup
Magnus Hagander wrote: > While pg_log is definitely the most common one being the default on many > platforms, we'll still be missing other ones. Should we really hardcode it, > or should we somehow derive it from the settings for log_directory instead? > > As a more general discussion, is this something we might want to expose as > a more general facility rather than hardcode it to the log directory only? > And is it perhaps something we'd rather have configured at the server than > specified in pg_basebackup - like a guc saying which directories should > always be excluded from a basebackup? So you don't have to remember it > every time? So it'd be an array, and by default you'd have something like:basebackup_skip_path = $log_directory ? Maybe use it to skip backup labels by default as well.basebackup_skip_path = $log_directory, $backup_label_files -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: New option in pg_basebackup to exclude pg_log files during base backup
Magnus Hagander wrote:So it'd be an array, and by default you'd have something like:
> While pg_log is definitely the most common one being the default on many
> platforms, we'll still be missing other ones. Should we really hardcode it,
> or should we somehow derive it from the settings for log_directory instead?
>
> As a more general discussion, is this something we might want to expose as
> a more general facility rather than hardcode it to the log directory only?
> And is it perhaps something we'd rather have configured at the server than
> specified in pg_basebackup - like a guc saying which directories should
> always be excluded from a basebackup? So you don't have to remember it
> every time?
basebackup_skip_path = $log_directory
?
Maybe use it to skip backup labels by default as well.
basebackup_skip_path = $log_directory, $backup_label_files
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Re: New option in pg_basebackup to exclude pg_log files during base backup
Magnus Hagander wrote: > On Wed, Apr 9, 2014 at 4:55 PM, Alvaro Herrera <alvherre@2ndquadrant.com>wrote: > > So it'd be an array, and by default you'd have something like: > > basebackup_skip_path = $log_directory > > ? > > > > Maybe use it to skip backup labels by default as well. > > basebackup_skip_path = $log_directory, $backup_label_files > > > > I hadn't considered any details, but yes, someting along that line. And > then you could also include arbitrary filenames or directories should you > want. E.g. if you use the data directory to store your torrents or > something. Man, that's a great idea. Database servers have lots of diskspace in that partition, so it should work really well. Thanks! -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: New option in pg_basebackup to exclude pg_log files during base backup
On Thursday, Apr 10,2014 at 1:15Am, Álvaro Herrera wrote: >Magnus Hagander wrote: >>On Wed, Apr 9, 2014 at 4:55 PM, Alvaro Herrera <alvherre@2ndquadrant.com>wrote: >> > So it'd be an array, and by default you'd have something like: >> > basebackup_skip_path = $log_directory ? >> > >> > Maybe use it to skip backup labels by default as well. >> > basebackup_skip_path = $log_directory, $backup_label_files >> > >> >> I hadn't considered any details, but yes, someting along that line. >> And then you could also include arbitrary filenames or directories >> should you want. E.g. if you use the data directory to store your >> torrents or something. >Man, that's a great idea. Database servers have lots of diskspace in that partition, so it should work really well. Thanks! Yes, It sounds like a good idea. I will look into this and start working in sometime. Thanks & Regards, Vaishnavi Fujitsu Australia
Re: New option in pg_basebackup to exclude pg_log files during base backup
On 4/9/14, 10:57 AM, Magnus Hagander wrote: > So it'd be an array, and by default you'd have something like: > basebackup_skip_path = $log_directory > ? > > Maybe use it to skip backup labels by default as well. > basebackup_skip_path = $log_directory, $backup_label_files > > > I hadn't considered any details, but yes, someting along that line. And > then you could also include arbitrary filenames or directories should > you want. What are the use cases for excluding anything else? pg_basebackup ought to have some intelligence about what files are appropriate to include or exclude, depending on what the user is trying to do. It shouldn't become a general file copying tool.