Re: [PATCH] pg_basebackup: progress report max once per second - Mailing list pgsql-hackers

From Mika Eloranta
Subject Re: [PATCH] pg_basebackup: progress report max once per second
Date
Msg-id 1389321665.24989.68880809.1A7C69A6@webmail.messagingengine.com
Whole thread Raw
In response to Re: [PATCH] pg_basebackup: progress report max once per second  (Magnus Hagander <magnus@hagander.net>)
List pgsql-hackers
On Thu, Jan 9, 2014, at 20:05, Magnus Hagander wrote:
> On Thu, Nov 14, 2013 at 10:27 AM, Mika Eloranta <mel@ohmu.fi> wrote:
>> On 13 Nov 2013, at 20:51, Mika Eloranta <mel@ohmu.fi> wrote:
>>
>>
>>
> Prevent excessive progress reporting that can grow to gigabytes
>>
> of output with large databases.
>>
>>
>> Same patch as an attachment.
>
> Would it not make more sense to instead store the last number printed, and only print it if the percentage has
changed?AIUI with this patch we still print the same thing on top of itself if it takes >1 second to get 1% further. 
>
> (Except for verbose mode - but if you're asking for verbose mode, you are *asking* to get lots of output) 

Printing out progress periodically is IMHO slightly better as the
interactive user can see that there is some progress (e.g. by pressing
enter for a new empty console line) during a huge basebackup operation.

The original problem I wanted to address was that I had a daemon
watching over the basebackup process and reading its output to make sure
that the basebackup is proceeding properly. It also wrote all the output
to a logfile for postmortem analysis. The log file grew to a very big
size (multiple gigabytes due to the progress prints). With my patch the
log was only a few kilos, without any negative effects. The excessive
progress reporting can also be an issue in an interactive session over a
slow network (mobile), hogging both time and bandwidth.

Cheers,
   Mika



pgsql-hackers by date:

Previous
From: Adrian Klaver
Date:
Subject: Re: Standalone synchronous master
Next
From: Tom Lane
Date:
Subject: Re: integrate pg_upgrade analyze_new_cluster.sh into vacuumdb