Re: [BUGS] BUG #7534: walreceiver takes long time to detect n/w breakdown - Mailing list pgsql-hackers

On Fri, Nov 9, 2012 at 3:03 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Thursday, November 08, 2012 10:42 PM Fujii Masao wrote:
>> On Thu, Nov 8, 2012 at 5:53 PM, Amit Kapila <amit.kapila@huawei.com>
>> wrote:
>> > On Thursday, November 08, 2012 2:04 PM Heikki Linnakangas wrote:
>> >> On 19.10.2012 14:42, Amit kapila wrote:
>> >> > On Thursday, October 18, 2012 8:49 PM Fujii Masao wrote:
>> >> >> Before implementing the timeout parameter, I think that it's
>> better
>> >> to change
>> >> >> both pg_basebackup background process and pg_receivexlog so that
>> they
>> >> >> send back the reply message immediately when they receive the
>> >> keepalive
>> >> >> message requesting the reply. Currently, they always ignore such
>> >> keepalive
>> >> >> message, so status interval parameter (-s) in them always must be
>> set
>> >> to
>> >> >> the value less than replication timeout. We can avoid this
>> >> troublesome
>> >> >> parameter setting by introducing the same logic of walreceiver
>> into
>> >> both
>> >> >> pg_basebackup background process and pg_receivexlog.
>> >> >
>> >> > Please find the patch attached to address the modification
>> mentioned
>> >> by you (send immediate reply for keepalive).
>> >> > Both basebackup and pg_receivexlog uses the same function
>> >> ReceiveXLogStream, so single change for both will address the issue.
>> >>
>> >> Thanks, committed this one after shuffling it around the changes I
>> >> committed yesterday. I also updated the docs to not claim that -s
>> option
>> >> is required to avoid timeout disconnects anymore.
>> >
>> > Thank you.
>> > However I think still the issue will not be completely solved.
>> > pg_basebackup/pg_receivexlog can still take long time to
>> > detect network break as they don't have timeout concept. To do that I
>> have
>> > sent one proposal which is mentioned at end of mail chain:
>> > http://archives.postgresql.org/message-
>> id/6C0B27F7206C9E4CA54AE035729E9C3828
>> > 53BBED@szxeml509-mbs
>> >
>> > Do you think there is any need to introduce such mechanism in
>> > pg_basebackup/pg_receivexlog?
>>
>> Are you planning to introduce the timeout mechanism in pg_basebackup
>> main process? Or background process? It's useful to implement both.
>
> By background process, you mean ReceiveXlogStream?
> For both.
>
> I think for background process, it can be done in a way similar to what we
> have done for walreceiver.

Yes.

> But I have some doubts for how to do for main process:
>
> Logic similar to walreceiver can not be used incase network goes down during
> getting other database file from server.
> The reason for the same is to receive the data files PQgetCopyData() is
> called in synchronous mode, so it keeps waiting for infinite time till it
> gets some data.
> In order to solve this issue, I can think of following options:
> 1. Making this call also asynchronous (but now sure about impact of this).

+1

Walreceiver already calls PQgetCopyData() asynchronously. ISTM you can
solve the issue in the similar way to walreceiver's.

> 2. In function pqWait, instead of passing hard-code value -1 (i.e. infinite
> wait), we can send some finite time. This time can be received as command
> line argument
>     from respective utility and set the same in PGconn structure.
>     In order to have timeout value in PGconn, we can have:
>         a. Add new parameter in PGconn to indicate the receive timeout.
>         b. Use the existing parameter connect_timeout for receive timeout
> also but this may lead to confusion.
> 3. Any other better option?
>
> Apart from above issue, there is possibility that if during connect time
> network goes down, then it might hang,  because connect_timeout by default
> will be NULL and connectDBComplete will start waiting inifinitely for
> connection to become successful.
> So shall we have command line argument separately for this also or any other
> way as you suugest.

Yes, I think that we should add something like --conninfo option to
pg_basebackup
and pg_receivexlog. We can easily set not only connect_timeout but also sslmode,
application_name, ... by using such option accepting conninfo string.

>> BTW, IIRC the walsender has no timeout mechanism during sending
>> backup data to pg_basebackup. So it's also useful to implement the
>> timeout mechanism for the walsender during backup.
>
> Yes, its useful, but for walsender the main problem is that it uses blocking
> send call to send the data.
> I have tried using tcp_keepalive settings, but the send call doesn't comeout
> incase of network break.
> The only way I could get it out is:
> change in the corresponding file /proc/sys/net/ipv4/tcp_retries2 by using
> the command
>                         echo "8" > /proc/sys/net/ipv4/tcp_retries2
> As per recommendation, its value should be at-least 8 (equivalent to 100
> sec)
>
> Do you have any idea, how it can be achieved?

What about using pq_putmessage_noblock()?

Regards,

-- 
Fujii Masao