Thread: pgsql: Have pg_receivexlog always send an invalid log position in statu

pgsql: Have pg_receivexlog always send an invalid log position in statu

From
Magnus Hagander
Date:
Have pg_receivexlog always send an invalid log position in status messages

This prevents pg_basebackup and pg_receivexlog from becoming a synchronous
standby in case 'write' is used for synchronous_commit.

Fujii Masao

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/d7ea9193d18d29bea82b30604cdd087e90c5ee27

Modified Files
--------------
src/bin/pg_basebackup/receivelog.c |    2 +-
1 files changed, 1 insertions(+), 1 deletions(-)


Re: pgsql: Have pg_receivexlog always send an invalid log position in statu

From
Heikki Linnakangas
Date:
On 09.02.2012 15:14, Magnus Hagander wrote:
> Have pg_receivexlog always send an invalid log position in status messages
>
> This prevents pg_basebackup and pg_receivexlog from becoming a synchronous
> standby in case 'write' is used for synchronous_commit.

It's not completely crazy to use pg_receivexlog as a synchronous
standby. It provides the zero-loss property like a real standby does,
ie. if the master dies after sending the WAL to pg_receivexlog, that
transaction is safe in the archive.

--
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com

Re: pgsql: Have pg_receivexlog always send an invalid log position in statu

From
Jaime Casanova
Date:
On Thu, Feb 9, 2012 at 11:35 AM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
> On 09.02.2012 15:14, Magnus Hagander wrote:
>>
>> Have pg_receivexlog always send an invalid log position in status messages
>>
>> This prevents pg_basebackup and pg_receivexlog from becoming a synchronous
>> standby in case 'write' is used for synchronous_commit.
>
>
> It's not completely crazy to use pg_receivexlog as a synchronous standby. It
> provides the zero-loss property like a real standby does, ie. if the master
> dies after sending the WAL to pg_receivexlog, that transaction is safe in
> the archive.
>

what if we ignore the application_name "pg_receivexlog" for a sync rep
and give pg_receivexlog an option to set another application_name?

that will avoid the problem for normal cases and still accept it for a
sync rep if that is what we really want

--
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación

Re: pgsql: Have pg_receivexlog always send an invalid log position in statu

From
Magnus Hagander
Date:
On Thu, Feb 9, 2012 at 17:35, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
> On 09.02.2012 15:14, Magnus Hagander wrote:
>>
>> Have pg_receivexlog always send an invalid log position in status messages
>>
>> This prevents pg_basebackup and pg_receivexlog from becoming a synchronous
>> standby in case 'write' is used for synchronous_commit.
>
>
> It's not completely crazy to use pg_receivexlog as a synchronous standby. It
> provides the zero-loss property like a real standby does, ie. if the master
> dies after sending the WAL to pg_receivexlog, that transaction is safe in
> the archive.

Yes, but as I stated in the email in the thread that the patch was
posted in, I think this should not be the default behaviour, but it
should be available as a commandline option, or something along that
line.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: pgsql: Have pg_receivexlog always send an invalid log position in statu

From
Fujii Masao
Date:
On Fri, Feb 10, 2012 at 8:14 AM, Magnus Hagander <magnus@hagander.net> wrote:
> On Thu, Feb 9, 2012 at 17:35, Heikki Linnakangas
> <heikki.linnakangas@enterprisedb.com> wrote:
>> On 09.02.2012 15:14, Magnus Hagander wrote:
>>>
>>> Have pg_receivexlog always send an invalid log position in status messages
>>>
>>> This prevents pg_basebackup and pg_receivexlog from becoming a synchronous
>>> standby in case 'write' is used for synchronous_commit.
>>
>>
>> It's not completely crazy to use pg_receivexlog as a synchronous standby. It
>> provides the zero-loss property like a real standby does, ie. if the master
>> dies after sending the WAL to pg_receivexlog, that transaction is safe in
>> the archive.
>
> Yes, but as I stated in the email in the thread that the patch was
> posted in, I think this should not be the default behaviour, but it
> should be available as a commandline option, or something along that
> line.

Even if we make that the default behavior, pg_receivexlog doesn't work as
a sync standby unless synchronous_standby_names is set to "pg_receivexlog"
or "*". There is little risk that we make that the default, I think... No?

Anyway, to consider pg_receivexlog as a sync standby, we need to change it
so that its status report includes the valid write and flush
positions, and so that
it replies as soon as it writes or flushes the received WAL, like real
sync standby
does. Otherwise, the master has to wait for the status report interval (which is
specified in -s or --statusint option of pg_receivexlog).

The proposed change would increase the frequency for pg_receivexlog to send
back the report very much. Which might be a problem. For people who want to
avoid such frequent reports, we might need to introduce the option
which specifies
whether frequent report is allowed or not.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: pgsql: Have pg_receivexlog always send an invalid log position in statu

From
Magnus Hagander
Date:
On Fri, Feb 10, 2012 at 04:34, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Fri, Feb 10, 2012 at 8:14 AM, Magnus Hagander <magnus@hagander.net> wrote:
>> On Thu, Feb 9, 2012 at 17:35, Heikki Linnakangas
>> <heikki.linnakangas@enterprisedb.com> wrote:
>>> On 09.02.2012 15:14, Magnus Hagander wrote:
>>>>
>>>> Have pg_receivexlog always send an invalid log position in status messages
>>>>
>>>> This prevents pg_basebackup and pg_receivexlog from becoming a synchronous
>>>> standby in case 'write' is used for synchronous_commit.
>>>
>>>
>>> It's not completely crazy to use pg_receivexlog as a synchronous standby. It
>>> provides the zero-loss property like a real standby does, ie. if the master
>>> dies after sending the WAL to pg_receivexlog, that transaction is safe in
>>> the archive.
>>
>> Yes, but as I stated in the email in the thread that the patch was
>> posted in, I think this should not be the default behaviour, but it
>> should be available as a commandline option, or something along that
>> line.
>
> Even if we make that the default behavior, pg_receivexlog doesn't work as
> a sync standby unless synchronous_standby_names is set to "pg_receivexlog"
> or "*". There is little risk that we make that the default, I think... No?

We discussed this at some previous time, and since it's fairly common
for people to use "*" - in fact, I believe it's what most people do.
Which would lead to unintended consequences. I guess we could document
that very clearly in the docs of that parameter...


> Anyway, to consider pg_receivexlog as a sync standby, we need to change it
> so that its status report includes the valid write and flush
> positions, and so that
> it replies as soon as it writes or flushes the received WAL, like real
> sync standby
> does. Otherwise, the master has to wait for the status report interval (which is
> specified in -s or --statusint option of pg_receivexlog).

Yes, that's what I suggested be controled by a separate parameter.

Having it sync standby and only send status reports every now and then
seems like a really bad idea.


> The proposed change would increase the frequency for pg_receivexlog to send
> back the report very much. Which might be a problem. For people who want to
> avoid such frequent reports, we might need to introduce the option
> which specifies
> whether frequent report is allowed or not.

Exactly my point...


--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: pgsql: Have pg_receivexlog always send an invalid log position in statu

From
Fujii Masao
Date:
On Wed, Feb 15, 2012 at 6:44 PM, Magnus Hagander <magnus@hagander.net> wrote:
> On Fri, Feb 10, 2012 at 04:34, Fujii Masao <masao.fujii@gmail.com> wrote:
>> On Fri, Feb 10, 2012 at 8:14 AM, Magnus Hagander <magnus@hagander.net> wrote:
>>> On Thu, Feb 9, 2012 at 17:35, Heikki Linnakangas
>>> <heikki.linnakangas@enterprisedb.com> wrote:
>>>> On 09.02.2012 15:14, Magnus Hagander wrote:
>>>>>
>>>>> Have pg_receivexlog always send an invalid log position in status messages
>>>>>
>>>>> This prevents pg_basebackup and pg_receivexlog from becoming a synchronous
>>>>> standby in case 'write' is used for synchronous_commit.
>>>>
>>>>
>>>> It's not completely crazy to use pg_receivexlog as a synchronous standby. It
>>>> provides the zero-loss property like a real standby does, ie. if the master
>>>> dies after sending the WAL to pg_receivexlog, that transaction is safe in
>>>> the archive.
>>>
>>> Yes, but as I stated in the email in the thread that the patch was
>>> posted in, I think this should not be the default behaviour, but it
>>> should be available as a commandline option, or something along that
>>> line.
>>
>> Even if we make that the default behavior, pg_receivexlog doesn't work as
>> a sync standby unless synchronous_standby_names is set to "pg_receivexlog"
>> or "*". There is little risk that we make that the default, I think... No?
>
> We discussed this at some previous time, and since it's fairly common
> for people to use "*" - in fact, I believe it's what most people do.
> Which would lead to unintended consequences. I guess we could document
> that very clearly in the docs of that parameter...
>
>
>> Anyway, to consider pg_receivexlog as a sync standby, we need to change it
>> so that its status report includes the valid write and flush
>> positions, and so that
>> it replies as soon as it writes or flushes the received WAL, like real
>> sync standby
>> does. Otherwise, the master has to wait for the status report interval (which is
>> specified in -s or --statusint option of pg_receivexlog).
>
> Yes, that's what I suggested be controled by a separate parameter.
>
> Having it sync standby and only send status reports every now and then
> seems like a really bad idea.
>
>
>> The proposed change would increase the frequency for pg_receivexlog to send
>> back the report very much. Which might be a problem. For people who want to
>> avoid such frequent reports, we might need to introduce the option
>> which specifies
>> whether frequent report is allowed or not.
>
> Exactly my point...

Okay, barring objection, I will make the patch.

But before making the patch, could you commit the patch which fix the incorrect
handling of the timeout in pg_receivexlog? Or I should merge them into
one patch?
http://archives.postgresql.org/pgsql-hackers/2012-02/msg00257.php

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Re: pgsql: Have pg_receivexlog always send an invalid log position in statu

From
Magnus Hagander
Date:
On Wed, Feb 15, 2012 at 18:28, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Wed, Feb 15, 2012 at 6:44 PM, Magnus Hagander <magnus@hagander.net> wrote:
>> On Fri, Feb 10, 2012 at 04:34, Fujii Masao <masao.fujii@gmail.com> wrote:
>>> On Fri, Feb 10, 2012 at 8:14 AM, Magnus Hagander <magnus@hagander.net> wrote:
>>>> On Thu, Feb 9, 2012 at 17:35, Heikki Linnakangas
>>>> <heikki.linnakangas@enterprisedb.com> wrote:
>>>>> On 09.02.2012 15:14, Magnus Hagander wrote:
>>>>>>
>>>>>> Have pg_receivexlog always send an invalid log position in status messages
>>>>>>
>>>>>> This prevents pg_basebackup and pg_receivexlog from becoming a synchronous
>>>>>> standby in case 'write' is used for synchronous_commit.
>>>>>
>>>>>
>>>>> It's not completely crazy to use pg_receivexlog as a synchronous standby. It
>>>>> provides the zero-loss property like a real standby does, ie. if the master
>>>>> dies after sending the WAL to pg_receivexlog, that transaction is safe in
>>>>> the archive.
>>>>
>>>> Yes, but as I stated in the email in the thread that the patch was
>>>> posted in, I think this should not be the default behaviour, but it
>>>> should be available as a commandline option, or something along that
>>>> line.
>>>
>>> Even if we make that the default behavior, pg_receivexlog doesn't work as
>>> a sync standby unless synchronous_standby_names is set to "pg_receivexlog"
>>> or "*". There is little risk that we make that the default, I think... No?
>>
>> We discussed this at some previous time, and since it's fairly common
>> for people to use "*" - in fact, I believe it's what most people do.
>> Which would lead to unintended consequences. I guess we could document
>> that very clearly in the docs of that parameter...
>>
>>
>>> Anyway, to consider pg_receivexlog as a sync standby, we need to change it
>>> so that its status report includes the valid write and flush
>>> positions, and so that
>>> it replies as soon as it writes or flushes the received WAL, like real
>>> sync standby
>>> does. Otherwise, the master has to wait for the status report interval (which is
>>> specified in -s or --statusint option of pg_receivexlog).
>>
>> Yes, that's what I suggested be controled by a separate parameter.
>>
>> Having it sync standby and only send status reports every now and then
>> seems like a really bad idea.
>>
>>
>>> The proposed change would increase the frequency for pg_receivexlog to send
>>> back the report very much. Which might be a problem. For people who want to
>>> avoid such frequent reports, we might need to introduce the option
>>> which specifies
>>> whether frequent report is allowed or not.
>>
>> Exactly my point...
>
> Okay, barring objection, I will make the patch.
>
> But before making the patch, could you commit the patch which fix the incorrect
> handling of the timeout in pg_receivexlog? Or I should merge them into
> one patch?
> http://archives.postgresql.org/pgsql-hackers/2012-02/msg00257.php

(Sorry about the delays)

That one was sitting in my queue of "waiting for feedback". Did you
have a comment on the topic of backwards compatibility raised in that
thread? (and do comment in that thread in that case)

And I think we're better off keeping them as separate patches, if
nothing else that makes the reviews shorter :-)

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/