Thread: Using ini file to setup replication

Using ini file to setup replication

From
Samrat Revagade
Date:
Hi,

I was going through the archives and there was a discussion about
using  ini file to setup
replication.(http://www.postgresql.org/message-id/4C9876B4.9020803@enterprisedb.com).
I think if we work on this proposal and separate out the replication
setup from postgresql.conf file then we can provide more granularity
while setting up the replication parameters.
for example, we can set different values of wal_sender_timeout for
each standby sever.

So i think it is good idea to separate out the replication settings
from postgresql.conf file and put into ini file.
Once it is confirmed then we can extend the ini file to support future
developments into replication.
*for example: for failback safe standby.*

Below I have explained how to to use ini file for failback safe stadby setup:
While discussing feature of fail-back safe standby
(CAF8Q-Gy7xa60HwXc0MKajjkWFEbFDWTG=gGyu1KmT+s2xcQ-bw@mail.gmail.com)
We have decided to use the *ini* file  to configure the fail-back safe standby
here is the link for that:
CAD21AoCY2_bQVPzJeY7S77amncCBXfJ+1gpHgGDbULKLAv0t+Q@mail.gmail.com
But there is no strong positive/negative feedback for the concept of
introducing the ini file.please have a look at it and  give feedback.


In todays scenario, In replication we only have the 2 ways of
configuring the standby server
1. Asynchronous standby
2. Synchronous standby

With the patch of failback safe standby we have more granularity in
setting up the  standby servers.
1. Default synchronous standby.(AAA)
2. Default asynchronous standby. (BBB)
3. Synchronous standby and also make same standby as a failback safe
standby.(CCC)
4. Asynchronous standby and also make same standby as a failback safe
standby.(DDD)

In failback safe standby we are thinking to add to more granular
settings of replication parameters for example
1. User can set seperate value for wal_sender_timeout for each server.
2. User can set seperate value of synchronous_transfer for each server.

Consider the scenario where user want to setup the 4 standby servers
as explained above so setting for them will be:--------- ini file -------------------------
[Server]
standby_name = 'AAA'
synchronous_transfer = commit
wal_sender_timeout = 60

[Server]
standby_name = 'BBB'
synchronous_transfer = none
wal_sender_timeout = 40

[Server]
standby_name = 'CCC'
synchronous_transfer = all
wal_sender_timeout = 50

[Server]
standby_name = 'DDD'
synchronous_transfer = data_flush
wal_sender_timeout = 50
--------------------------------------------------------
so setting up such a scenario through postgresql.conf file is
impossible and if we try to do that it will add lot of complexity to
the code.
so use of ini file will be the very good choice in this case.

Thank you ,
Samrat



Re: Using ini file to setup replication

From
Samrat Revagade
Date:
Please find updated hyperlinks:

> Below I have explained how to to use ini file for failback safe stadby setup:
> While discussing feature of fail-back safe standby
> (CAF8Q-Gy7xa60HwXc0MKajjkWFEbFDWTG=gGyu1KmT+s2xcQ-bw@mail.gmail.com)

http://www.postgresql.org/message-id/CAF8Q-Gy7xa60HwXc0MKajjkWFEbFDWTG=gGyu1KmT+s2xcQ-bw@mail.gmail.com

> We have decided to use the *ini* file  to configure the fail-back safe standby
> here is the link for that:
> CAD21AoCY2_bQVPzJeY7S77amncCBXfJ+1gpHgGDbULKLAv0t+Q@mail.gmail.com

http://www.postgresql.org/message-id/CAD21AoCY2_bQVPzJeY7S77amncCBXfJ+1gpHgGDbULKLAv0t+Q@mail.gmail.com

Regards,
Samrat Revgade



Re: Using ini file to setup replication

From
Andres Freund
Date:
On 2013-07-19 14:54:16 +0530, Samrat Revagade wrote:
> I was going through the archives and there was a discussion about
> using  ini file to setup
> replication.(http://www.postgresql.org/message-id/4C9876B4.9020803@enterprisedb.com).
> I think if we work on this proposal and separate out the replication
> setup from postgresql.conf file then we can provide more granularity
> while setting up the replication parameters.
> for example, we can set different values of wal_sender_timeout for
> each standby sever.
> 
> So i think it is good idea to separate out the replication settings
> from postgresql.conf file and put into ini file.
> Once it is confirmed then we can extend the ini file to support future
> developments into replication.
> *for example: for failback safe standby.*

I think that introducing another configuration format is a pretty bad
idea. While something new might not turn out to be as bad, we've seen
how annoying a separate configuration format turned out for
recovery.conf.

I'd much rather go ahead and remove the nesting limit of GUCs. That
should give us just about all that can be achieved with an ini file with
a 1 line change. Sometime we might want to extend our format to add ini
like sections but I think that *definitely* should be a separate
proposal.

I've even proposed that in the past in
20130225211533.GD3849@awork2.anarazel.de . I plan to propose an updated
version of that patch (not allowing numeric 2nd level ids) for the next
CF.

So you can just do stuff like:

server.foo.grand_unified_config = value.

Greetings,

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services



Re: Using ini file to setup replication

From
Samrat Revagade
Date:
>> *for example: for failback safe standby.*

>I think that introducing another configuration format is a pretty bad
>idea. While something new might not turn out to be as bad, we've seen
>how annoying a separate configuration format turned out for
>recovery.conf.

Its not totally different way of configuration.
ini file will be parsed in the same way as postgresql.conf.
just want to separate out the replication parameters, to make simpler
configuration for future developments in the field of replication such
as failback-safe standby.

> So you can just do stuff like:
>
> server.foo.grand_unified_config = value.

But according to your approach and considering the use case of
failback safe standby
the parameters into the postgresql.conf will vary dynamically, and i
don't think so doing this in the postgresql.conf is a good idea
because it already contains whole bunch of parameters:

for example:
if i want to configure 2 servers then it will add 6 lines,for 3 -9, for 4-12
setting's for particular server will be:

considering the way of setting value to conf parameters  : guc . value

standby_name.'AAA'
synchronous_transfer. commit
wal_sender_timeout.60


Regards,
Samrat
Samrat Revgade



Re: Using ini file to setup replication

From
Josh Berkus
Date:
> I've even proposed that in the past in
> 20130225211533.GD3849@awork2.anarazel.de . I plan to propose an updated
> version of that patch (not allowing numeric 2nd level ids) for the next
> CF.
> 
> So you can just do stuff like:
> 
> server.foo.grand_unified_config = value.

Sounds good to me.  I can see lots of uses for that.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com



Re: Using ini file to setup replication

From
Sawada Masahiko
Date:
On Fri, Jul 19, 2013 at 6:53 PM, Andres Freund <andres@2ndquadrant.com> wrote:
> So you can just do stuff like:
>
> server.foo.grand_unified_config = value.
>
it looks good to me too. when server parse values which is written in
postgresql.conf, server handles those parameter as item list value.
after that, those parameter overwrite to corresponding values. so, we
can allocate area of those values dynamically when server first parse
those values. (of cause we should count number of servers)
but I think that when server setting value is no set in order by
server name, parsing is a little difficult. we should check whether
server name has already registered at all time.

for example :
---- postgresql.conf -----
server.AAA.wal_sender_timeout = 60
server.BBB.synchronous_transfer = data_flush
server.AAA.synchronous_transfer = all
---- postgresql.conf -----
what do you think?

and if the parameter(e.g., wal_sender_timeout)  is not set in
configure file(or postgresql,conf) like above, what value should we
set value to parameter? and how we handle originally
wal_sender_timeout? will it leave? or delete?

Regards,

-------
Sawada Masahiko



Re: Using ini file to setup replication

From
Greg Stark
Date:
On Fri, Jul 19, 2013 at 2:20 PM, Samrat Revagade
<revagade.samrat@gmail.com> wrote:

> for example:
> if i want to configure 2 servers then it will add 6 lines,for 3 -9, for 4-12
> setting's for particular server will be:
>
> considering the way of setting value to conf parameters  : guc . value
>
> standby_name.'AAA'
> synchronous_transfer. commit
> wal_sender_timeout.60


I have a feeling Samrat and Sawada-san have some good use cases where
this extra syntax could be a big step up in expressiveness. But I'm
having a hard time figuring out exactly what they have in mind. If
either of you could explain in more detail how the extra syntax would
apply to your use case and how it would let you do something that you
can't already do it might be helpful.

I'm assuming the idea is something like having a single config file
which can work for the server regardless of whether it's acting as the
primary or standby and then be able to switch roles by switching a
single flag which would control which parts of the config file were
applied. But I'm having a hard time seeing how exactly that would
work.
-- 
greg



Re: Using ini file to setup replication

From
Sawada Masahiko
Date:
On Mon, Jul 22, 2013 at 10:59 PM, Greg Stark <stark@mit.edu> wrote:
> On Fri, Jul 19, 2013 at 2:20 PM, Samrat Revagade
> <revagade.samrat@gmail.com> wrote:
>
>> for example:
>> if i want to configure 2 servers then it will add 6 lines,for 3 -9, for 4-12
>> setting's for particular server will be:
>>
>> considering the way of setting value to conf parameters  : guc . value
>>
>> standby_name.'AAA'
>> synchronous_transfer. commit
>> wal_sender_timeout.60
>
>
> I have a feeling Samrat and Sawada-san have some good use cases where
> this extra syntax could be a big step up in expressiveness. But I'm
> having a hard time figuring out exactly what they have in mind. If
> either of you could explain in more detail how the extra syntax would
> apply to your use case and how it would let you do something that you
> can't already do it might be helpful.
>
> I'm assuming the idea is something like having a single config file
> which can work for the server regardless of whether it's acting as the
> primary or standby and then be able to switch roles by switching a
> single flag which would control which parts of the config file were
> applied. But I'm having a hard time seeing how exactly that would
> work.
in this approach which GUC parameters is written in postgresql.conf,
user would write many extra line in postgresql.conf by a standby
server as Samrat suggested. It will increase size of postgresql.conf.
I think it is not good that all those parameters are written in
postgresql.conf.
that is, I think that those parameters should be written in separately
file. e.g., we can set separately any parameter using "include" (or
"include if exist") in postgresql.conf.
if include file doesn't exist, we would set default value to each wal
sender. that is, we give up ini file, and we provide mechanism of
setting to each wal sender as option of overwrite.
of course to support this approach, it needs to use the patch which
Andres suggested, and server should be able to handle toke which is
two or mote separated by a dot. so we would like to help this
approach.

Regards,

----
Sawada Masahiko