Thread: Don'st start streaming after creating a slot in pg_receivexlog

Hi,

Heikki complained about pg_receivexlog --create-slot starting to stream
in his talk in St. Petersburg. Saying that that makes it a hard to
script feature - and I have to agree. Since that option is new to 9.5 we
can should change that behaviour now if we decide to.

Michael, what do you think?

Greetings,

Andres Freund

On 07/29/2015 10:37 AM, Andres Freund wrote:
> Heikki complained about pg_receivexlog --create-slot starting to stream
> in his talk in St. Petersburg. Saying that that makes it a hard to
> script feature - and I have to agree. Since that option is new to 9.5 we
> can should change that behaviour now if we decide to.

To be clear, I think "pg_receivexlog --create-slot" should only create 
the slot, and exit.

- Heikki

On Wed, Jul 29, 2015 at 4:51 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> On 07/29/2015 10:37 AM, Andres Freund wrote:
>>
>> Heikki complained about pg_receivexlog --create-slot starting to stream
>> in his talk in St. Petersburg. Saying that that makes it a hard to
>> script feature - and I have to agree. Since that option is new to 9.5 we
>> can should change that behaviour now if we decide to.
>
> To be clear, I think "pg_receivexlog --create-slot" should only create the
> slot, and exit.

Even if I would like to make pg_recvlogical and pg_receivexlog behave
as similarly as possible by having an additional switch --start in
pg_receivexlog to control if it starts to stream or not, this ship has
already sailed for backward-compatibility's sake... Then let's just
create the slot and exit().
-- 
Michael

On 2015-07-29 08:54:40 +0100, Simon Riggs wrote:
> --drop-slot seems pointless, since you can just do that with psql
> 
> If we make --create-slot do nothing but add the slot, then that seems
> pointless also

> Would we need to add those options to all commands, when it can be done
> with psql?

They work over the replication protocol, which is handy, because it can
be done with the same user/permissions as the normal pg_receivexlog.

On 07/29/2015 10:58 AM, Michael Paquier wrote:
> On Wed, Jul 29, 2015 at 4:51 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> On 07/29/2015 10:37 AM, Andres Freund wrote:
>>>
>>> Heikki complained about pg_receivexlog --create-slot starting to stream
>>> in his talk in St. Petersburg. Saying that that makes it a hard to
>>> script feature - and I have to agree. Since that option is new to 9.5 we
>>> can should change that behaviour now if we decide to.
>>
>> To be clear, I think "pg_receivexlog --create-slot" should only create the
>> slot, and exit.
>
> Even if I would like to make pg_recvlogical and pg_receivexlog behave
> as similarly as possible by having an additional switch --start in
> pg_receivexlog to control if it starts to stream or not, this ship has
> already sailed for backward-compatibility's sake... Then let's just
> create the slot and exit().

Hmm. pg_receivelogical is basically a debugging tool. I don't think 
anyone will have it integrated into production scripts etc. So maybe we 
could just change it.

I'm not sure I understand the proposal though. If you don't specify 
--create-slot, nor --start, what would the program do? Nothing?

- Heikki

On 2015-07-29 09:23:43 +0100, Simon Riggs wrote:
> Creating a temporary slot goes against the whole concept of slots, so using
> the same id in the same script isn't actually needed, except maybe to
> simplify testing.

The concept of a slot is to reserve resources. I don't see why it's
wrong to reserve resources temporarily.

On Wed, Jul 29, 2015 at 5:02 PM, Heikki Linnakangas wrote:
> Hmm. pg_receivelogical is basically a debugging tool. I don't think anyone
> will have it integrated into production scripts etc. So maybe we could just
> change it.

This sounds good to me as well.

> I'm not sure I understand the proposal though.

In short, I would propose the following:
- Have --create-slot only create a slot, then exit for both
pg_recvlogical and pg_receivexlog.
- Have --drop-slot drop a slot, then exit.
- Remove the --start switch in pg_recvlogical, and let the utility
start streaming by default.
By doing so both utilities will behave similarly with the same set of options.

> If you don't specify
> --create-slot, nor --start, what would the program do? Nothing?

Complain if neither --create-slot nor --drop-slot nor --start are specified:
$ pg_recvlogical -f - --slot toto -d postgres
pg_recvlogical: at least one action needs to be specified
Try "pg_recvlogical --help" for more information.
$ echo $?
1
-- 
Michael

On 2015-07-29 13:53:31 +0100, Simon Riggs wrote:
> It makes more sense to create one new utility to issue replication commands
> than to enhance multiple utility commands to have bizarre looking
> additional features and modes.
> 
> pg_reputil --create-slot
> pg_reputil --drop-slot
> etc

Logical slots behave differently than physical slots and need different
options. So to me there's a fair point in having support for
manipulating logical slot in a tool around logical slots.

Additionally that support was there in 9.4 and I know at least of three
scripts that call pg_recvlogical to create logical slots. Admittedly one
of them is of my own creation.

> though I prefer the idea of using psql and the already existing slot
> functions than doing all of this extra code that needs maintaining.

It's not that uncommon to have replicas only access the primary for
replication type connections. So it seems completely sensible to use the
replication protocol to manage slots. And that you can't really do with
psql.

Andres

On Wed, Jul 29, 2015 at 8:51 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Wed, Jul 29, 2015 at 5:02 PM, Heikki Linnakangas wrote:
>> Hmm. pg_receivelogical is basically a debugging tool. I don't think anyone
>> will have it integrated into production scripts etc. So maybe we could just
>> change it.
>
> This sounds good to me as well.
>
>> I'm not sure I understand the proposal though.
>
> In short, I would propose the following:
> - Have --create-slot only create a slot, then exit for both
> pg_recvlogical and pg_receivexlog.
> - Have --drop-slot drop a slot, then exit.
> - Remove the --start switch in pg_recvlogical, and let the utility
> start streaming by default.
> By doing so both utilities will behave similarly with the same set of options.
>
>> If you don't specify
>> --create-slot, nor --start, what would the program do? Nothing?
>
> Complain if neither --create-slot nor --drop-slot nor --start are specified:
> $ pg_recvlogical -f - --slot toto -d postgres
> pg_recvlogical: at least one action needs to be specified
> Try "pg_recvlogical --help" for more information.
> $ echo $?
> 1

Here is a patch implementing those things. IMO if-not-exists does not
make much sense anymore. Documentation has been shuffled a bit as
well.
--
Michael

On 2015-07-29 22:17:27 +0900, Michael Paquier wrote:
> Here is a patch implementing those things. IMO if-not-exists does not
> make much sense anymore

What? It's rather useful to be able to discern between 'slot was already
there' and 'oops, some error occured'. -1

To me the pg_recvlogical changes are pretty pointless.

Greetings,

Andres Freund

On Wed, Jul 29, 2015 at 10:15 PM, Andres Freund wrote:
> It's not that uncommon to have replicas only access the primary for
> replication type connections. So it seems completely sensible to use the
> replication protocol to manage slots. And that you can't really do with
> psql.

Actually, you can. CREATE_REPLICATION_SLOT is one of the commands that
work with psql, but that's not really practical compared to what
pg_recvlogical and pg_receivexlog can do.
$ psql -d "replication=1"
=# CREATE_REPLICATION_SLOT hoge PHYSICAL;slot_name | consistent_point | snapshot_name | output_plugin
-----------+------------------+---------------+---------------hoge      | 0/0              | null          | null
(1 row)
-- 
Michael

On 2015-07-29 22:21:04 +0900, Michael Paquier wrote:
> On Wed, Jul 29, 2015 at 10:15 PM, Andres Freund wrote:
> > It's not that uncommon to have replicas only access the primary for
> > replication type connections. So it seems completely sensible to use the
> > replication protocol to manage slots. And that you can't really do with
> > psql.
>
> Actually, you can. CREATE_REPLICATION_SLOT is one of the commands that
> work with psql

I do know that, I wrote the bit in the docs explaining that that's
possible...

But using replication connections that way in psql makes requires it to
be invoked in a different way and makes it it impossible to specify the
database name in a normal way etc. You can also run into commands that
give you 'unexpected result status' errors and such. I don't think
that's a good default interface.

On Wed, Jul 29, 2015 at 10:20 PM, Andres Freund <andres@anarazel.de> wrote:
> On 2015-07-29 22:17:27 +0900, Michael Paquier wrote:
>> Here is a patch implementing those things. IMO if-not-exists does not
>> make much sense anymore
>
> What? It's rather useful to be able to discern between 'slot was already
> there' and 'oops, some error occured'. -1

OK, fine.

> To me the pg_recvlogical changes are pretty pointless.

OK, you wrote it after all. I won't insist on it. So, perhaps the
attached is more convincing then? It just changes --create-slot to
leave immediately after creation to address the complain of this
thread.
--
Michael

On 2015-07-30 17:14:16 +0900, Michael Paquier wrote:
>So, perhaps the attached is more convincing then? It just changes
>--create-slot to leave immediately after creation to address the
>complain of this thread.  -- Michael

Pushed that. Thanks!

Andres