Thread: Backend protocol wanted features

Backend protocol wanted features

From
Vladimir Sitnikov
Date:
Hi,

I've collected "wanted features" page (see [1], [2]).
A dedicated file in pgjdbc seems fine for me. If you know better
location, waive a hand.

If you remember more, or if you want your name with "+1" to appear
there, comment on the PR or file a PR as that one is merged in.

[1]: https://github.com/pgjdbc/pgjdbc/pull/478
[2]: https://github.com/pgjdbc/pgjdbc/blob/protocol_wanted_features/backend_protocol_v4_wanted_features.md

PS. As always, it would be nice if you proofread the file as I am
having troubles with mastering English.

Vladimir


Re: Backend protocol wanted features

From
Álvaro Hernández Tortosa
Date:
On 29/12/15 20:07, Vladimir Sitnikov wrote:
> Hi,
>
> I've collected "wanted features" page (see [1], [2]).
> A dedicated file in pgjdbc seems fine for me. If you know better
> location, waive a hand.
>
> If you remember more, or if you want your name with "+1" to appear
> there, comment on the PR or file a PR as that one is merged in.
>
> [1]: https://github.com/pgjdbc/pgjdbc/pull/478
> [2]: https://github.com/pgjdbc/pgjdbc/blob/protocol_wanted_features/backend_protocol_v4_wanted_features.md
>
> PS. As always, it would be nice if you proofread the file as I am
> having troubles with mastering English.
>
> Vladimir

     Excellent!

     I'd like to reformat, explain and organize my brain dump, but don't
have much time now. Please ping me if anyone needs clarification.

     Regardless, I think it's good to have this wanted features
documented somewhere, and pgjdbc may be a good start, but I don't think
it has to sit there. This is a non-JDBC effort (although we may have a
lot to say) and should involve a lot of people. Maybe the wiki,
complementing and/or replacing the current one
(https://wiki.postgresql.org/wiki/Todo, section 28.2) may make sense.

     Álvaro

--
Álvaro Hernández Tortosa


-----------
8Kdata



Re: Backend protocol wanted features

From
Vladimir Sitnikov
Date:
> Maybe the wiki, complementing and/or replacing the current one

Wiki (at least, GitHub's wiki) is bad for collaboration. One does not
simply send a PR for a wiki page, thus I've created a plain markdown
file.
Vladimir


Re: Backend protocol wanted features

From
Kevin Wooten
Date:
Add “schema change notifications”.

Currently “-ng” uses a bit of magic, mid query, to detect when schema changes have occurred. This is how the binary
typesremain correctly mapped.  A notification sent whenever a type (basically anything in pg_type) is altered in any
waywould solve this problem perfectly. It would be best if the notification included the type oids that changed.  

The infrastructure is already there in both needed respects, notifications & it already knows internally about the
changes)so this should be something that could be easily added to the current protocol. 

> On Dec 29, 2015, at 12:07 PM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>
> Hi,
>
> I've collected "wanted features" page (see [1], [2]).
> A dedicated file in pgjdbc seems fine for me. If you know better
> location, waive a hand.
>
> If you remember more, or if you want your name with "+1" to appear
> there, comment on the PR or file a PR as that one is merged in.
>
> [1]: https://github.com/pgjdbc/pgjdbc/pull/478
> [2]: https://github.com/pgjdbc/pgjdbc/blob/protocol_wanted_features/backend_protocol_v4_wanted_features.md
>
> PS. As always, it would be nice if you proofread the file as I am
> having troubles with mastering English.
>
> Vladimir
>
>
> --
> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-jdbc



Re: Backend protocol wanted features

From
Vladimir Sitnikov
Date:

Re: Backend protocol wanted features

From
Kevin Wooten
Date:
Maybe we could format this wishlist into things that require “changes” to the backend protocol (such that a new version
iscreated) and things that are just additions without changing the protocol.  

Adding features, using the current protocol, would probably be much more attainable in a short term.  There are quite a
fewitems that fall into this category. 

> On Dec 29, 2015, at 12:22 PM, Kevin Wooten <kdubb@me.com> wrote:
>
> Add “schema change notifications”.
>
> Currently “-ng” uses a bit of magic, mid query, to detect when schema changes have occurred. This is how the binary
typesremain correctly mapped.  A notification sent whenever a type (basically anything in pg_type) is altered in any
waywould solve this problem perfectly. It would be best if the notification included the type oids that changed.  
>
> The infrastructure is already there in both needed respects, notifications & it already knows internally about the
changes)so this should be something that could be easily added to the current protocol. 
>
>> On Dec 29, 2015, at 12:07 PM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>>
>> Hi,
>>
>> I've collected "wanted features" page (see [1], [2]).
>> A dedicated file in pgjdbc seems fine for me. If you know better
>> location, waive a hand.
>>
>> If you remember more, or if you want your name with "+1" to appear
>> there, comment on the PR or file a PR as that one is merged in.
>>
>> [1]: https://github.com/pgjdbc/pgjdbc/pull/478
>> [2]: https://github.com/pgjdbc/pgjdbc/blob/protocol_wanted_features/backend_protocol_v4_wanted_features.md
>>
>> PS. As always, it would be nice if you proofread the file as I am
>> having troubles with mastering English.
>>
>> Vladimir
>>
>>
>> --
>> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-jdbc
>
>
>
> --
> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-jdbc



Re: Backend protocol wanted features

From
Vladimir Sitnikov
Date:
Can you provide an example which items require "changes" to backend
protocol and which do not?

Personally, I do not care if it would be named v3.0.1 or v4

I think almost all the features can be implemented on top of current
v3 messages by customizing payload (e.g. protobuf over
NotificationMessage stuff).
Just pick one and I'll elaborate :) Please, do not pick "Uniform
headers (type byte)"

In fact, it is up to backend developers to identify if a new version
of the protocol is required or a new message is required or whatever
is required to meed the requirements.

Vladimir


Re: Backend protocol wanted features

From
Kevin Wooten
Date:
None of my wishlist items are… any of the complications related to items on my whishlist can be mitigated with making
themopt in “set preferred_format=‘binary’” or “set schema_notifications=‘true’”. 

So maybe they all are fairly easily implementable in the current protocol? (although some of Alvaro’s items seem pretty
broad).

> On Dec 29, 2015, at 1:55 PM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>
> Can you provide an example which items require "changes" to backend
> protocol and which do not?
>
> Personally, I do not care if it would be named v3.0.1 or v4
>
> I think almost all the features can be implemented on top of current
> v3 messages by customizing payload (e.g. protobuf over
> NotificationMessage stuff).
> Just pick one and I'll elaborate :) Please, do not pick "Uniform
> headers (type byte)"
>
> In fact, it is up to backend developers to identify if a new version
> of the protocol is required or a new message is required or whatever
> is required to meed the requirements.
>
> Vladimir
>
>
> --
> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-jdbc



Re: Backend protocol wanted features

From
Vladimir Sitnikov
Date:
> So maybe they all are fairly easily implementable in the current protocol?

New messages => new protocol.

For instance "schema_notification" message need to be well-defined,
thus it deserves its own entry in the protocol documentation.
Doesn't it?
Vladimir


Re: Backend protocol wanted features

From
Kevin Wooten
Date:
Ok well if you define as new protocol as any change, regardless of backwards compatibility, then yes.  I would define a
“newprotocol” as something that has breaking changes with a previous version or at the very least a known deviation
fromexisting behavior. 

Extending the protocol with some “well-defined” notifications (using the system that is already well-defined) is not
somethingI would consider a new protocol.  

I guess like you suggested we’re talking about the semantics of a “3.1” versus “4.0”.  I’m looking for mostly “3.1”
typeof stuff. 

> On Dec 29, 2015, at 2:08 PM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>
>> So maybe they all are fairly easily implementable in the current protocol?
>
> New messages => new protocol.
>
> For instance "schema_notification" message need to be well-defined,
> thus it deserves its own entry in the protocol documentation.
> Doesn't it?
> Vladimir



Re: Backend protocol wanted features

From
Dave Cramer
Date:
Well we would be adding messages that current protocol handlers may not know how to deal with which will likely require them to rewrite their code. That being said it would be easier to define a new protocol, even if it is just an extension and then older clients can still deal with the current protocol, and newer clients can request the newer protocol.




On 29 December 2015 at 16:19, Kevin Wooten <kdubb@me.com> wrote:
Ok well if you define as new protocol as any change, regardless of backwards compatibility, then yes.  I would define a “new protocol” as something that has breaking changes with a previous version or at the very least a known deviation from existing behavior.

Extending the protocol with some “well-defined” notifications (using the system that is already well-defined) is not something I would consider a new protocol.

I guess like you suggested we’re talking about the semantics of a “3.1” versus “4.0”.  I’m looking for mostly “3.1” type of stuff.

> On Dec 29, 2015, at 2:08 PM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>
>> So maybe they all are fairly easily implementable in the current protocol?
>
> New messages => new protocol.
>
> For instance "schema_notification" message need to be well-defined,
> thus it deserves its own entry in the protocol documentation.
> Doesn't it?
> Vladimir



--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

Re: Backend protocol wanted features

From
Kevin Wooten
Date:
That was the point of my opt-in suggestion (set schema_notifications=true). Then you aren’t changing anything for clients that work currently.  

Alternatively you could wrap it all into a new protocol and just use a “set protocol_version=4”, or even better, push the protocol version in the startup parameters.

On Dec 29, 2015, at 2:29 PM, Dave Cramer <pg@fastcrypt.com> wrote:

Well we would be adding messages that current protocol handlers may not know how to deal with which will likely require them to rewrite their code. That being said it would be easier to define a new protocol, even if it is just an extension and then older clients can still deal with the current protocol, and newer clients can request the newer protocol.




On 29 December 2015 at 16:19, Kevin Wooten <kdubb@me.com> wrote:
Ok well if you define as new protocol as any change, regardless of backwards compatibility, then yes.  I would define a “new protocol” as something that has breaking changes with a previous version or at the very least a known deviation from existing behavior.

Extending the protocol with some “well-defined” notifications (using the system that is already well-defined) is not something I would consider a new protocol.

I guess like you suggested we’re talking about the semantics of a “3.1” versus “4.0”.  I’m looking for mostly “3.1” type of stuff.

> On Dec 29, 2015, at 2:08 PM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>
>> So maybe they all are fairly easily implementable in the current protocol?
>
> New messages => new protocol.
>
> For instance "schema_notification" message need to be well-defined,
> thus it deserves its own entry in the protocol documentation.
> Doesn't it?
> Vladimir



--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc


Re: Backend protocol wanted features

From
Vladimir Sitnikov
Date:
>Well we would be adding messages that current protocol handlers may not know how to deal with which will likely
requirethem to rewrite their code 

Not sure if backend developers would be happy to support multiple
protocols at the same time.
opt-in approach with "facelifted messages" seems more plausible.
Vladimir


Re: Backend protocol wanted features

From
Kevin Wooten
Date:
Agreed. Opting in to each specific feature wouldn’t be the cleanest but it would mean sticking with a single protocol.
Itwould also mean that existing clients would continue work without modification for the foreseeable future. 

> On Dec 29, 2015, at 2:47 PM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>
>> Well we would be adding messages that current protocol handlers may not know how to deal with which will likely
requirethem to rewrite their code 
>
> Not sure if backend developers would be happy to support multiple
> protocols at the same time.
> opt-in approach with "facelifted messages" seems more plausible.
> Vladimir



Re: Backend protocol wanted features

From
Dave Cramer
Date:
I did float this balloon and the response was "there will be a fairly high bar to adoption". Not that that should stop this effort. Just sayin'


On 29 December 2015 at 16:47, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>Well we would be adding messages that current protocol handlers may not know how to deal with which will likely require them to rewrite their code

Not sure if backend developers would be happy to support multiple
protocols at the same time.
opt-in approach with "facelifted messages" seems more plausible.
Vladimir

Re: Backend protocol wanted features

From
Álvaro Hernández Tortosa
Date:
On 29/12/15 20:22, Kevin Wooten wrote:
> Add “schema change notifications”.

     This is an interesting idea.

     I wonder, however, if a protocol message is the place for it. I
think it belongs to logical decoding (and yes, I know there is no
current support for DDL in logical decoding, but there should be). So I
wouldn't add this specifically to the febe protocol.

     On another topic, I'd look into considering some of the ideas of
the "protocol" designed by the pglogical_output plugin
(http://www.postgresql.org/message-id/CAMsr+YGc6AYKjsCj0Zfz=X4Aczonq1SfQx9C=hUYUN4j2pKwHA@mail.gmail.com)
for possible inclusion as first-class constructs in a new version of the
protocol.

     Álvaro


--
Álvaro Hernández Tortosa


-----------
8Kdata


>
> Currently “-ng” uses a bit of magic, mid query, to detect when schema changes have occurred. This is how the binary
typesremain correctly mapped.  A notification sent whenever a type (basically anything in pg_type) is altered in any
waywould solve this problem perfectly. It would be best if the notification included the type oids that changed. 
>
> The infrastructure is already there in both needed respects, notifications & it already knows internally about the
changes)so this should be something that could be easily added to the current protocol. 
>
>> On Dec 29, 2015, at 12:07 PM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
>>
>> Hi,
>>
>> I've collected "wanted features" page (see [1], [2]).
>> A dedicated file in pgjdbc seems fine for me. If you know better
>> location, waive a hand.
>>
>> If you remember more, or if you want your name with "+1" to appear
>> there, comment on the PR or file a PR as that one is merged in.
>>
>> [1]: https://github.com/pgjdbc/pgjdbc/pull/478
>> [2]: https://github.com/pgjdbc/pgjdbc/blob/protocol_wanted_features/backend_protocol_v4_wanted_features.md
>>
>> PS. As always, it would be nice if you proofread the file as I am
>> having troubles with mastering English.
>>
>> Vladimir
>>
>>
>> --
>> Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-jdbc



Re: Backend protocol wanted features

From
Dave Cramer
Date:



On 2 January 2016 at 15:55, Álvaro Hernández Tortosa <aht@8kdata.com> wrote:

On 29/12/15 20:22, Kevin Wooten wrote:
Add “schema change notifications”.

    This is an interesting idea.

    I wonder, however, if a protocol message is the place for it. I think it belongs to logical decoding (and yes, I know there is no current support for DDL in logical decoding, but there should be). So I wouldn't add this specifically to the febe protocol.

    On another topic, I'd look into considering some of the ideas of the "protocol" designed by the pglogical_output plugin (http://www.postgresql.org/message-id/CAMsr+YGc6AYKjsCj0Zfz=X4Aczonq1SfQx9C=hUYUN4j2pKwHA@mail.gmail.com) for possible inclusion as first-class constructs in a new version of the protocol.

    Álvaro


Putting this into logical decoding doesn't really help us as that would require the driver to connect to the logical decoding stream as well.  

While I understand it is required for LD. The driver needs it for a different reason.


--
Álvaro Hernández Tortosa


-----------
8Kdata



Currently “-ng” uses a bit of magic, mid query, to detect when schema changes have occurred. This is how the binary types remain correctly mapped.  A notification sent whenever a type (basically anything in pg_type) is altered in any way would solve this problem perfectly. It would be best if the notification included the type oids that changed.

The infrastructure is already there in both needed respects, notifications & it already knows internally about the changes) so this should be something that could be easily added to the current protocol.

On Dec 29, 2015, at 12:07 PM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:

Hi,

I've collected "wanted features" page (see [1], [2]).
A dedicated file in pgjdbc seems fine for me. If you know better
location, waive a hand.

If you remember more, or if you want your name with "+1" to appear
there, comment on the PR or file a PR as that one is merged in.

[1]: https://github.com/pgjdbc/pgjdbc/pull/478
[2]: https://github.com/pgjdbc/pgjdbc/blob/protocol_wanted_features/backend_protocol_v4_wanted_features.md

PS. As always, it would be nice if you proofread the file as I am
having troubles with mastering English.

Vladimir


--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc



--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc

Re: Backend protocol wanted features

From
Álvaro Hernández Tortosa
Date:

On 04/01/16 18:26, Dave Cramer wrote:



On 2 January 2016 at 15:55, Álvaro Hernández Tortosa <aht@8kdata.com> wrote:

On 29/12/15 20:22, Kevin Wooten wrote:
Add “schema change notifications”.

    This is an interesting idea.

    I wonder, however, if a protocol message is the place for it. I think it belongs to logical decoding (and yes, I know there is no current support for DDL in logical decoding, but there should be). So I wouldn't add this specifically to the febe protocol.

    On another topic, I'd look into considering some of the ideas of the "protocol" designed by the pglogical_output plugin (http://www.postgresql.org/message-id/CAMsr+YGc6AYKjsCj0Zfz=X4Aczonq1SfQx9C=hUYUN4j2pKwHA@mail.gmail.com) for possible inclusion as first-class constructs in a new version of the protocol.

    Álvaro


Putting this into logical decoding doesn't really help us as that would require the driver to connect to the logical decoding stream as well.  

While I understand it is required for LD. The driver needs it for a different reason.

    I agree. Somehow :)

    I mean: I understand the driver needs this information. 100% agreed there. But adding new protocol messages that "duplicate" the functionality that should belong (IMHO) only to LD, doesn't make sense. Sure, it is a poor fit a LD client for a JDBC driver. But I still think it's the way to go. That was the main starting point for us thinking about Phoebe, you know :)

    So rather than asking everybody to add new messages to the protocol to support this, wouldn't it be better to support LD in the driver?

    Cheers,

    Álvaro


-- 
Álvaro Hernández Tortosa


-----------
8Kdata





--
Álvaro Hernández Tortosa


-----------
8Kdata



Currently “-ng” uses a bit of magic, mid query, to detect when schema changes have occurred. This is how the binary types remain correctly mapped.  A notification sent whenever a type (basically anything in pg_type) is altered in any way would solve this problem perfectly. It would be best if the notification included the type oids that changed.

The infrastructure is already there in both needed respects, notifications & it already knows internally about the changes) so this should be something that could be easily added to the current protocol.

On Dec 29, 2015, at 12:07 PM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:

Hi,

I've collected "wanted features" page (see [1], [2]).
A dedicated file in pgjdbc seems fine for me. If you know better
location, waive a hand.

If you remember more, or if you want your name with "+1" to appear
there, comment on the PR or file a PR as that one is merged in.

[1]: https://github.com/pgjdbc/pgjdbc/pull/478
[2]: https://github.com/pgjdbc/pgjdbc/blob/protocol_wanted_features/backend_protocol_v4_wanted_features.md

PS. As always, it would be nice if you proofread the file as I am
having troubles with mastering English.

Vladimir


--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc



--
Sent via pgsql-jdbc mailing list (pgsql-jdbc@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-jdbc


Re: Backend protocol wanted features

From
Vladimir Sitnikov
Date:
>So rather than asking everybody to add new messages to the protocol to support this, wouldn't it be better to support
LDin the driver? 

Well, it would still require to wrap one's mind around to get that efficient.
You do not like to deallocate all server-prepared statements after
each DDL, do you?
On the other hand, JDBC driver does not know changes to which
tables/views/functions/types would impact statements prepared in
current session, thus JDBC driver has no idea which changes it should
subscribe to.

>wouldn't it be better to support LD in the driver?

That's another question. +1 for supporting LD in the driver (for both
internal and external uses).

Vladimir


Re: Backend protocol wanted features

From
Álvaro Hernández Tortosa
Date:
On 05/01/16 15:35, Vladimir Sitnikov wrote:
>> So rather than asking everybody to add new messages to the protocol to support this, wouldn't it be better to
supportLD in the driver? 
> Well, it would still require to wrap one's mind around to get that efficient.
> You do not like to deallocate all server-prepared statements after
> each DDL, do you?
> On the other hand, JDBC driver does not know changes to which
> tables/views/functions/types would impact statements prepared in
> current session, thus JDBC driver has no idea which changes it should
> subscribe to.

     That's a tough question whether to drop or re-create prepared
statements if they point to database objects that have been modified.
Needless to say, the naive approach is to drop them all and re-create
them when required. Any more clever algorithm than this one would be an
improvements.

     However, I still fail to see how this is related to how to acquire
the knowledge of schema changes. Whether you get it via v4 protocol
messages or LD, either way, you have the same problem with the prepared
statements. And, in any case, if schema changes were to be implemented
as part of the protocol, it would surely be push messages sent
asynchronously. Not that different from consuming LD.

     My point is that I envision strong opposition to add duplicate
functionality. If schema changes could be obtained from LD, I presume
there will be opposition to *also* add it to the protocol just because
it may not be a great fit for the JDBC driver.

>
>> wouldn't it be better to support LD in the driver?
> That's another question. +1 for supporting LD in the driver (for both
> internal and external uses).

     Nice, me too :)

     Álvaro


--
Álvaro Hernández Tortosa


-----------
8Kdata



Re: Backend protocol wanted features

From
Dave Cramer
Date:

On 5 January 2016 at 11:07, Álvaro Hernández Tortosa <aht@8kdata.com> wrote:

On 05/01/16 15:35, Vladimir Sitnikov wrote:
So rather than asking everybody to add new messages to the protocol to support this, wouldn't it be better to support LD in the driver?
Well, it would still require to wrap one's mind around to get that efficient.
You do not like to deallocate all server-prepared statements after
each DDL, do you?
On the other hand, JDBC driver does not know changes to which
tables/views/functions/types would impact statements prepared in
current session, thus JDBC driver has no idea which changes it should
subscribe to.

    That's a tough question whether to drop or re-create prepared statements if they point to database objects that have been modified. Needless to say, the naive approach is to drop them all and re-create them when required. Any more clever algorithm than this one would be an improvements.

    However, I still fail to see how this is related to how to acquire the knowledge of schema changes. Whether you get it via v4 protocol messages or LD, either way, you have the same problem with the prepared statements. And, in any case, if schema changes were to be implemented as part of the protocol, it would surely be push messages sent asynchronously. Not that different from consuming LD.

    My point is that I envision strong opposition to add duplicate functionality. If schema changes could be obtained from LD, I presume there will be opposition to *also* add it to the protocol just because it may not be a great fit for the JDBC driver.

This is a systemic problem that we need to figure out. JDBC is arguably one of the top clients for PostgreSQL, but seems to be treated by a second class citizen. I don't think it should be necessary to make a second connection to the backend just to get schema changes.

Dave

Re: Backend protocol wanted features

From
Álvaro Hernández Tortosa
Date:

On 05/01/16 17:31, Dave Cramer wrote:

On 5 January 2016 at 11:07, Álvaro Hernández Tortosa <aht@8kdata.com> wrote:

On 05/01/16 15:35, Vladimir Sitnikov wrote:
So rather than asking everybody to add new messages to the protocol to support this, wouldn't it be better to support LD in the driver?
Well, it would still require to wrap one's mind around to get that efficient.
You do not like to deallocate all server-prepared statements after
each DDL, do you?
On the other hand, JDBC driver does not know changes to which
tables/views/functions/types would impact statements prepared in
current session, thus JDBC driver has no idea which changes it should
subscribe to.

    That's a tough question whether to drop or re-create prepared statements if they point to database objects that have been modified. Needless to say, the naive approach is to drop them all and re-create them when required. Any more clever algorithm than this one would be an improvements.

    However, I still fail to see how this is related to how to acquire the knowledge of schema changes. Whether you get it via v4 protocol messages or LD, either way, you have the same problem with the prepared statements. And, in any case, if schema changes were to be implemented as part of the protocol, it would surely be push messages sent asynchronously. Not that different from consuming LD.

    My point is that I envision strong opposition to add duplicate functionality. If schema changes could be obtained from LD, I presume there will be opposition to *also* add it to the protocol just because it may not be a great fit for the JDBC driver.

This is a systemic problem that we need to figure out. JDBC is arguably one of the top clients for PostgreSQL, but seems to be treated by a second class citizen. I don't think it should be necessary to make a second connection to the backend just to get schema changes.

    If anything, I am a really strong defender of Java and Postgres. So I would never allow it to be considered a second class citizen. However, I'm unsure pushing "our" requirements to a broader support for everybody *when* that can be obtained by other means, may not be enough justification.

    There are other parts of the protocol that require a separate connection. Those I don't like either, but it's not unheard of. Maybe the solution is different: that LD may be consumed within the same connection (if that would be possible).

    In any case, it won't be me rejecting that messages informing of schema changes, of course not. I just think they belong more to LD anyway :)

    Álvaro

-- 
Álvaro Hernández Tortosa


-----------
8Kdata

Re: Backend protocol wanted features

From
Sehrope Sarkuni
Date:
On Tue, Jan 5, 2016 at 7:45 PM, Álvaro Hernández Tortosa <aht@8kdata.com> wrote:
There are other parts of the protocol that require a separate connection. Those I don't like either, but it's not unheard of. Maybe the solution is different: that LD may be consumed within the same connection (if that would be possible).

Why not use the existing LISTEN/NOTIFY infrastructure for this? If there were a reserved namespace (ex: pg_% or pg_schema_%) then the existing protocol could be used to listen for schema changes.

There would still need to be backend support for publishing those changes but it'd be backwards compatible in both directions; it wouldn't break existing clients and a modern client that issued a LISTEN for those changes could work with an older backend that doesn't support it (it just won't receive any notifications).

Full support for something like this would require well defined message types to indicate what changed. For an initial version it might be easier to just have a "something changed" message without the specifics of which relations where impacted. A client could invalidate all prepared statements associated with the connection.

Just my 2 cents...

Regards,
-- Sehrope Sarkuni
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/