Thread: Protocol forced to V2 in low-memory conditions?
One of our customers seems to be running into exactly the issue hypothesized about by Tom here:
http://www.postgresql.org/message-id/8040.1314403175@sss.pgh.pa.us
That is, the server is in low-memory conditions, and the client occasionally issues an error saying it can't complete a certain action due to the protocol version:
PG::UnableToSend: function requires at least protocol version 3.0: SELECT ...
The server is 9.2.4, and the client is the Ruby pg gem using a 9.2.4 libpq.
Was the possibility of an inadvertent protocol downgrade addressed as part of that patch? I read through the thread, but it wasn't entirely clear.
http://www.postgresql.org/message-id/8040.1314403175@sss.pgh.pa.us
That is, the server is in low-memory conditions, and the client occasionally issues an error saying it can't complete a certain action due to the protocol version:
PG::UnableToSend: function requires at least protocol version 3.0: SELECT ...
The server is 9.2.4, and the client is the Ruby pg gem using a 9.2.4 libpq.
Was the possibility of an inadvertent protocol downgrade addressed as part of that patch? I read through the thread, but it wasn't entirely clear.
Thanks,
Maciek
Maciek Sakrejda <m.sakrejda@gmail.com> writes: > One of our customers seems to be running into exactly the issue > hypothesized about by Tom here: > http://www.postgresql.org/message-id/8040.1314403175@sss.pgh.pa.us > Was the possibility of an inadvertent protocol downgrade addressed as part > of that patch? I read through the thread, but it wasn't entirely clear. No, a quick look at report_fork_failure_to_client shows it still always sends V2 protocol. We fixed some of the lesser issues discussed in that thread, but I don't think we ever agreed how to deal with this one. I've been thinking of late that it might be time to retire libpq's support for V2 protocol (other than in the specific context of the first error message received while trying to make a connection). If we did that, we'd remove the code path that thinks it should downgrade to V2 protocol, and thus fix this problem by removing code not adding more. However, that doesn't sound like a back-patchable solution, and also it remains unclear whether non-libpq clients such as JDBC have an issue with this. regards, tom lane
On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Maciek Sakrejda <m.sakrejda@gmail.com> writes: >> One of our customers seems to be running into exactly the issue >> hypothesized about by Tom here: >> http://www.postgresql.org/message-id/8040.1314403175@sss.pgh.pa.us >> Was the possibility of an inadvertent protocol downgrade addressed as part >> of that patch? I read through the thread, but it wasn't entirely clear. > > No, a quick look at report_fork_failure_to_client shows it still always > sends V2 protocol. We fixed some of the lesser issues discussed in that > thread, but I don't think we ever agreed how to deal with this one. > > I've been thinking of late that it might be time to retire libpq's > support for V2 protocol (other than in the specific context of the first > error message received while trying to make a connection). If we did > that, we'd remove the code path that thinks it should downgrade to V2 > protocol, and thus fix this problem by removing code not adding more. > > However, that doesn't sound like a back-patchable solution, and also > it remains unclear whether non-libpq clients such as JDBC have an issue > with this. It's probably worth polling for that. I believe the jdbc driver at least has code for it, but I don't know if it's a requirement at this point. If it is, that might be a "10.0 release" feature, in how it would break things :( -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote: > On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Maciek Sakrejda <m.sakrejda@gmail.com> writes: > >> One of our customers seems to be running into exactly the issue > >> hypothesized about by Tom here: > >> http://www.postgresql.org/message-id/8040.1314403175@sss.pgh.pa.us > >> Was the possibility of an inadvertent protocol downgrade addressed as part > >> of that patch? I read through the thread, but it wasn't entirely clear. > > > > No, a quick look at report_fork_failure_to_client shows it still always > > sends V2 protocol. We fixed some of the lesser issues discussed in that > > thread, but I don't think we ever agreed how to deal with this one. > > > > I've been thinking of late that it might be time to retire libpq's > > support for V2 protocol (other than in the specific context of the first > > error message received while trying to make a connection). If we did > > that, we'd remove the code path that thinks it should downgrade to V2 > > protocol, and thus fix this problem by removing code not adding more. > > > > However, that doesn't sound like a back-patchable solution, and also > > it remains unclear whether non-libpq clients such as JDBC have an issue > > with this. > > It's probably worth polling for that. I believe the jdbc driver at > least has code for it, but I don't know if it's a requirement at this > point. Yes, it has code for it and I think it's still used pretty frequently to circumvent prepared statement planning problems (misestimation, indeterminate types). So I think we need convincing reasons to break their usage. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Andres Freund <andres@2ndquadrant.com> writes:
> On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote:
>> On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> I've been thinking of late that it might be time to retire libpq's
>>> support for V2 protocol (other than in the specific context of the first
>>> error message received while trying to make a connection).
>> It's probably worth polling for that. I believe the jdbc driver at
>> least has code for it, but I don't know if it's a requirement at this
>> point.
> Yes, it has code for it and I think it's still used pretty frequently to
> circumvent prepared statement planning problems (misestimation,
> indeterminate types). So I think we need convincing reasons to break
> their usage.
Note that I was proposing removing libpq's support for V2 connections.
Not the backend's.
regards, tom lane
On Tue, Sep 10, 2013 at 3:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Andres Freund <andres@2ndquadrant.com> writes: >> On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote: >>> On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>>> I've been thinking of late that it might be time to retire libpq's >>>> support for V2 protocol (other than in the specific context of the first >>>> error message received while trying to make a connection). > >>> It's probably worth polling for that. I believe the jdbc driver at >>> least has code for it, but I don't know if it's a requirement at this >>> point. > >> Yes, it has code for it and I think it's still used pretty frequently to >> circumvent prepared statement planning problems (misestimation, >> indeterminate types). So I think we need convincing reasons to break >> their usage. > > Note that I was proposing removing libpq's support for V2 connections. > Not the backend's. Oh. I blame the fact that we call the backend site libpq as well :) Anyway. In that case, it seems a lot more reasonable. But definitely not something backpatchable. But it's been a very long time since we had a supported backend version that didn't speak v3. The possible thing to consider there is if there's a common pg fork that uses v2 only, that would then no longer be compatible with the standard libpq. I have no idea if such a thing exists, and I'm not sure we even care if it does, given how far behind they're lagging in that case... -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/
On Tue, Sep 10, 2013 at 9:53 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Andres Freund <andres@2ndquadrant.com> writes: >> On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote: >>> On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>>> I've been thinking of late that it might be time to retire libpq's >>>> support for V2 protocol (other than in the specific context of the first >>>> error message received while trying to make a connection). > >>> It's probably worth polling for that. I believe the jdbc driver at >>> least has code for it, but I don't know if it's a requirement at this >>> point. > >> Yes, it has code for it and I think it's still used pretty frequently to >> circumvent prepared statement planning problems (misestimation, >> indeterminate types). So I think we need convincing reasons to break >> their usage. > > Note that I was proposing removing libpq's support for V2 connections. > Not the backend's. I vote against this. If we remove V2 support from libpq, then we'll have no easy way to test that the backend's support still works. And we've got too many people using V2 to think that it's OK not to have an easy way of testing that. I think the question we ought to be asking is: how can we get widely-used connectors to stop relying on V2 in the first place? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 09/11/2013 02:30 PM, Robert Haas wrote: > On Tue, Sep 10, 2013 at 9:53 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Andres Freund <andres@2ndquadrant.com> writes: >>> On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote: >>>> On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>>>> I've been thinking of late that it might be time to retire libpq's >>>>> support for V2 protocol (other than in the specific context of the first >>>>> error message received while trying to make a connection). >>>> It's probably worth polling for that. I believe the jdbc driver at >>>> least has code for it, but I don't know if it's a requirement at this >>>> point. >>> Yes, it has code for it and I think it's still used pretty frequently to >>> circumvent prepared statement planning problems (misestimation, >>> indeterminate types). So I think we need convincing reasons to break >>> their usage. >> Note that I was proposing removing libpq's support for V2 connections. >> Not the backend's. > I vote against this. If we remove V2 support from libpq, then we'll > have no easy way to test that the backend's support still works. And > we've got too many people using V2 to think that it's OK not to have > an easy way of testing that. I think the question we ought to be > asking is: how can we get widely-used connectors to stop relying on V2 > in the first place? > How is it tested now, and who is doing the testing? cheers andrew
On Tue, Sep 10, 2013 at 03:55:30PM +0200, Magnus Hagander wrote: > On Tue, Sep 10, 2013 at 3:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Andres Freund <andres@2ndquadrant.com> writes: > >> On 2013-09-10 12:31:22 +0200, Magnus Hagander wrote: > >>> On Tue, Sep 10, 2013 at 5:29 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > >>>> I've been thinking of late that it might be time to retire libpq's > >>>> support for V2 protocol (other than in the specific context of the first > >>>> error message received while trying to make a connection). > > > >>> It's probably worth polling for that. I believe the jdbc driver at > >>> least has code for it, but I don't know if it's a requirement at this > >>> point. > > > >> Yes, it has code for it and I think it's still used pretty frequently to > >> circumvent prepared statement planning problems (misestimation, > >> indeterminate types). So I think we need convincing reasons to break > >> their usage. > > > > Note that I was proposing removing libpq's support for V2 connections. > > Not the backend's. > > Oh. I blame the fact that we call the backend site libpq as well :) > > Anyway. In that case, it seems a lot more reasonable. But definitely > not something backpatchable. But it's been a very long time since we > had a supported backend version that didn't speak v3. > > The possible thing to consider there is if there's a common pg fork > that uses v2 only, that would then no longer be compatible with the > standard libpq. I have no idea if such a thing exists, and I'm not > sure we even care if it does, given how far behind they're lagging > in that case... How could we care anyhow? It's not like we have the resources to maintain all our current released versions anyhow, e.g. current support for 8.4 is a good bit less solid than for 9.2. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
Andrew Dunstan <andrew@dunslane.net> writes:
> On 09/11/2013 02:30 PM, Robert Haas wrote:
>> On Tue, Sep 10, 2013 at 9:53 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> Note that I was proposing removing libpq's support for V2 connections.
>>> Not the backend's.
>> I vote against this. If we remove V2 support from libpq, then we'll
>> have no easy way to test that the backend's support still works.
> How is it tested now, and who is doing the testing?
Exactly. The current support in libpq is nigh useless for testing
purposes, because there's no way to activate that code path on command.
It only runs if libpq (thinks it) is connecting to a pre-7.4 backend.
regards, tom lane
On Wed, Sep 11, 2013 at 2:35 PM, Andrew Dunstan <andrew@dunslane.net> wrote: >> I vote against this. If we remove V2 support from libpq, then we'll >> have no easy way to test that the backend's support still works. And >> we've got too many people using V2 to think that it's OK not to have >> an easy way of testing that. I think the question we ought to be >> asking is: how can we get widely-used connectors to stop relying on V2 >> in the first place? > > How is it tested now, and who is doing the testing? I don't know that there's any automated testing, but if someone reports a bug that can only be reproduced using the v2 protocol, the first thing I'm going to try to do is reproduce that using libpq, just as I would with any other connector malfunction. From what's being said here it sounds like I might have to hack libpq a bit to get it to speak the v2 protocol, but how long is that going to take? Less time than setting up an environment that can run whatever crazy connector the client is using and trying to figure out whether the client or the server is to blame, for sure. Don't get me wrong, I think that the v2 protocol should go away, but the real issue is the connectors that are still relying on it as a primary way of talking to the server, not libpq. We could force all of those connectors to update or die by removing server support, and once the server support is gone I don't care about libpq any more. Or maybe there's some friendlier way to wean those connectors off v2. But in the meantime I'm skeptical that removing the code from libpq gets us anywhere. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
[ still catching up on old email ]
I wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>> On 09/11/2013 02:30 PM, Robert Haas wrote:
>>> On Tue, Sep 10, 2013 at 9:53 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>> Note that I was proposing removing libpq's support for V2 connections.
>>>> Not the backend's.
>>> I vote against this. If we remove V2 support from libpq, then we'll
>>> have no easy way to test that the backend's support still works.
>> How is it tested now, and who is doing the testing?
> Exactly. The current support in libpq is nigh useless for testing
> purposes, because there's no way to activate that code path on command.
> It only runs if libpq (thinks it) is connecting to a pre-7.4 backend.
Actually ... there's another way we might deal with this. Suppose we
invent a libpq connection option to specify whether to use v2 or v3
protocol, defaulting to the latter, and then just remove the protocol-
fallback-during-connection code path. If there is anyone out there using
a modern libpq to talk to a pre-7.4 server, they can be told to invoke
the connection option. This gets rid of the unexpected-protocol-downgrade
problem in a reliable manner, and it also gives us a way to test V2 code
paths in both libpq and the backend, which Andrew is correct to finger as
something that goes nearly totally untested right now.
The main objections I can see to this are (1) it wouldn't provide
a back-patchable fix, and (2) it'd be adding more legacy code instead
of removing it. But the other approaches that we've talked about didn't
sound very back-patchable either, so I think only (2) really holds
much water.
regards, tom lane
On Fri, Nov 8, 2013 at 2:23 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > [ still catching up on old email ] > > I wrote: >> Andrew Dunstan <andrew@dunslane.net> writes: >>> On 09/11/2013 02:30 PM, Robert Haas wrote: >>>> On Tue, Sep 10, 2013 at 9:53 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>>>> Note that I was proposing removing libpq's support for V2 connections. >>>>> Not the backend's. > >>>> I vote against this. If we remove V2 support from libpq, then we'll >>>> have no easy way to test that the backend's support still works. > >>> How is it tested now, and who is doing the testing? > >> Exactly. The current support in libpq is nigh useless for testing >> purposes, because there's no way to activate that code path on command. >> It only runs if libpq (thinks it) is connecting to a pre-7.4 backend. > > Actually ... there's another way we might deal with this. Suppose we > invent a libpq connection option to specify whether to use v2 or v3 > protocol, defaulting to the latter, and then just remove the protocol- > fallback-during-connection code path. If there is anyone out there using > a modern libpq to talk to a pre-7.4 server, they can be told to invoke > the connection option. This gets rid of the unexpected-protocol-downgrade > problem in a reliable manner, and it also gives us a way to test V2 code > paths in both libpq and the backend, which Andrew is correct to finger as > something that goes nearly totally untested right now. > > The main objections I can see to this are (1) it wouldn't provide > a back-patchable fix, and (2) it'd be adding more legacy code instead > of removing it. But the other approaches that we've talked about didn't > sound very back-patchable either, so I think only (2) really holds > much water. This approach sounds promising to me. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company