Re: GUC_REPORT for protocol tunables was: Re: Optimize binary serialization format of arrays with fixed size elements - Mailing list pgsql-hackers

On Wed, Jan 25, 2012 at 02:50:09PM -0600, Merlin Moncure wrote:
> On Wed, Jan 25, 2012 at 2:29 PM, Marko Kreen <markokr@gmail.com> wrote:
> >> well, I see the following cases:
> >> 1) Vserver > Vapplication: server downgrades wire formats to
> >> applications version
> >> 2) Vapplication > Vlibpq > Vserver: since the application is
> >> reading/writing formats the server can't understand, an error should
> >> be raised if they are used in either direction
> >> 3) Vlibpq >= VApplication > Vserver: same as above, but libpq can
> >> 'upconvert' low version wire format to application's wire format or
> >> error otherwise.
> >
> > I don't see why you special-case libpq here.  There is no reason
> > libpq cannot pass older/newer formats through.  Only thing that
> > matters it parser/formatter version.  If that is done in libpq,
> > then app version does not matter.  If it's done in app, then
> > libpq version does not matter.
> 
> Only because if the app is targeting wire format N, but the server can
> only handle N-1, libpq has the opportunity to fix it up.  That's could
> be just over thinking it though.

I think it's over thinking.  The value should be formatted/parsed just
once.  Server side must support processing different versions.
Whether client side supports downgrading, it's up to client-side
programmers.

If you want to write compatible client, you have a choice of using
proper wrapper API, or simply writing baseline formatting, ignoring
format changes in new versions.

Both are valid approaches and I think we should keep it that way.

> >> By far, the most common cause of problems (both in terms of severity
> >> and frequency) is case #1.  #3 allows a 'compatibility mode' via
> >> libpq, but that comes at significant cost of complexity since libpq
> >> needs to be able to translate wire formats up (but not down).  #2/3 is
> >> a less common problem though as it's more likely the application can
> >> be adjusted to get up to speed: so to keep things simple we can maybe
> >> just error out in those scenarios.
> >
> > I don't like the idea of "conversion".  Instead either client
> > writes values through API that picks format based on server version,
> > or it writes them for specific version only.  In latter case it cannot
> > work with older server.  Unless the fixed version is the baseline.
> 
> ok.  another point about that: libpq isn't really part of the solution
> anyways since there are other popular fully native protocol consumers,
> including (and especially) jdbc, but also python, node.js etc etc.
> 
> that's why I was earlier insisting on a protocol bump, so that we
> could in the new protocol force application version to be advertised.
> v3 would remain caveat emptor for wire formats but v4 would not.

We can bump major/minor anyway to inform clients about new
functionality.  I don't particularly care about that.  What
I'm interested in is what the actual type negotation looks like.

It might be possible we could get away without bumpping anything.
But I have not thought about that angle too deeply yet.

> >> In the database, we need to maintain outdated send/recv functions
> >> basically forever and as much as possible try and translate old wire
> >> format data to and from newer backend structures (maybe in very
> >> specific cases that will be impossible such that the application is
> >> SOL, but that should be rare).  All send/recv functions, including
> >> user created ones need to be stamped with a version token (database
> >> version?).  With the versions of the application, libpq, and all
> >> server functions, we can determine all wire formats as long as we
> >> assume the application's targeted database version represents all the
> >> wire formats it was using.
> >>
> > My good ideas stop there: the exact mechanics of how the usable set of
> >> functions are determined, how exactly the adjusted type look ups will
> >> work, etc. would all have to be sorted out.  Most of the nastier parts
> >> though (protocol changes notwithstanding) are not in libpq, but the
> >> server.  There's just no quick fix on the client side I can see.
> >
> > It does not need to be complex - just bring the version number to
> > i/o function and let it decide whether it cares about it or not.
> > Most functions will not..  Only those that we want to change in
> > compatible manner need to look at it.
> 
> well, maybe instead of passing version number around, the server
> installs the proper compatibility send/recv functions just once on
> session start up so your code isn't littered with stuff like
> if(version > n) do this; else do this;?

Seems confusing.  Note that type i/o functions are user-callable.
How should they act then?

Also note that if()s are needed only for types that want to change their
on-wire formatting.  Considering the mess incompatible on-wire format change
can cause, it's good price to pay.

> > But seriously - on-wire compatibility is good thing, do not fear it...
> 
> sure -- but for postgres I just don't think it's realistic, especially
> for the binary wire formats.  a json based data payload could give it
> to you (and I'm only half kidding) :-).

I think we are pretty compatible already...  Minus the bytea mess.
But mostly thanks to not changing the i/o formats.  The question is
how to keep compatibility while allowing changes in type formatting.

-- 
marko