Re: Processing long AND/OR lists - Mailing list pgsql-hackers

From Christopher Browne
Subject Re: Processing long AND/OR lists
Date
Msg-id CAFNqd5X0pRWzzz=56Pee+35de+cZZqRQ=h7mwQ32Hr8yW1jMTQ@mail.gmail.com
Whole thread Raw
In response to Re: Processing long AND/OR lists  (Tom Lane <tgl@sss.pgh.pa.us>)
Responses Re: Processing long AND/OR lists  (Gurjeet Singh <gurjeet@singh.im>)
Re: Processing long AND/OR lists  (Marko Kreen <markokr@gmail.com>)
List pgsql-hackers
This situation falls from a problem that we noticed a mighty long time ago in Slony, where the set of XIDs outstanding gets very large, and, attendant to that, the set of "action id" values by which tuples are being filtered, gets correspondingly large.

It happens when there is a long pause in application of replication data, and is commonly the consequence of setting up replication on a very large data table that takes a long time for the initial data copy.

At the time, Neil Conway observed this query breakage with a query that was roughly 640K in size, from whence fell jokes to the effect, "who would ever need a query larger than 640K?"

The resolution that I introduced at the time was to write a little parser that would recognize sequences of adjacent values and merge them into "BETWEEN A and B" clauses, which would bring the query size back to a reasonable size.

In Slony 2.1, the issue re-emerged because the ordering of the "action id" values was lost; the query had previously been implicitly forcing them into order; we had to add an "ORDER BY" clause, to make the "compressor" work again.
http://git.postgresql.org/gitweb/?p=slony1-engine.git;a=blobdiff;f=src/slon/remote_worker.c;h=b1f48043f8e25b4a74a392b0dbceeae8f3e18c27;hp=7fbf67c16f97cb7c3f209cf3be903ea52c4490a9;hb=c4ac435308a78a2db63bf267d401d842c169e87d;hpb=d4612aab78bac5a9836e3e2425c403878f7091c8

Joking about "640K" aside, it doesn't seem reasonable to expect a truly enormous query as is generated by the broken forms of this logic to turn out happily.  I'd rather fix Slony (as done in the above patch).


pgsql-hackers by date:

Previous
From: Josh Berkus
Date:
Subject: Re: getting rid of freezing
Next
From: Christopher Browne
Date:
Subject: Re: Planning incompatibilities for Postgres 10.0