Re: [RFC] CREATE QUEUE (log-only table) for londiste/pgQ ccompatibility - Mailing list pgsql-hackers

On 10/23/2012 06:47 PM, Robert Haas wrote:
> On Wed, Oct 17, 2012 at 4:25 PM, Josh Berkus <josh@agliodbs.com> wrote:
...
>> 3. Double-down on #2 in a multithreaded environment.
>>
>> For an application-level queue, the base functionality is:
>>
>> ADD ITEM
>> READ NEXT (#) ITEM(S)
>> LOCK ITEM
>> DELETE ITEM
>>
>> More sophisticated an useful queues also allow:
>>
>> READ NEXT UNLOCKED ITEM
>> LOCK NEXT UNLOCKED ITEM
>> UPDATE ITEM
>> READ NEXT (#) UNSEEN ITEM(S)
>>
>> The design you describe seems to prohibit pretty much all of the above
>> operations after READ NEXT.  This makes it completely useless as a
>> application-level queue.

By the above logic MVCC "prohibits" UPDATES and DELETES on table data ;)

WAL-only tables/queues "prohobit" none of what you claim above, you just
implement in a (loosely) MVCC way by keeping track of what events are
processed.


>>
>> And, for that matter, if your new queue only accepts INSERTs, why not
>> just improve LISTEN/NOTIFY so that it's readable on replicas?  What does
>> this design buy you that that doesn't?
I get the ability to easily keep track of which events are already acted on
and which are not.

And you really can't fall back on processing LISTEN/NOTIFY - they
come when they come.

For WAL based event stream you only need to track LSN and in case of multiple cooperative consumers (which I think Josh
meantby
 
"multithreaded" above) a small structure to keep track of locking
and event consumption while  The WAL part takes care of consistency,
order and durability.

> I've read the whole thread, but I still don't see that anyone's said
> it better than this, and I agree with these comments.  (I don't find
> them ad hominem, either.)
>
> It's also worth noting that in order to be useful, this feature
> intrinsically requires the logical replication stuff to be committed
> first.
I agree that this feature - at least if implemented as
proposed - does need some underlying features from the Logical
Replication.

Otoh it does not really _need_ to have full logical replication 
integrated - just
having a special WAL type and easy way for your own WAL reader (something
like pg_basebackup cold work well a a sample).

Without WAL-based logical replication I already can do the same
thing in a bit more expensive way by having a before trigger which
logs the insert in Slony/Londiste style event table and then cancels it
on the main table.
> It's not entirely clear that there's not enough time to get
> logical replication committed for 9.3, and the chances of getting any
> follow-on features getting committed as well seems remote.  Besides
> the shortness of the time, I think all experience has shown that it's
> best not to rush into the design of follow-on features before we've
> got the basic feature well nailed down.  This certainly can't be said
> of logical replication at this point.  Andres seems to be making good
> progress and I'm grateful for his work on it, but I think there's a
> lot left to do before that one is in the bag (as I think Andres would
> agree).
>