Thread: implementing NOTIFY with message parameter
Greetings everyone, Beginner in PostgreSQL internals, seeking the insight of more experienced. " (At present, the extra field is unused and will always point to an empty string.)" - http://www.postgresql.org/docs/8.0/static/libpq-notify.html regarding PGnotify returned by PQnotifies I would like to implement the functionality that is the idea behind that extra message field: I would like to be able to give NOTIFY an extra string parameter, and I would like that string to end up in that extra field of that PGnotify structure above. I have been using PostgreSQL for different projects in the last 5-6 years, but have never looked under the hood until now. Please guide me. Pointers to previous relevant discussions would also be kindly welcomed. ---------------- Command/syntax: Being unfamiliar with flex/yacc/bison, I chose the simplest route, and added an optional Sconst: opt_notifymessage: Sconst { $$ = $1; } | /*EMPTY*/ { $$ = ""; } ---------------- Storage: In this scenario, we need to store the message with each notification, so notifications would no longer be conceptually unique(relname), but unique(relname,message) instead. As a result, they will need a separate table. I therefore invented pg_notify: pg_notify: NameData relname; int4 senderpid; int4 recipientpid; NameData message; //FIXME: what about type text instead? and as senderpid now has a place in pg_notify, I removed it from pg_listener: pg_listener: NameData relname; int4 listenerpid; ---------------- backend/commands/async.c: Async_Notify - now takes NotifyStmt * as input AtCommit_Notify - scans pg_listener, and for each tuple that has pending notifies for its relname, insertsa tuple into pg_notify ProcessIncomingNotify - now scans pg_notify instead of pg_listener, delivers notifies sent to thatBE PID, and deletes their tuples NotifyMyFrontend - now has got a new parameter, char *message, that it sends to the frontend ---------------- Issues: I don't know much about string internals in PostgreSQL, so I just duplicated code for the string type already used in pg_listener: NameData; but NameData is limited to NAMEDATALEN bytes (64 currently) and for certain applications this might be a limiting factor. It would be nice to have something longer - say NOTIFYMESSAGELEN = 256 bytes? I guess it would be easy to come up with another, size-limited string type, but then again 256 - nevertheless sufficient for my current application, where 64 wouldn't do - is just another arbitrary number, and as such, hard to objectively reason for or against. Is text/bpchar/varlena (with its TOASTedness) viable here and for this purpose? Its capacity would certainly waive that concern. I am also not very proficient in char* and NameData semantics, storage issues and accessors - please scrutinize the code in this respect as well. My modifications leave behavior of LISTEN/NOTIFY mostly unchanged, but they change the structure and behavior of pg_listener. Should we worry about this and try to add code to emulate the old behavior? ----------------- I attached a - never compiled, never tested - patch, as a vehicle to demonstrate my ideas, solicit further discussion and ask for guidance. Thank you in advance. Best Regards, Andras
Andras Kadinger <bandit@surfnonstop.com> writes:
> ... I therefore invented pg_notify:
Please review the archived discussions about reimplementing NOTIFY using
only shared memory, no tables. The pg_listener table causes a number of
performance problems, and I think that getting rid of it should be a top
priority if any work is going to get done in that area. Your proposed
patch would make the performance issues substantially worse by
increasing the volume of dead tuples :-(
Also, any time you are planning new user-visible behavior, it's a good
idea to write the documentation *first*. In particular I'd like to see
a spec for how this looks to a program using libpq: what is the API for
receiving notify messages with arguments? Can an empty-string argument
be distinguished from no argument at all? (Possibly the protocol
prevents that, I'm not sure at the moment.)
regards, tom lane
On Thu, May 12, 2005 at 10:09:23AM -0400, Tom Lane wrote: > Andras Kadinger <bandit@surfnonstop.com> writes: > > ... I therefore invented pg_notify: > > Please review the archived discussions about reimplementing NOTIFY using > only shared memory, no tables. Maybe this can be done using two SLRU areas like we did for multixact. Notifiers create entries, listeners destroy them. Unsolved questions are how would the listeners know that messages were for them, when would messages be destroyed if they are multi-recipient, and what performance issues there are with this approach. This hammer of mine ... nails, more nails! -- Alvaro Herrera (<alvherre[a]surnet.cl>) "Oh, great altar of passive entertainment, bestow upon me thy discordant images at such speed as to render linear thought impossible" (Calvin a la TV)
Alvaro Herrera <alvherre@surnet.cl> writes:
> On Thu, May 12, 2005 at 10:09:23AM -0400, Tom Lane wrote:
>> Please review the archived discussions about reimplementing NOTIFY using
>> only shared memory, no tables.
> Maybe this can be done using two SLRU areas like we did for multixact.
If you are looking for an existing tool to adapt, I think the sinval
messaging mechanism is a FAR better prototype.
regards, tom lane
On Thu, May 12, 2005 at 10:48:50AM -0400, Tom Lane wrote: > Alvaro Herrera <alvherre@surnet.cl> writes: > > On Thu, May 12, 2005 at 10:09:23AM -0400, Tom Lane wrote: > >> Please review the archived discussions about reimplementing NOTIFY using > >> only shared memory, no tables. > > > Maybe this can be done using two SLRU areas like we did for multixact. > > If you are looking for an existing tool to adapt, I think the sinval > messaging mechanism is a FAR better prototype. Hmm. The problem is that it's easy to do something if the sinval queue fills up -- just reset everybody's cache. But you can't just drop all pending notifies if the queue fills up. A possible approach to that problem would be spilling to disk the entries of the idle backends. -- Alvaro Herrera (<alvherre[a]surnet.cl>) "[PostgreSQL] is a great group; in my opinion it is THE best open source development communities in existence anywhere." (Lamar Owen)
Alvaro Herrera <alvherre@surnet.cl> writes:
> On Thu, May 12, 2005 at 10:48:50AM -0400, Tom Lane wrote:
>> If you are looking for an existing tool to adapt, I think the sinval
>> messaging mechanism is a FAR better prototype.
> Hmm. The problem is that it's easy to do something if the sinval queue
> fills up -- just reset everybody's cache. But you can't just drop all
> pending notifies if the queue fills up.
I think you can. See previous discussions.
regards, tom lane