Thread: Re: [HACKERS] Hooks
David Fetter <david@fetter.org> writes:
> One of our hidden treasures is the hook system, documented only in
> random presentations, if you can find them, and in the source code, if
> you know to look.
> I'd like to document the hooks that we consider public APIs.
The main reason we send people to the source code for that is that
it's often not very clear what the extent of a hook's API is. It would
not be terribly useful to document, say, planner_hook just by listing
its arguments and result type. To do anything useful with that hook
requires a pretty extensive understanding of what the planner does,
and you're not going to get that without a willingness to read source.
So I'm a bit suspicious of this project in the first place, but it's
hard to discuss which hooks should be documented when you haven't
defined what you mean by documentation.
Anyway, there aren't any hooks that weren't put in with the expectation
of third-party code using them, so I'm not following your proposed
distinction between public and private hooks.
regards, tom lane
On Tue, Dec 27, 2016 at 01:32:46PM -0500, Tom Lane wrote: > David Fetter <david@fetter.org> writes: > > One of our hidden treasures is the hook system, documented only in > > random presentations, if you can find them, and in the source code, if > > you know to look. > > > I'd like to document the hooks that we consider public APIs. > > The main reason we send people to the source code for that is that > it's often not very clear what the extent of a hook's API is. It would > not be terribly useful to document, say, planner_hook just by listing > its arguments and result type. Indeed it wouldn't, but there are hooks where the API is, at least to me, a little clearer, and I'll start with those. > To do anything useful with that hook requires a pretty extensive > understanding of what the planner does, and you're not going to get > that without a willingness to read source. I would assume willingness to read source for anything hook related. Documenting the hooks will help with places to look. > So I'm a bit suspicious of this project in the first place, but it's > hard to discuss which hooks should be documented when you haven't > defined what you mean by documentation. I haven't quite come up with that, but I'd pictured a part of the SGML docs that goes over, at a minimum, what all the hooks are and what they do, at least at the level of a sentence or paragraph's worth of description. > Anyway, there aren't any hooks that weren't put in with the expectation > of third-party code using them, so I'm not following your proposed > distinction between public and private hooks. That answers the question I had, which is essentially, "are there any private hooks?" and the answer is no. Best, David. -- David Fetter <david(at)fetter(dot)org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On 12/27/16 1:43 PM, David Fetter wrote: >> So I'm a bit suspicious of this project in the first place, but it's >> hard to discuss which hooks should be documented when you haven't >> defined what you mean by documentation. > I haven't quite come up with that, but I'd pictured a part of the SGML > docs that goes over, at a minimum, what all the hooks are and what > they do, at least at the level of a sentence or paragraph's worth of > description. AFAIK there's no way to get a list of hooks today, short of something like `git grep hook`. I think a simple list of what hooks we have, when they fire and where to find them in code would be sufficient. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532)
From: pgsql-hackers-owner@postgresql.org > [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Jim Nasby > AFAIK there's no way to get a list of hooks today, short of something like > `git grep hook`. I think a simple list of what hooks we have, when they > fire and where to find them in code would be sufficient. How about putting a descriptive comment at the location where each hook variable is defined, using some convention (e.g.like Javadoc-style)? A separate document such as README and wiki can fail to be updated. OTOH, if someone wants toadd a new hook, we can expect him to add appropriate comment by following existing hooks. Using a fixed tag, e.g. "<Hook>",would facilitate finding all hooks. Regards Takayuki Tsunakawa
On Wed, Dec 28, 2016 at 12:47:10AM +0000, Tsunakawa, Takayuki wrote: > From: pgsql-hackers-owner@postgresql.org > > [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Jim Nasby > > AFAIK there's no way to get a list of hooks today, short of > > something like `git grep hook`. I think a simple list of what > > hooks we have, when they fire and where to find them in code would > > be sufficient. > > How about putting a descriptive comment at the location where each > hook variable is defined, using some convention (e.g. like > Javadoc-style)? A separate document such as README and wiki can > fail to be updated. OTOH, if someone wants to add a new hook, we > can expect him to add appropriate comment by following existing > hooks. Using a fixed tag, e.g. "<Hook>", would facilitate finding > all hooks. I like this idea, but it's a much bigger one than mine because it's essentially inventing (or adopting, whatever we settle on) literate programming for the PostgreSQL project. https://en.wikipedia.org/wiki/Literate_programming In the realm of generated documentation, we do have a doxygen https://doxygen.postgresql.org/ for the project, but I haven't really found it helpful thus far. Let's take up literate programming in a separate thread. At the moment, our practice is that (most--hooks being an exception) user-facing features must come with with user-facing docs which are written separately from the source code implementing them. Best, David. -- David Fetter <david(at)fetter(dot)org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
From: David Fetter [mailto:david@fetter.org] > > How about putting a descriptive comment at the location where each > > hook variable is defined, using some convention (e.g. like > > Javadoc-style)? A separate document such as README and wiki can fail > > to be updated. OTOH, if someone wants to add a new hook, we can > > expect him to add appropriate comment by following existing hooks. > > Using a fixed tag, e.g. "<Hook>", would facilitate finding all hooks. > > I like this idea, but it's a much bigger one than mine because it's > essentially inventing (or adopting, whatever we settle on) literate > programming for the PostgreSQL project. > > https://en.wikipedia.org/wiki/Literate_programming I didn't intend to invent a new heavy rule or tool. I just meant comments just like the existing function descriptions,something like /** Hook name: Authentication hook* Description: ...* Arguments: ...* Return value: ...* Note: ...*/ > > In the realm of generated documentation, we do have a doxygen > https://doxygen.postgresql.org/ for the project, but I haven't really found > it helpful thus far. Me, too. > At the moment, our practice is that (most--hooks being an exception) > user-facing features must come with with user-facing docs which are written > separately from the source code implementing them. OK. Anyway, if we can see in the PostgreSQL documentation what hooks are available, it would be the best. I imagine youmeant adding a new chapter under the part "V. Server Programming". Regards Takayuki Tsunakawa
On Wed, Dec 28, 2016 at 01:33:13AM +0000, Tsunakawa, Takayuki wrote: > From: David Fetter [mailto:david@fetter.org] > > > How about putting a descriptive comment at the location where each > > > hook variable is defined, using some convention (e.g. like > > > Javadoc-style)? A separate document such as README and wiki can fail > > > to be updated. OTOH, if someone wants to add a new hook, we can > > > expect him to add appropriate comment by following existing hooks. > > > Using a fixed tag, e.g. "<Hook>", would facilitate finding all hooks. > > > > I like this idea, but it's a much bigger one than mine because it's > > essentially inventing (or adopting, whatever we settle on) literate > > programming for the PostgreSQL project. > > > > https://en.wikipedia.org/wiki/Literate_programming > > I didn't intend to invent a new heavy rule or tool. I just meant comments just like the existing function descriptions,something like > > /* > * Hook name: Authentication hook > * Description: ... > * Arguments: ... > * Return value: ... > * Note: ... > */ It's still new. For the record, I think it's an excellent idea. I see it as larger in scope than mine because it changes how we do things as a project. An example of the kind of thing that this raises is enforcement. Will something (or someone) check that new hooks have this? Will somebody check for comment skew when the APIs change? What happens when somebody forgets? > > At the moment, our practice is that (most--hooks being an > > exception) user-facing features must come with with user-facing > > docs which are written separately from the source code > > implementing them. > > OK. Anyway, if we can see in the PostgreSQL documentation what > hooks are available, it would be the best. I imagine you meant > adding a new chapter under the part "V. Server Programming". Exactly :) Best, David. -- David Fetter <david(at)fetter(dot)org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On 12/27/16 7:41 PM, David Fetter wrote: > I see it as larger in scope than mine because it changes how we do > things as a project. An example of the kind of thing that this raises > is enforcement. Will something (or someone) check that new hooks have > this? Will somebody check for comment skew when the APIs change? > What happens when somebody forgets? Can we reduce the scope of this to a manageable starting point? I'm guessing that all existing hooks share certain characteristics that it'd be pretty easy to detect. If you can detect the hook (which I guess means finding a static variable with hook in the name) then you can verify that there's an appropriate comment block. I'm guessing someone familiar with tools like doxygen could set that up without too much effort, and I'd be surprised if the community had a problem with it. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532)
On 28 December 2016 at 12:15, Jim Nasby <Jim.Nasby@bluetreble.com> wrote: > Can we reduce the scope of this to a manageable starting point? I'm guessing > that all existing hooks share certain characteristics that it'd be pretty > easy to detect. If you can detect the hook (which I guess means finding a > static variable with hook in the name) then you can verify that there's an > appropriate comment block. I'm guessing someone familiar with tools like > doxygen could set that up without too much effort, and I'd be surprised if > the community had a problem with it. Lets just make sure the comment blocks are nice and grep-able too. I think this is a great idea FWIW. Discovering the extension points within Pg isn't easy. Callbacks aren't easy to find either. -- Craig Ringer http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Tue, Dec 27, 2016 at 10:15:55PM -0600, Jim Nasby wrote: > On 12/27/16 7:41 PM, David Fetter wrote: > > I see it as larger in scope than mine because it changes how we do > > things as a project. An example of the kind of thing that this raises > > is enforcement. Will something (or someone) check that new hooks have > > this? Will somebody check for comment skew when the APIs change? > > What happens when somebody forgets? > > Can we reduce the scope of this to a manageable starting point? That is what I'm trying to do. > I'm guessing that all existing hooks share certain characteristics > that it'd be pretty easy to detect. If you can detect the hook > (which I guess means finding a static variable with hook in the > name) then you can verify that there's an appropriate comment block. > I'm guessing someone familiar with tools like doxygen could set that > up without too much effort, and I'd be surprised if the community > had a problem with it. Sure, but that seems like an effort somewhat orthogonal to the one I proposed, which is to get some user-facing i.e. SGML docs up for the current hooks. Here's everything that matches ^\w+_hook$ that I've found so far in git master. There are very likely false positives in this list. ClientAuthentication_hook ExecutorCheckPerms_hook ExecutorEnd_hook ExecutorFinish_hook ExecutorRun_hook ExecutorStart_hook ExplainOneQuery_hook ProcessUtility_hook assign_hook autocommit_hook call_bool_check_hook call_enum_check_hook call_int_check_hook call_real_check_hook call_string_check_hook check_hook check_password_hook comp_keyword_case_hook create_upper_paths_hook echo_hidden_hook echo_hook emit_log_hook explain_get_index_name_hook fetch_count_hook fixed_paramref_hook fmgr_hook get_attavgwidth_hook get_index_stats_hook get_relation_info_hook get_relation_stats_hook histcontrol_hook join_search_hook needs_fmgr_hook next_client_auth_hook next_exec_check_perms_hook object_access_hook on_error_rollback_hook on_error_stop_hook original_client_auth_hook original_post_parse_analyze_hook p_coerce_param_hook p_paramref_hook p_post_columnref_hook p_pre_columnref_hook pgstat_beshutdown_hook planner_hook plpgsql_extra_checks_check_hook plpgsql_extra_errors_assign_hook plpgsql_extra_warnings_assign_hook post_parse_analyze_hook prev_post_parse_analyze_hook prompt1_hook prompt2_hook prompt3_hook quiet_hook sepgsql_audit_hook set_join_pathlist_hook set_rel_pathlist_hook shmem_startup_hook show_context_hook show_hook singleline_hook singlestep_hook variable_coerce_param_hook variable_paramref_hook verbosity_hook Some appear to be client-side, some server-side. I hope that no hook is named other than that way, and I'll do my best to check. Best, David. -- David Fetter <david(at)fetter(dot)org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Wed, Dec 28, 2016 at 2:14 PM, David Fetter <david@fetter.org> wrote: > Here's everything that matches ^\w+_hook$ that I've found so far in > git master. There are very likely false positives in this list. > > [... long list of hooks ...] > > Some appear to be client-side, some server-side. I hope that no hook > is named other than that way, and I'll do my best to check. This list includes some of the hooks of psql used to assign a pset variable. You had better just list the backend-side ones, or if you want just those in src/backend/. Those are the ones that matter to extension and plugin developers. -- Michael
On Wed, Dec 28, 2016 at 03:07:52PM +0900, Michael Paquier wrote: > On Wed, Dec 28, 2016 at 2:14 PM, David Fetter <david@fetter.org> wrote: > > Here's everything that matches ^\w+_hook$ that I've found so far in > > git master. There are very likely false positives in this list. > > > > [... long list of hooks ...] > > > > Some appear to be client-side, some server-side. I hope that no hook > > is named other than that way, and I'll do my best to check. > > This list includes some of the hooks of psql used to assign a pset > variable. You had better just list the backend-side ones, or if you > want just those in src/backend/. Those are the ones that matter to > extension and plugin developers. Thanks for the pointer! Best, David. -- David Fetter <david(at)fetter(dot)org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Wed, Dec 28, 2016 at 12:19:11PM +0800, Craig Ringer wrote: > On 28 December 2016 at 12:15, Jim Nasby <Jim.Nasby@bluetreble.com> wrote: > > > Can we reduce the scope of this to a manageable starting point? > > I'm guessing that all existing hooks share certain characteristics > > that it'd be pretty easy to detect. If you can detect the hook > > (which I guess means finding a static variable with hook in the > > name) then you can verify that there's an appropriate comment > > block. I'm guessing someone familiar with tools like doxygen could > > set that up without too much effort, and I'd be surprised if the > > community had a problem with it. > > Lets just make sure the comment blocks are nice and grep-able too. > > I think this is a great idea FWIW. Discovering the extension points > within Pg isn't easy. > > Callbacks aren't easy to find either. Should callbacks be another chapter in the docs? Best, David. -- David Fetter <david(at)fetter(dot)org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On 12/27/16 11:14 PM, David Fetter wrote: > Sure, but that seems like an effort somewhat orthogonal to the one I > proposed, which is to get some user-facing i.e. SGML docs up for the > current hooks. My point was that a (presumably small) amount of effort towards earmarking hooks in code so that a tool can find them would be a step in the direction of allowing documentation to be generated direct from code, which several people agreed would be very worthwhile. Is that effort *required* to meet your goal? No. Does the journey of 1,000 miles start with a single step? Yes. So if there's a reasonably easy way to start that journey here, please consider doing so. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532)
On 12/28/16 10:43 AM, David Fetter wrote: >> Callbacks aren't easy to find either. > Should callbacks be another chapter in the docs? That would also be nice, but I suspect that will be harder than finding all the hooks. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com 855-TREBLE2 (855-873-2532)