Unfiltered server logs routing via a new elog hook or existing emit_log_hook bypassing log_min_message check - Mailing list pgsql-hackers

From Bharath Rupireddy
Subject Unfiltered server logs routing via a new elog hook or existing emit_log_hook bypassing log_min_message check
Date
Msg-id CALj2ACVg4mtRc5sBYLi7aXR46ZzgUrn6quoK+cPxqZ=aCzpFCQ@mail.gmail.com
Whole thread Raw
Responses Re: Unfiltered server logs routing via a new elog hook or existing emit_log_hook bypassing log_min_message check
List pgsql-hackers
Hi,

Currently the emit_log_hook gets called only for the log messages of
type <= log_min_message i.e when edata->output_to_server is true [1],
which means that I can't use an implementation of emit_log_hook to
just intercept, say, all DEBUGX messages without interrupting the
actual server logs flow and changing the log_min_message. The use case
is this, in production environments, say an issue is occuring every
time or sporadically, to figure out what the issue is or do root cause
analysis, I might need some of the DEBUGX messages (not all of course)
and I may not want to set log_min_message to DEBUGX as it might
overload the server logs (can lead to server out of disk crashes) or
writing to the postgres container console at a higher pace. If I had
postgres elog hook, say emit_unfiltered_log_hook [2], I can basically
write an external module (with a bunch of GUCs say log_level to route,
place to store the logs, even an option to filter logs based on text
say logs containing word 'replication', max disk space that these
routed logs would occupy etc.) implementing emit_unfiltered_log_hook
to just route the interested logs to a cheaper storage (for debugging
purposes), after analysis I can disable the external module and blow
away the routed logs.

In the production environments such a hook and extension will be super
useful IMO. Many times, we would have better debugged issues had there
been certain logs without disturbing the main flow of server logs. We
could've used the existing hook emit_log_hook but that breaks the
existing external modules implementing emit_log_hook, that's why a new
hook emit_unfiltered_log_hook.

Thoughts?

[1]
    if (edata->output_to_server && emit_log_hook)
        (*emit_log_hook) (edata);
[2]
   if (emit_unfiltered_log_hook)
      (*emit_unfiltered_log_hook) (edata);

Regards,
Bharath Rupireddy.



pgsql-hackers by date:

Previous
From: Masahiko Sawada
Date:
Subject: Re: Perform streaming logical transactions by background workers and parallel apply
Next
From: Alvaro Herrera
Date:
Subject: Re: bogus: logical replication rows/cols combinations