Thread: logfmt and application_context

Hi everyone,

I just release a logfmt log collector for PostgreSQL :
https://pgxn.org/dist/logfmt/1.0.0/ . This works quite well but I have
a few issues I would like to share with hackers.

First, what do you think of having logfmt output along json and CSV ?
PostgreSQL internal syslogger has builtin support for the different
LOG_DESTINATION_*. Thus logfmt does not send log collector headers
using write_syslogger_file or write_pipe_chunks but plain log line with
write_console. Do you have some hint about this ? The consequences ?
How much is it a good bet to write a custom log collector in a shared
preload library ?

Second issue, logfmt provides a guc called
`logfmt.application_context`. The purpose of application_context is the
same as `application_name` but for a more varying value like request
UUID, task ID, etc. What do you think of this ? Would it be cool to
have this GUC in PostgreSQL and available in log_line_prefix ?

Anyway, it's my first attempt at writing C code for PostgreSQL, with
the help of Guillaume LELARGE and Jehan-Guillaume de RORTHAIS and it's
a pleasure ! PostgreSQL C code is very readable. Thanks everyone for
this !

Regards,
Étienne BERSAC
Developer at Dalibo

> On 30 Aug 2023, at 14:36, Étienne BERSAC <etienne.bersac@dalibo.com> wrote:

> ..what do you think of having logfmt output along json and CSV ?

logfmt is widely supported by log ingestion and analysis tools, and have been
for a long enoug time (IMHO) to be called mature, which is good.  Less ideal is
that there is no official formal definition of what logfmt is, some consumers
of it (like Splunk) even support it while not calling it logfmt.  If we add
support for it, can we reasonably expect that what we emit is what consumers of
it assume it will look like?  Given the simplicity of it I think it can be
argued, but I'm far from an expert in this area.

--
Daniel Gustafsson

Hi Daniel,

Thanks for the feedback.

Le mardi 05 septembre 2023 à 11:35 +0200, Daniel Gustafsson a écrit :
> > On 30 Aug 2023, at 14:36, Étienne BERSAC <etienne.bersac@dalibo.com> wrote:
>
> > ..what do you think of having logfmt output along json and CSV ?
>
> Less ideal is
> that there is no official formal definition of what logfmt is [...]  If we add
> support for it, can we reasonably expect that what we emit is what consumers of
> it assume it will look like?

I didn't know logfmt had variation. Do you have a case of
incompatibility ?

Anyway, I think that logfmt will be better handled inside Postgres
rather than in an extension due to limitation in syslogger
extendability. I could send a patch if more people are interested in
this.


What do you think about application_context as a way to render e.g. a
web request UUID to all backend log messages ?


Regards,
Étienne

> On 26 Sep 2023, at 09:56, Étienne BERSAC <etienne.bersac@dalibo.com> wrote:
> Le mardi 05 septembre 2023 à 11:35 +0200, Daniel Gustafsson a écrit :
>>> On 30 Aug 2023, at 14:36, Étienne BERSAC <etienne.bersac@dalibo.com> wrote:
>>
>>> ..what do you think of having logfmt output along json and CSV ?
>>
>> Less ideal is
>> that there is no official formal definition of what logfmt is [...]  If we add
>> support for it, can we reasonably expect that what we emit is what consumers of
>> it assume it will look like?
>
> I didn't know logfmt had variation. Do you have a case of
> incompatibility ?

Like I said upthread, it might be reasonable to assume that the format is
fairly stable, but without a formal definition there is no way of being
certain.  Formats without specifications that become popular tend to diverge,
Markdown being the textbook example.

Being a common format in ingestion tools makes it interesting though, but I
wonder if those tools aren't alreday supporting CSV such that adding logfmt
won't move the compatibility markers much?

--
Daniel Gustafsson

Hi,

Le mercredi 27 septembre 2023 à 10:14 +0200, Daniel Gustafsson a écrit :
> Being a common format in ingestion tools makes it interesting though, but I
> wonder if those tools aren't alreday supporting CSV such that adding logfmt
> won't move the compatibility markers much?

Compared to CSV, logfmt has explicit named fields. This helps tools to
apply generic rules like : ok this is pid, this is timestamp, etc.
without any configuration. Loki and Grafana indexes a subset of known
fields. This is harder to achieve with a bunch a semi-colon separated
values.

Compared to JSON, logfmt is terser and easier for human eyes and
fingers.

This is why I think logfmt for PostgreSQL could be a good option.

Regards,
Étienne