Re: On Logging - Mailing list pgsql-hackers

On Mon, Sep 26, 2005 at 01:13:08PM -0400, Christopher Petrilli wrote:
> On 9/26/05, David Fetter <david@fetter.org> wrote:
> > I've run into something that concerns me.  It's pretty much an 8.2
> > issue, but I'm hoping to stimulate some discussion on it.  It's
> > PostgreSQL's log files.  Right now, they're (sometimes just barely
> > ;) human-readable, but they take significant effort to parse.  For
> > example, pqa, a very clever piece of code, is mostly devoted to
> > parsing said files and works only with significant tweaking and
> > restrictions on log file formats in 8.0.
> 
> In a previous life (oh, like 6 months ago), I spent all my time
> working on parsing log files from dozens of different software
> products, and I learned something that made parsing some files
> orders of magnitude easier than others:
> 
>     Always use message codes.

Could you elucidate a bit on this as to how this might affect
PostgreSQL logging?

> Cisco does this, and it helps a lot.  A few other vendors do this,
> and it helps a lot.  While this might seem an old mainframeism,
^^^^^^^^^^^^^^^^^
You say that like it's a *bad* thing.  I think some fruitful
communication is possible and has been missed over the decades between
mainframe people and *n*x people.  The same applies to supercomputing
people and *n*x people, but that's a story for another day.

> it's terribly useful to have something at the beginning that tells
> you what the message is, what it means, and most importantly, how to
> parse the rest.

OK

> I would be happy to help create this catalog, though it's definately
> a big step to implement.  It would also require identifying every
> message that could be generated -- something few open source
> projects do, but it is critical to those of us who have to process
> the output!

Right.  How big a project is this, and what kind of framework would we
need in order assure that new messages come with new message codes?

> > Simple logging is a default that should probably not change, but
> > I'm thinking that for people who want to find something out from
> > the logs, we could see about a kind of plugin architecture which
> > would enable things like:
> >
> > * CSV
> 
> CSV is the best format, ever.  Trivially simple to parse, it
> requires no extra processing so long as you abide by a few extra
> rules, such as escaping.

I agree that it's nice, but seeing as how many smart people have
stubbed their toes on the various incarnations of "CSV," I must
disagree as to its simplicity.

> > * YAML
> 
> Nice, but I think perhaps not the best format for logging.  It's
> more of a configuration file format in my mind, and it requires a
> bit more oompf to parse.  Not going to happen in AWK. :-)

It's not bad for logging, partly because it's a lot fewer bytes than
XML or SGML, but it maintains a structure.  Of course, it's not as
"simple" in some sense as CSV.

> > * Piped logs, as Apache can do
> 
> Useful, but doesn't create any new capabilities, just simplifies
> some of them.  Focus on "new capabilities" first, then added
> functionality if required.

Fair enough :)

> > * DB handle.  I know this one will be controversial.
> 
> I can't imagine why. :-)

Heh

> > 1.  Am I the only one who would wants an option for machine-readable
> > logs?
> 
> Not likely.  I'd love it.  It makes monitoring and reporting easier.

That's where I've run across this :)

Cheers,
D
-- 
David Fetter david@fetter.org http://fetter.org/
phone: +1 510 893 6100   mobile: +1 415 235 3778

Remember to vote!