Thread: continuous data from stdin

On Tue, 18 Feb 2003 13:09:45 -0500, ÷ÅÒÅÔÅÎÎÉËÏ× áÌÅËÓÅÊ wrote:

> I think it's a common question, though I couldn't find any information
> aboo= ut it so far. I've got a program that puts its logs into stdout.
> What is th= e best solution to insert these logs into a table as they
> occur?


I think you're going to have to write a program (Perl using DBI would be
my choice) to use this output as input, and do INSERT statements as each
record is received.  (The COPY command in SQL won't do the trick, if
that's what you're thinking of.)

--
Jeff Boes                                      vox 269.226.9550 ext 24
Database Engineer                                     fax 269.349.9076
Nexcerpt, Inc.                                 http://www.nexcerpt.com
           ...Nexcerpt... Extend your Expertise

If you can route the stuff to syslog, consider the modular syslog daemon
(msyslogd). It has a Postgresql output module that puts the log data directly
into a database. I also have modified the module for one specific log stream
to parse out additional fields for entry into the database.

It has been quite robust (I have my syslog->postgres connection up for months
at a time and even then I'm only breaking the connection due to restarting
msyslogd after code tweaking).

Cheers,
Steve

On Tuesday 18 February 2003 10:34 am, Jeff Boes wrote:
> On Tue, 18 Feb 2003 13:09:45 -0500, ÷ÅÒÅÔÅÎÎÉËÏ× áÌÅËÓÅÊ wrote:
> > I think it's a common question, though I couldn't find any information
> > aboo= ut it so far. I've got a program that puts its logs into stdout.
> > What is th= e best solution to insert these logs into a table as they
> > occur?
>
> I think you're going to have to write a program (Perl using DBI would be
> my choice) to use this output as input, and do INSERT statements as each
> record is received.  (The COPY command in SQL won't do the trick, if
> that's what you're thinking of.)

On Tue, 18 Feb 2003 18:09:45 +0000 (UTC),
Alexey.Veretennikov@comline.uu.ru ("???????????? ???????") wrote:

>Hello, everybody!
>
>I think it's a common question, though I couldn't find any information aboo=
>ut it so far. I've got a program that puts its logs into stdout. What is th=
>e best solution to insert these logs into a table as they occur?
>

    It rather depends on the volume of these logs, but a reasonable
first pass, if the volume isn't too big, would be to write a Perl prog
to listen on stdin and write to the database.

    If it was me I'd use tee as part of the pipe so I could have an
alternative record if something didn't work.

    Regards,
          Harry.

On Tue, 18 Feb 2003 18:09:45 +0000 (UTC),
Alexey.Veretennikov@comline.uu.ru ("???????????? ???????") wrote:

>Hello, everybody!
>
>I think it's a common question, though I couldn't find any information aboo=
>ut it so far. I've got a program that puts its logs into stdout. What is th=
>e best solution to insert these logs into a table as they occur?
>

    It rather depends on the volume of these logs, but a reasonable
first pass, if the volume isn't too big, would be to write a Perl prog
to listen on stdin and write to the database.

    If it was me I'd use tee as part of the pipe so I could have an
alternative record if something didn't work.

    Regards,
          Harry.

On Tue, 2003-02-18 at 13:34, Jeff Boes wrote:
> I think you're going to have to write a program (Perl using DBI would be
> my choice) to use this output as input, and do INSERT statements as each
> record is received.  (The COPY command in SQL won't do the trick, if
> that's what you're thinking of.)

Actually, COPY should work fine, AFAICS. Write a client app to open a
connection to PostgreSQL, execute "COPY xxx FROM stdin;", and then push
lines of COPY input over the wire to PostgreSQL using PQputline(), and
end the COPY using PQendcopy().

That assumes you're using libpq -- other interfaces should provide a
similar way to do the same thing.

Cheers,

Neil
--
Neil Conway <neilc@samurai.com> || PGP Key ID: DB3C29FC