Thread: apache logs to pgsql
In the contrib directory, there is an example on how to redirect apache logs to a postgresql database. It says to create a table and change two lines in the apache configuration file to use psql for copying the logs to the database. My question is: using an sql db is mostly compelling for very active sites but, if this is the case, is it sensible to use psql? First, I find that using the psql command could have too much overhead. Seeing the executable is 46696 bytes and dynamically linked, I'm not really sure what is the strain on the system for bringing up psql. Second, I have tried to see the alternative of writing an apache module or, more sensibly, changing the current module responsible for logging. After reading mod_log_config.c, I'm not really sure how I can integrate libpq and avoid having to connect and disconnect repeatedly. If I can't somehow sustain a connection, I imagine the module would be just as efficient as using psql. Lastly, I'm confused and overwhelmed by the size of the task at hand. I'd appreciate any suggestions or recommendations or thoughts on whether writing a module is even worthwhile at all. Thanks, Marc
Hi, I am currently using a less elegant -but efficient- solution which is to use the standard Apache log module and to post process the log files into postgreSQL tables. The advantage of this method is that you can perform this post processing on another server, removing all the strain from the web server. If anyone is interested, I can contribute the scripts I have developed. They are tailored to my specific needs but could easily be adapted. Hope this helps. Eric Marc Tardif wrote: > > In the contrib directory, there is an example on how to redirect apache > logs to a postgresql database. It says to create a table and change two > lines in the apache configuration file to use psql for copying the > logs to the database. > > My question is: using an sql db is mostly compelling for very active > sites but, if this is the case, is it sensible to use psql? > > First, I find that using the psql command could have too much overhead. > Seeing the executable is 46696 bytes and dynamically linked, I'm not > really sure what is the strain on the system for bringing up psql. > > Second, I have tried to see the alternative of writing an apache module > or, more sensibly, changing the current module responsible for logging. > After reading mod_log_config.c, I'm not really sure how I can integrate > libpq and avoid having to connect and disconnect repeatedly. If I can't > somehow sustain a connection, I imagine the module would be just as > efficient as using psql. > > Lastly, I'm confused and overwhelmed by the size of the task at hand. I'd > appreciate any suggestions or recommendations or thoughts on whether > writing a module is even worthwhile at all. > > Thanks, > Marc > > ************ -- ------------------------------------------------------------------------ Eric van der Vlist Dyomedea http://www.dyomedea.com http://www.ducotede.com ------------------------------------------------------------------------
On Thu, 20 Jan 2000, Marc Tardif wrote: > In the contrib directory, there is an example on how to redirect apache > logs to a postgresql database. It says to create a table and change two > lines in the apache configuration file to use psql for copying the > logs to the database. > > My question is: using an sql db is mostly compelling for very active > sites but, if this is the case, is it sensible to use psql? That's definitely overkill and possibly too slow on busy sites. What you might want to do is write a "psql light" that simply takes strings on the standard input and sends them to the backend. You'd still have to link that against libpq but you might find it to be more efficient. > First, I find that using the psql command could have too much overhead. > Seeing the executable is 46696 bytes and dynamically linked, I'm not > really sure what is the strain on the system for bringing up psql. Not to mention the fact that every time you start it up psql does a whole bunch of initializing, and every line of input you pass it is first parsed before deciding what to do with it. -- Peter Eisentraut Sernanders vaeg 10:115 peter_e@gmx.net 75262 Uppsala http://yi.org/peter-e/ Sweden
I did even dirtier job: the cgi parser the log every time people request a report! -- for the time being, it is used as a prototype and also used by a few people. so, not a problem. batch processing is bad when people expect instant real time report. seems that to parse the log, it depends a lot on the url (get string). so, we need mod_perl to allow the customization -- that is the apache side. seems to me it is totally a apache question, not a pg question. the parser has to be written in perl with regular exression for easy development and maintenaince. On Thu, 20 Jan 2000, Eric van der Vlist wrote: > Hi, > > I am currently using a less elegant -but efficient- solution which is to > use the standard Apache log module and to post process the log files > into postgreSQL tables. > > The advantage of this method is that you can perform this post > processing on another server, removing all the strain from the web > server. > > If anyone is interested, I can contribute the scripts I have developed. > They are tailored to my specific needs but could easily be adapted. > > Hope this helps. > > Eric > > Marc Tardif wrote: > > > > In the contrib directory, there is an example on how to redirect apache > > logs to a postgresql database. It says to create a table and change two > > lines in the apache configuration file to use psql for copying the > > logs to the database. > > > > My question is: using an sql db is mostly compelling for very active > > sites but, if this is the case, is it sensible to use psql? > > > > First, I find that using the psql command could have too much overhead. > > Seeing the executable is 46696 bytes and dynamically linked, I'm not > > really sure what is the strain on the system for bringing up psql. > > > > Second, I have tried to see the alternative of writing an apache module > > or, more sensibly, changing the current module responsible for logging. > > After reading mod_log_config.c, I'm not really sure how I can integrate > > libpq and avoid having to connect and disconnect repeatedly. If I can't > > somehow sustain a connection, I imagine the module would be just as > > efficient as using psql. > > > > Lastly, I'm confused and overwhelmed by the size of the task at hand. I'd > > appreciate any suggestions or recommendations or thoughts on whether > > writing a module is even worthwhile at all. > > > > Thanks, > > Marc > > > > ************ > > -- > ------------------------------------------------------------------------ > Eric van der Vlist Dyomedea > > http://www.dyomedea.com http://www.ducotede.com > ------------------------------------------------------------------------ > > ************ >