Thread: Logging just SQL commands in an SQL script format

Logging just SQL commands in an SQL script format

From
"Campbell, Lance"
Date:

I want to log all SQL commands as an executable script file for use in performance testing.  Currently there is a lot of information in the log that is generated that would not be executable.  Below is an example of what I would want to see; which is just the SQL.

 

Example of log output:

Select a,b,c from calendar where a=12;

Select a,b,c from calendar where a=13;

 

Example:

1) I would take a snap shot of our production database. 

2) I would then turn on SQL logging. 

3) I would put the copy of the database onto a test server. 

4) After a few days of logging the SQL commands I would copy the file to the test server.

5) I would then have a basic script that would display the start time, execute the postgres SQL log (pgsql –d database_name –f sql_log_file_name), and then display the end time.

 

Now I can test configuration changes and index modifications on real world data to see what impact the changes would have.  I just keep reloading the database, change the configurations settings, and then execute the script again.

 

The key is that the SQL log be in a format that can be executed as a postgres SQL file.

 

How do I do this?

 

Thanks,

 

 

 

Lance Campbell

Project Manager/Software Architect

Web Services at Public Affairs

University of Illinois

217.333.0382

http://webservices.uiuc.edu

 

Re: Logging just SQL commands in an SQL script format

From
Rigmor Ukuhe
Date:
Campbell, Lance wrote:
> I want to log all SQL commands as an executable script file for use in
> performance testing.  Currently there is a lot of information in the log
> that is generated that would not be executable.  Below is an example of
> what I would want to see; which is just the SQL.
>
>
>
> Example of log output:
>
> Select a,b,c from calendar where a=12;
>
> Select a,b,c from calendar where a=13;
>
>

Hi,

Wouldn't SQL statements in log possible be out of order, also transaction info
would be missing?

Even if you do it in application-side, logging query start time and transaction
info, and feed this data to some application that starts to excecute them in
correct time and context, you wouldnt be able to reliably replicate exact
behavior, cause configuration change might change query times too much.
But i see it _can_ be useful in some cases.

Rigmor

>
>
>
> How do I do this?
>
>
>
> Thanks,
>
>
>
>
>
>
>
> Lance Campbell
>
> Project Manager/Software Architect
>
> Web Services at Public Affairs
>
> University of Illinois
>
> 217.333.0382
>
> http://webservices.uiuc.edu
>
>
>
> !DSPAM:5,45b7c785287772091230466!


--
Rigmor Ukuhe
Finestmedia Ltd | Software Development Team Manager
gsm : (+372)56467729 | tel : (+372)6558043 | e-mail : rigmor.ukuhe@finestmedia.ee

Re: Logging just SQL commands in an SQL script format

From
"Daniel Cristian Cruz"
Date:
Look at pgfouine.

It can extract SQL commands from log, and build a xml file to be used
with tsung to generate some load testing.

Kind regards...

On 1/24/07, Campbell, Lance <lance@uiuc.edu> wrote:
>
>
>
>
> I want to log all SQL commands as an executable script file for use in
> performance testing.  Currently there is a lot of information in the log
> that is generated that would not be executable.  Below is an example of what
> I would want to see; which is just the SQL.
>
>
>
> Example of log output:
>
> Select a,b,c from calendar where a=12;
>
> Select a,b,c from calendar where a=13;
>
>
>
> Example:
>
> 1) I would take a snap shot of our production database.
>
> 2) I would then turn on SQL logging.
>
> 3) I would put the copy of the database onto a test server.
>
> 4) After a few days of logging the SQL commands I would copy the file to the
> test server.
>
> 5) I would then have a basic script that would display the start time,
> execute the postgres SQL log (pgsql –d database_name –f sql_log_file_name),
> and then display the end time.
>
>
>
> Now I can test configuration changes and index modifications on real world
> data to see what impact the changes would have.  I just keep reloading the
> database, change the configurations settings, and then execute the script
> again.
>
>
>
> The key is that the SQL log be in a format that can be executed as a
> postgres SQL file.
>
>
>
> How do I do this?
>
>
>
> Thanks,
>
>
>
>
>
>
>
> Lance Campbell
>
> Project Manager/Software Architect
>
> Web Services at Public Affairs
>
> University of Illinois
>
> 217.333.0382
>
> http://webservices.uiuc.edu
>
>


--
Daniel Cristian Cruz
Analista de Sistemas
Especialista postgreSQL e Linux
Instrutor Certificado Mandriva

Re: Logging just SQL commands in an SQL script format

From
"Campbell, Lance"
Date:
I actually wrote a java application that can strip out the SQL and build
a SQL script file.

If you take a snap shot of your database and then log the SQL commands
you have at least a real world snapshot of the SQL being executed for a
given time.  This gives you some level of measurement when comparing
some settings like memory allocation and index changes.

You are correct it is not perfect.  But it does give you a way to
measure the performance difference to some degree.  Performance tuning a
database is considered an art by many because every database is
different and every server is different.  I am just trying to take a
little of the art out of the equation.

Thanks,

Lance Campbell
Project Manager/Software Architect
Web Services at Public Affairs
University of Illinois
217.333.0382
http://webservices.uiuc.edu


-----Original Message-----
From: pgsql-admin-owner@postgresql.org
[mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Rigmor Ukuhe
Sent: Thursday, January 25, 2007 7:21 AM
To: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Logging just SQL commands in an SQL script format

Campbell, Lance wrote:
> I want to log all SQL commands as an executable script file for use in

> performance testing.  Currently there is a lot of information in the
log
> that is generated that would not be executable.  Below is an example
of
> what I would want to see; which is just the SQL.
>
>
>
> Example of log output:
>
> Select a,b,c from calendar where a=12;
>
> Select a,b,c from calendar where a=13;
>
>

Hi,

Wouldn't SQL statements in log possible be out of order, also
transaction info
would be missing?

Even if you do it in application-side, logging query start time and
transaction
info, and feed this data to some application that starts to excecute
them in
correct time and context, you wouldnt be able to reliably replicate
exact
behavior, cause configuration change might change query times too much.
But i see it _can_ be useful in some cases.

Rigmor

>
>
>
> How do I do this?
>
>
>
> Thanks,
>
>
>
>
>
>
>
> Lance Campbell
>
> Project Manager/Software Architect
>
> Web Services at Public Affairs
>
> University of Illinois
>
> 217.333.0382
>
> http://webservices.uiuc.edu
>
>
>
> !DSPAM:5,45b7c785287772091230466!


--
Rigmor Ukuhe
Finestmedia Ltd | Software Development Team Manager
gsm : (+372)56467729 | tel : (+372)6558043 | e-mail :
rigmor.ukuhe@finestmedia.ee

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
       subscribe-nomail command to majordomo@postgresql.org so that your
       message can get through to the mailing list cleanly