Thread: PREPARE in ECPG
We have a large application written in C language (not C++) using embedded SQL. The number of tables in the database is about 60, with about 80 SQL indexes. Total number of SQL statements is about 800 to 1000. All SQL statements are generated by a program generator handling the differences between Oracle/Informix/PostgreSQL. All SELECT statements are "simple", extracting data from only one table (no joins or similar).
The application works under Oracle (using Pro-C preprocessor) and Informix (using ESQL preprocessor). Under PostgreSQL we use the ECPG preprocessor. FYI it took us only a few weeks to port the application to PostgreSQL.
The application works under all these database systems and has been trimmed with the necessary SQL indexes etc. For the PostgreSQL we continually run the necessary VACUUM + ANALYZE to ensure the PostgreSQL optimizer uses the proper SQL indexes (there are no table scans).
The documentation for PostgreSQL 7.3 explains about introducing the optimized PREPARE/EXECUTE statements, but we have discovered this is not implemented (yet) in ECPG (we are using release 7.3.2). See comments in "ecpg/preproc/preproc.y".
We have a major installation currently using Oracle, but they want to switch to PostgreSQL when the access time reaches a reasonable level. The size of database (tables + indexes) is about 120 GB, and the number of SELECTs from the tables is of the magnitude 200 to 500 million per day, split over a number of SUN E10K machines (about 30 processors each with 1 GB RAM). (The Oracle license and annual maintenance fee on this setup are exorbitant!!)
Without the optimized PREPARE/EXECUTE we see a SELECT time of about 30 milliseconds (measured on the SUN E10K) of which most seems to be CPU time spent inside postmaster. The similar time from Oracle is about 1 to 5 milliseconds on the same hardware. Both PostgreSQL and Oracle have been setup with a cache (shared_buffers) of 120000.
To our customer the 30 milliseconds are not acceptable. Based on the PostgreSQL 7.3 documentation we should expect "improved performance" when using prepared queries.
This leads to a couple of questions regarding the optimized PREPARE/EXECUTE:
- Do you have any indication of the performance improvement (measured in percentage)?
- When will the optimized PREPARE/EXECUTE be implemented in ECPG?
Best Regards
Bo Tveden
Mads Madsen
BTS Solutions
Okay, I couldn't work out what to trim from your post so I just left it all in. You've got 800-1000 sql statements but are they reused in a single connection? If not you're not going to be gaining much, if anything, by using prepare/execute so the ecpg implementation is a moot point. This is because the prepare only prepares a statment in the backend serving that particular client. The other side of this is, it sounds from your description that your client might be long lived and therefore maintain database connection(s) and use the same statements repeatedly. That's obviously the situation to take advantage of prepare/execute. As for the specific question you asked, well I have no idea, sorry. -- Nigel J. Andrews On Fri, 23 May 2003, BTS wrote: > We have a large application written in C language (not C++) using embedded SQL. The number of tables in the database isabout 60, with about 80 SQL indexes. Total number of SQL statements is about 800 to 1000. All SQL statements are generatedby a program generator handling the differences between Oracle/Informix/PostgreSQL. All SELECT statements are "simple",extracting data from only one table (no joins or similar). > > The application works under Oracle (using Pro-C preprocessor) and Informix (using ESQL preprocessor). Under PostgreSQLwe use the ECPG preprocessor. FYI it took us only a few weeks to port the application to PostgreSQL. > > The application works under all these database systems and has been trimmed with the necessary SQL indexes etc. For thePostgreSQL we continually run the necessary VACUUM + ANALYZE to ensure the PostgreSQL optimizer uses the proper SQL indexes(there are no table scans). > > The documentation for PostgreSQL 7.3 explains about introducing the optimized PREPARE/EXECUTE statements, but we have discoveredthis is not implemented (yet) in ECPG (we are using release 7.3.2). See comments in "ecpg/preproc/preproc.y". > > We have a major installation currently using Oracle, but they want to switch to PostgreSQL when the access time reachesa reasonable level. The size of database (tables + indexes) is about 120 GB, and the number of SELECTs from the tablesis of the magnitude 200 to 500 million per day, split over a number of SUN E10K machines (about 30 processors eachwith 1 GB RAM). (The Oracle license and annual maintenance fee on this setup are exorbitant!!) > > Without the optimized PREPARE/EXECUTE we see a SELECT time of about 30 milliseconds (measured on the SUN E10K) of whichmost seems to be CPU time spent inside postmaster. The similar time from Oracle is about 1 to 5 milliseconds on thesame hardware. Both PostgreSQL and Oracle have been setup with a cache (shared_buffers) of 120000. > > To our customer the 30 milliseconds are not acceptable. Based on the PostgreSQL 7.3 documentation we should expect "improvedperformance" when using prepared queries. > > This leads to a couple of questions regarding the optimized PREPARE/EXECUTE: > > - Do you have any indication of the performance improvement (measured in percentage)? > > - When will the optimized PREPARE/EXECUTE be implemented in ECPG? > > > Best Regards > Bo Tveden > Mads Madsen > BTS Solutions
Our application consists of several programs with one major running in
online mode. In online mode we start typically 40 to 60 incarnations of this
program, and it runs for months (until the machine is rebooted). Therefor
the effect on an efficient PREPARE/EXECUTE will be significant.
We know the importance of this from running our application under Informix,
because there we use the efficient PREPARE/EXECUTE.
We look forward for the ECPG implementation of
PREPARE plan_name [ (datatype [, ...] ) ] AS query
and
EXECUTE plan_name [ (parameter [, ...] ) ] INTO...
We hope to be able to write statements like:
EXEC SQL
PREPARE stm (char(5), int) as SELECT a,b,c INTO :host_a, :host_b,
:host_c
FROM mytable WHERE d=$1 AND e=$2;
...
EXEC SQL
EXECUTE stm (:host_d, :host_e);
or
EXEC SQL
PREPARE stm (char(5), int) as SELECT a,b,c FROM mytable WHERE d=$1 AND
e=$2;
...
EXEC SQL
EXECUTE stm (:host_d, :host_e) INTO :host_a, :host_b, :host_c;
Best Regards
Bo Tveden
Mads Madsen
BTS Solutions
----- Original Message -----
From: "Nigel J. Andrews" <nandrews@investsystems.co.uk>
To: "BTS" <bts_postgresql@hotmail.com>
Cc: <pgsql-interfaces@postgresql.org>
Sent: Friday, May 23, 2003 12:14 PM
Subject: Re: [INTERFACES] PREPARE in ECPG
>
> Okay, I couldn't work out what to trim from your post so I just left it
all in.
>
> You've got 800-1000 sql statements but are they reused in a single
connection?
> If not you're not going to be gaining much, if anything, by using
> prepare/execute so the ecpg implementation is a moot point. This is
because the
> prepare only prepares a statment in the backend serving that particular
client.
> The other side of this is, it sounds from your description that your
client
> might be long lived and therefore maintain database connection(s) and use
the
> same statements repeatedly. That's obviously the situation to take
advantage
> of prepare/execute.
>
> As for the specific question you asked, well I have no idea, sorry.
>
>
> --
> Nigel J. Andrews
>
>
> On Fri, 23 May 2003, BTS wrote:
>
> > We have a large application written in C language (not C++) using
embedded SQL. The number of tables in the database is about 60, with about
80 SQL indexes. Total number of SQL statements is about 800 to 1000. All SQL
statements are generated by a program generator handling the differences
between Oracle/Informix/PostgreSQL. All SELECT statements are "simple",
extracting data from only one table (no joins or similar).
> >
> > The application works under Oracle (using Pro-C preprocessor) and
Informix (using ESQL preprocessor). Under PostgreSQL we use the ECPG
preprocessor. FYI it took us only a few weeks to port the application to
PostgreSQL.
> >
> > The application works under all these database systems and has been
trimmed with the necessary SQL indexes etc. For the PostgreSQL we
continually run the necessary VACUUM + ANALYZE to ensure the PostgreSQL
optimizer uses the proper SQL indexes (there are no table scans).
> >
> > The documentation for PostgreSQL 7.3 explains about introducing the
optimized PREPARE/EXECUTE statements, but we have discovered this is not
implemented (yet) in ECPG (we are using release 7.3.2). See comments in
"ecpg/preproc/preproc.y".
> >
> > We have a major installation currently using Oracle, but they want to
switch to PostgreSQL when the access time reaches a reasonable level. The
size of database (tables + indexes) is about 120 GB, and the number of
SELECTs from the tables is of the magnitude 200 to 500 million per day,
split over a number of SUN E10K machines (about 30 processors each with 1 GB
RAM). (The Oracle license and annual maintenance fee on this setup are
exorbitant!!)
> >
> > Without the optimized PREPARE/EXECUTE we see a SELECT time of about 30
milliseconds (measured on the SUN E10K) of which most seems to be CPU time
spent inside postmaster. The similar time from Oracle is about 1 to 5
milliseconds on the same hardware. Both PostgreSQL and Oracle have been
setup with a cache (shared_buffers) of 120000.
> >
> > To our customer the 30 milliseconds are not acceptable. Based on the
PostgreSQL 7.3 documentation we should expect "improved performance" when
using prepared queries.
> >
> > This leads to a couple of questions regarding the optimized
PREPARE/EXECUTE:
> >
> > - Do you have any indication of the performance improvement (measured in
percentage)?
> >
> > - When will the optimized PREPARE/EXECUTE be implemented in ECPG?
> >
> >
> > Best Regards
> > Bo Tveden
> > Mads Madsen
> > BTS Solutions
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>
online mode. In online mode we start typically 40 to 60 incarnations of this
program, and it runs for months (until the machine is rebooted). Therefor
the effect on an efficient PREPARE/EXECUTE will be significant.
We know the importance of this from running our application under Informix,
because there we use the efficient PREPARE/EXECUTE.
We look forward for the ECPG implementation of
PREPARE plan_name [ (datatype [, ...] ) ] AS query
and
EXECUTE plan_name [ (parameter [, ...] ) ] INTO...
We hope to be able to write statements like:
EXEC SQL
PREPARE stm (char(5), int) as SELECT a,b,c INTO :host_a, :host_b,
:host_c
FROM mytable WHERE d=$1 AND e=$2;
...
EXEC SQL
EXECUTE stm (:host_d, :host_e);
or
EXEC SQL
PREPARE stm (char(5), int) as SELECT a,b,c FROM mytable WHERE d=$1 AND
e=$2;
...
EXEC SQL
EXECUTE stm (:host_d, :host_e) INTO :host_a, :host_b, :host_c;
Best Regards
Bo Tveden
Mads Madsen
BTS Solutions
----- Original Message -----
From: "Nigel J. Andrews" <nandrews@investsystems.co.uk>
To: "BTS" <bts_postgresql@hotmail.com>
Cc: <pgsql-interfaces@postgresql.org>
Sent: Friday, May 23, 2003 12:14 PM
Subject: Re: [INTERFACES] PREPARE in ECPG
>
> Okay, I couldn't work out what to trim from your post so I just left it
all in.
>
> You've got 800-1000 sql statements but are they reused in a single
connection?
> If not you're not going to be gaining much, if anything, by using
> prepare/execute so the ecpg implementation is a moot point. This is
because the
> prepare only prepares a statment in the backend serving that particular
client.
> The other side of this is, it sounds from your description that your
client
> might be long lived and therefore maintain database connection(s) and use
the
> same statements repeatedly. That's obviously the situation to take
advantage
> of prepare/execute.
>
> As for the specific question you asked, well I have no idea, sorry.
>
>
> --
> Nigel J. Andrews
>
>
> On Fri, 23 May 2003, BTS wrote:
>
> > We have a large application written in C language (not C++) using
embedded SQL. The number of tables in the database is about 60, with about
80 SQL indexes. Total number of SQL statements is about 800 to 1000. All SQL
statements are generated by a program generator handling the differences
between Oracle/Informix/PostgreSQL. All SELECT statements are "simple",
extracting data from only one table (no joins or similar).
> >
> > The application works under Oracle (using Pro-C preprocessor) and
Informix (using ESQL preprocessor). Under PostgreSQL we use the ECPG
preprocessor. FYI it took us only a few weeks to port the application to
PostgreSQL.
> >
> > The application works under all these database systems and has been
trimmed with the necessary SQL indexes etc. For the PostgreSQL we
continually run the necessary VACUUM + ANALYZE to ensure the PostgreSQL
optimizer uses the proper SQL indexes (there are no table scans).
> >
> > The documentation for PostgreSQL 7.3 explains about introducing the
optimized PREPARE/EXECUTE statements, but we have discovered this is not
implemented (yet) in ECPG (we are using release 7.3.2). See comments in
"ecpg/preproc/preproc.y".
> >
> > We have a major installation currently using Oracle, but they want to
switch to PostgreSQL when the access time reaches a reasonable level. The
size of database (tables + indexes) is about 120 GB, and the number of
SELECTs from the tables is of the magnitude 200 to 500 million per day,
split over a number of SUN E10K machines (about 30 processors each with 1 GB
RAM). (The Oracle license and annual maintenance fee on this setup are
exorbitant!!)
> >
> > Without the optimized PREPARE/EXECUTE we see a SELECT time of about 30
milliseconds (measured on the SUN E10K) of which most seems to be CPU time
spent inside postmaster. The similar time from Oracle is about 1 to 5
milliseconds on the same hardware. Both PostgreSQL and Oracle have been
setup with a cache (shared_buffers) of 120000.
> >
> > To our customer the 30 milliseconds are not acceptable. Based on the
PostgreSQL 7.3 documentation we should expect "improved performance" when
using prepared queries.
> >
> > This leads to a couple of questions regarding the optimized
PREPARE/EXECUTE:
> >
> > - Do you have any indication of the performance improvement (measured in
percentage)?
> >
> > - When will the optimized PREPARE/EXECUTE be implemented in ECPG?
> >
> >
> > Best Regards
> > Bo Tveden
> > Mads Madsen
> > BTS Solutions
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>
On Fri, May 23, 2003 at 08:55:07AM +0200, BTS wrote: > The documentation for PostgreSQL 7.3 explains about introducing the optimized PREPARE/EXECUTE statements, but we have discoveredthis is not implemented (yet) in ECPG (we are using release 7.3.2). See comments in "ecpg/preproc/preproc.y". > ... > - When will the optimized PREPARE/EXECUTE be implemented in ECPG? As soon as we find a way to do it. I do not see a good way to pass variables to the backend's prepare statement. We had a longer discussion on this starting with the different syntax until I understood that you cannot prepare a statement like: select * from table where id=:id; until you already know the value of the variable id, i.e. the backend does not know a USING clause to specify the variables when executing the command. But then I may have missed something. It happened before. :-) Michael -- Michael Meskes Email: Michael@Fam-Meskes.De ICQ: 179140304, AIM: michaelmeskes, Jabber: meskes@jabber.org Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!
On Wed, May 28, 2003 at 08:58:56AM +0200, BTS wrote: > We hope to be able to write statements like: > EXEC SQL > PREPARE stm (char(5), int) as SELECT a,b,c INTO :host_a, :host_b, > :host_c > FROM mytable WHERE d=$1 AND e=$2; Are you able to do this via psql? I.e. can you specify a placeholder like $1 in psql? Michael -- Michael Meskes Email: Michael@Fam-Meskes.De ICQ: 179140304, AIM: michaelmeskes, Jabber: meskes@jabber.org Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!