Thread: Improve ODBC Throughput ?
Hi list ! I mostly use my PostgreSQL as a backend DB with Access2000 frontends. The tables are linked via psqlODBC. The problem I have is that even though queries / views run very fast, the data take a long time to arrive to the client. In the best situations, data is sent/received at ~500kB per second (for instance, tables with only one column, which is a very large text field >10.000 bytes per field). In bad situations, I am at ~50kB per second. If the queries / views return large resultsets, it can seconds for the table to open., even though the query ran in milliseconds on the server (explain analyze proves this). What can I do to improve this throughput ? Either on the server side, or on the ODBC client side ? As a side note : network congestion, server acitvity is not a good explanation. This server also is a file server, and I transfer files at 10MB per second all the time. Thanks for your advices ! -- Arnaud
What kinds of indexes do you have on your tables? I think the way Access works to limit transfer it first pulls the indexes and probably the primary one first to determine how to fetch and how many to fetch. I suspect if you have no indexes performance would be really bad. Also which version of psqlODBC driver are you using? -----Original Message----- From: pgsql-odbc-owner@postgresql.org [mailto:pgsql-odbc-owner@postgresql.org] On Behalf Of Arnaud Lesauvage Sent: Thursday, October 19, 2006 8:55 AM To: pgsql-odbc@postgresql.org Subject: [ODBC] Improve ODBC Throughput ? Hi list ! I mostly use my PostgreSQL as a backend DB with Access2000 frontends. The tables are linked via psqlODBC. The problem I have is that even though queries / views run very fast, the data take a long time to arrive to the client. In the best situations, data is sent/received at ~500kB per second (for instance, tables with only one column, which is a very large text field >10.000 bytes per field). In bad situations, I am at ~50kB per second. If the queries / views return large resultsets, it can seconds for the table to open., even though the query ran in milliseconds on the server (explain analyze proves this). What can I do to improve this throughput ? Either on the server side, or on the ODBC client side ? As a side note : network congestion, server acitvity is not a good explanation. This server also is a file server, and I transfer files at 10MB per second all the time. Thanks for your advices ! -- Arnaud ---------------------------(end of broadcast)--------------------------- TIP 2: Don't 'kill -9' the postmaster ----------------------------------------- The substance of this message, including any attachments, may be confidential, legally privileged and/or exempt from disclosure pursuant to Massachusetts law. It is intended solely for the addressee. If you received this in error, please contact the sender and delete the material from any computer.
Obe, Regina a écrit : > What kinds of indexes do you have on your tables? I think the way > Access works to limit transfer it first pulls the indexes and probably > the primary one first to determine how to fetch and how many to fetch. > I suspect if you have no indexes performance would be really bad. > > Also which version of psqlODBC driver are you using? I have integer primary keys on most tables, if not all of them. Even my views have an integer primary key (at least, something that can be used as an integer pkey in Access). My version of psqlODBC is 8.1.2.0. I forgot to mention something important : I obtain the best performances (500kB / sec, cf my first post) when I am NOT in access. In a VBA module (Autocad, not Access), I open an ADO connection to PostgreSQL and directly query the server (no linked table involved here). But still, this is "only" 500kB/sec...
Remember that any data access component is riding on the underlying physical layer. You need to make sure that your network infrastructure can pump that much data through it. Are you network cards running in full duples or half duplex? What sort of switching infrastructure do you have? The vast majority of switches have a blocking architecture. In a heavy environment this will cause oversubscription whichwill result in congestion. If you are using a managed switch, your network admins will hopefully know how to troubleshootand tell you if that is the case (they will see resends, etc.). Just another area to look at. > -----Original Message----- > From: pgsql-odbc-owner@postgresql.org > [mailto:pgsql-odbc-owner@postgresql.org] On Behalf Of Arnaud Lesauvage > Sent: Thursday, October 19, 2006 8:24 AM > To: Obe, Regina > Cc: pgsql-odbc@postgresql.org > Subject: Re: [ODBC] Improve ODBC Throughput ? > > Obe, Regina a écrit : > > What kinds of indexes do you have on your tables? I think the way > > Access works to limit transfer it first pulls the indexes > and probably > > the primary one first to determine how to fetch and how > many to fetch. > > I suspect if you have no indexes performance would be really bad. > > > > Also which version of psqlODBC driver are you using? > > I have integer primary keys on most tables, if not all of them. > Even my views have an integer primary key (at least, > something that can be used as an integer pkey in Access). > > My version of psqlODBC is 8.1.2.0. > > I forgot to mention something important : I obtain the best > performances (500kB / sec, cf my first post) when I am NOT in access. > In a VBA module (Autocad, not Access), I open an ADO > connection to PostgreSQL and directly query the server (no > linked table involved here). But still, this is "only" > 500kB/sec... > > > > ---------------------------(end of > broadcast)--------------------------- > TIP 1: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org > so that your > message can get through to the mailing list cleanly >
Benjamin Krajmalnik a écrit : > Remember that any data access component is riding on the underlying physical layer. Yes, but this network is fast. From the same server, file transfer is using the full ethernet capacity (~10MB / second).
You say that you have a performance issue.
1. Cache Size
I would guess that the problem may be in how many records you retrieve at a time.
In VBA code with ADO , it is sometimes helpful to use the Recordset CacheSize property to pull more than one row at a time.
2.Declare Fetch
In Access, you do not have this fine control. Some psqlODBC user have experimented with the DECLARE/FETCH option in the psqlODBC settngs.I am not entirely sure how Declare/Fetch works though, I have never used it.
3. General design for performance
I don't know much about your PostgreSQL database and Access setup, but I can imagine a situation where you are trying to open a linked table which has millions of rows and some of the fields are of type TEXT, essentially BLOBs that Access must treat as MEMO. Simply trying to open something like that whole is a pretty bad idea for a transactional application. It is usually best to filter,...retrieve the fewest rows to suit the users need, and to do other transactions explicitly one records at a time on primary keys.
Lastly, you could turn on the ODBC trace or the psqlODBC MyLog to looks for less obvious issues, since you have ruled out the network.
Much Luck to you.
Greg Campbell ENG-ASE/Michelin US5
Lexington, South Carolina
803-951-5561, x75561
Fax: 803-951-5531
greg.campbell@us.michelin.comArnaud Lesauvage <thewild@freesurf.fr>
Sent by: pgsql-odbc-owner@postgresql.org 10/19/2006 10:24 |
|
Obe, Regina a écrit :
> What kinds of indexes do you have on your tables? I think the way
> Access works to limit transfer it first pulls the indexes and probably
> the primary one first to determine how to fetch and how many to fetch.
> I suspect if you have no indexes performance would be really bad.
>
> Also which version of psqlODBC driver are you using?
I have integer primary keys on most tables, if not all of them.
Even my views have an integer primary key (at least,
something that can be used as an integer pkey in Access).
My version of psqlODBC is 8.1.2.0.
I forgot to mention something important : I obtain the best
performances (500kB / sec, cf my first post) when I am NOT
in access.
In a VBA module (Autocad, not Access), I open an ADO
connection to PostgreSQL and directly query the server (no
linked table involved here). But still, this is "only"
500kB/sec...
---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly
Attachment
Have you tried one of the systernals tools like ProcessExplorer
Or FileMonitor
To debug it.
Those often uncover all sorts of stupid little errors.
Thanks,
Regina
From: Arnaud Lesauvage [mailto:thewild@freesurf.fr]
Sent: Thu 10/19/2006 1:13 PM
To: Benjamin Krajmalnik
Cc: Obe, Regina; pgsql-odbc@postgresql.org
Subject: Re: [ODBC] Improve ODBC Throughput ?
Benjamin Krajmalnik a écrit :
> Remember that any data access component is riding on the underlying physical layer.
Yes, but this network is fast. From the same server, file transfer
is using the full ethernet capacity (~10MB / second).
The substance of this message, including any attachments, may be
confidential, legally
privileged and/or exempt from disclosure pursuant to Massachusetts
law. It is intended
solely for the addressee. If you received this in error, please
contact the sender and
delete the material from any computer.
Arnaud Lesauvage wrote: > Hi list ! > > I mostly use my PostgreSQL as a backend DB with Access2000 frontends. > The tables are linked via psqlODBC. > The problem I have is that even though queries / views run very fast, > the data take a long time to arrive to the client. > In the best situations, data is sent/received at ~500kB per second > (for instance, tables with only one column, which is a very large text > field >10.000 bytes per field). > How are you setting the "Text As LongVarchar" option ? Are the columns text or memo using Access ? regards, Hiroshi Inoue
> In a VBA module (Autocad, not Access), I open an ADO > connection to PostgreSQL and directly query the server (no > linked table involved here). Sorry if this is slightly off topic from the thread. I am curious, what functionality are you getting when you connect Autocad to Postgresql? Regards, Richard Broersma Jr.
greg.campbell@us.michelin.com a écrit : > You say that you have a performance issue. > > 1. Cache Size > I would guess that the problem may be in how many records you retrieve at a > time. > In VBA code with ADO , it is sometimes helpful to use the Recordset > CacheSize property to pull more than one row at a time. I will try this ! I never used this setting, thansk for pointing this out ! > 2.Declare Fetch > In Access, you do not have this fine control. Some psqlODBC user have > experimented with the DECLARE/FETCH option in the psqlODBC settngs.I am not > entirely sure how Declare/Fetch works though, I have never used it. My tables are linked with these options : UseDeclareFetch=1 UseServerSidePrepare=1 Fetch=10000 TrueIsMinus1=1 BoolsAsChar=0 TextAsLongVarchar=0 RowVersioning=1 RecognizeUniqueIndexes=1 > 3. General design for performance > I don't know much about your PostgreSQL database and Access setup, but I > can imagine a situation where you are trying to open a linked table which > has millions of rows and some of the fields are of type TEXT, essentially > BLOBs that Access must treat as MEMO. Simply trying to open something like > that whole is a pretty bad idea for a transactional application. It is > usually best to filter,...retrieve the fewest rows to suit the users need, > and to do other transactions explicitly one records at a time on primary > keys. All my text fields are maximum 254 characters long (to be recognized as varchar in Access). I only have very few memo fields, and only when they are compulsorey. I think the DB design is OK in my case. > Lastly, you could turn on the ODBC trace or the psqlODBC MyLog to looks for > less obvious issues, since you have ruled out the network. I'll do that as a last solution, I don't have much time to dig into these huge logs. But I will if I don't find anything else. > Much Luck to you. Thanks, I might need luck ! ;)
Richard Broersma Jr a écrit : >> In a VBA module (Autocad, not Access), I open an ADO >> connection to PostgreSQL and directly query the server (no >> linked table involved here). > > Sorry if this is slightly off topic from the thread. > > I am curious, what functionality are you getting when you connect Autocad to Postgresql? I retrieve the WKT representation of PostGIS geometries. I wrote some very simple Autocad functions to draw them, and some simple functions to convert Autocad entities to WKT. Nothing like a connector, but it works quite well Not perfect yet, I haven't found a simple way to make multipolygons in Autocad. (any Multi obect in fact).
Hiroshi Inoue a écrit : > Arnaud Lesauvage wrote: >> Hi list ! >> >> I mostly use my PostgreSQL as a backend DB with Access2000 frontends. >> The tables are linked via psqlODBC. >> The problem I have is that even though queries / views run very fast, >> the data take a long time to arrive to the client. >> In the best situations, data is sent/received at ~500kB per second >> (for instance, tables with only one column, which is a very large text >> field >10.000 bytes per field). >> > > How are you setting the "Text As LongVarchar" option ? > Are the columns text or memo using Access ? TextAsLongVarchar=0 but the columns are VarChar in Access.
Arnaud Lesauvage wrote: > Hiroshi Inoue a écrit : >> Arnaud Lesauvage wrote: >>> Hi list ! >>> >>> I mostly use my PostgreSQL as a backend DB with Access2000 frontends. >>> The tables are linked via psqlODBC. >>> The problem I have is that even though queries / views run very fast, >>> the data take a long time to arrive to the client. >>> In the best situations, data is sent/received at ~500kB per second >>> (for instance, tables with only one column, which is a very large >>> text field >10.000 bytes per field). >>> >> >> How are you setting the "Text As LongVarchar" option ? >> Are the columns text or memo using Access ? > > TextAsLongVarchar=0 but the columns are VarChar in Access. AFAIK the max text(VarChar) size in Access is 255. Can you get the content of fields > 10000 bytes correctly under the setting ? regards, Hiroshi Inoue
> >2.Declare Fetch > >In Access, you do not have this fine control. Some psqlODBC user have > >experimented with the DECLARE/FETCH option in the psqlODBC settngs.I am not > >entirely sure how Declare/Fetch works though, I have never used it. > > My tables are linked with these options : > UseDeclareFetch=1 > UseServerSidePrepare=1 > Fetch=10000 UseDeclareFetch and UseServerSidePrepare are disjoint options. If I remember it the right way the UseServerSidePrepare disallow UseDeclareFetch. Please try UseServerSidePrepare=0 when you want using DeclareFetch. Why you set Fetch so high? How many rows do you expect? The UseDeclareFetch declare the select statement as cursor (using declare statement) and then get the rows using fetch statement (see postgresql manual). I see no usage for UseDeclareFetch with so high Fetch. The UseServerSidePrepare use prepare and execute statement (also see postgresql manual). Regards, Luf
Hiroshi Inoue a écrit : > Arnaud Lesauvage wrote: >> Hiroshi Inoue a écrit : >>> Arnaud Lesauvage wrote: >>>> Hi list ! >>>> >>>> I mostly use my PostgreSQL as a backend DB with Access2000 frontends. >>>> The tables are linked via psqlODBC. >>>> The problem I have is that even though queries / views run very fast, >>>> the data take a long time to arrive to the client. >>>> In the best situations, data is sent/received at ~500kB per second >>>> (for instance, tables with only one column, which is a very large >>>> text field >10.000 bytes per field). >>>> >>> >>> How are you setting the "Text As LongVarchar" option ? >>> Are the columns text or memo using Access ? >> >> TextAsLongVarchar=0 but the columns are VarChar in Access. > > AFAIK the max text(VarChar) size in Access is 255. > Can you get the content of fields > 10000 bytes correctly > under the setting ? In fact, the only text fields I have are memo in access, but I have very fex text fields. Most of them are character varying of length <= 254.
Ludek Finstrle a écrit : >> >2.Declare Fetch >> >In Access, you do not have this fine control. Some psqlODBC user have >> >experimented with the DECLARE/FETCH option in the psqlODBC settngs.I am not >> >entirely sure how Declare/Fetch works though, I have never used it. >> >> My tables are linked with these options : >> UseDeclareFetch=1 >> UseServerSidePrepare=1 >> Fetch=10000 > > UseDeclareFetch and UseServerSidePrepare are disjoint options. > If I remember it the right way the UseServerSidePrepare disallow > UseDeclareFetch. Please try UseServerSidePrepare=0 when you want > using DeclareFetch. > > Why you set Fetch so high? How many rows do you expect? > The UseDeclareFetch declare the select statement as cursor (using > declare statement) and then get the rows using fetch statement > (see postgresql manual). > I see no usage for UseDeclareFetch with so high Fetch. > > The UseServerSidePrepare use prepare and execute statement (also > see postgresql manual). I thought that since memory consumption was not an issue, I could fetch a lot of rows at once. I read the ODBC documentation about that but not PostgreSQL's, but I'll correct this ASAP. What is your advice in fact ? UseDeclareFetch=0 UseServerSidePrepare=1 Or maybe UseDeclareFetch=1 UseServerSidePrepare=0 Fetch=100 I might just try both in fact (don't have the time right now, maybe later today though).
Arnaud Lesauvage wrote: > Hiroshi Inoue a écrit : >> Arnaud Lesauvage wrote: >>> Hiroshi Inoue a écrit : >>>> Arnaud Lesauvage wrote: >>>>> Hi list ! >>>>> >>>>> I mostly use my PostgreSQL as a backend DB with Access2000 frontends. >>>>> The tables are linked via psqlODBC. >>>>> The problem I have is that even though queries / views run very >>>>> fast, the data take a long time to arrive to the client. >>>>> In the best situations, data is sent/received at ~500kB per second >>>>> (for instance, tables with only one column, which is a very large >>>>> text field >10.000 bytes per field). >>>>> >>>> >>>> How are you setting the "Text As LongVarchar" option ? >>>> Are the columns text or memo using Access ? >>> >>> TextAsLongVarchar=0 but the columns are VarChar in Access. >> >> AFAIK the max text(VarChar) size in Access is 255. >> Can you get the content of fields > 10000 bytes correctly >> under the setting ? > > In fact, the only text fields I have are memo in access, but I have very > fex text fields. Most of them are character varying of length <= 254. What is the type of the field > 10000 bytes in PG ? If it's the text type, you have to turn on the Text As LongVarchar option. regards, Hiroshi Inoue
Hello, > >UseDeclareFetch and UseServerSidePrepare are disjoint options. > >If I remember it the right way the UseServerSidePrepare disallow > >UseDeclareFetch. Please try UseServerSidePrepare=0 when you want > >using DeclareFetch. > > > >Why you set Fetch so high? How many rows do you expect? > >The UseDeclareFetch declare the select statement as cursor (using > >declare statement) and then get the rows using fetch statement > >(see postgresql manual). > >I see no usage for UseDeclareFetch with so high Fetch. > > > >The UseServerSidePrepare use prepare and execute statement (also > >see postgresql manual). > > I thought that since memory consumption was not an issue, I > could fetch a lot of rows at once. I read the ODBC Yes, it could. When you have UseDeclareFetch=0 it get all rows at once. But sometimes it take too long when you need few of them ASAP and others later (so it's good time to use UseDeclareFetch=1). > documentation about that but not PostgreSQL's, but I'll > correct this ASAP. That's no problem. I only point you to the documentation to fully understand what psqlODBC makes for you internally ;o) It's always the good start point when you need some tuning. BTW the start reference point is: http://www.postgresql.org/docs/8.1/interactive/sql-commands.html > What is your advice in fact ? I don't know what you really want. Sometimes is better UseDeclareFetch=1, sometimes is better UseServerSidePrepare=1 and sometimes is better UseDeclareFetch=0 and UseServerSidePrepare=0. I don't know how internally works Access or CAD so I have no advice. Regards, Luf
Arnaud Lesauvage wrote: > Ludek Finstrle a écrit : >> >> UseDeclareFetch and UseServerSidePrepare are disjoint options. >> If I remember it the right way the UseServerSidePrepare disallow >> UseDeclareFetch. Please try UseServerSidePrepare=0 when you want >> using DeclareFetch. It's not true. Though the driver doesn't use PREPARE functionality for SELECT statements under UseDeclareFetch mode, it uses PREPARE functionality for other DML commands. regards, Hiroshi Inoue
Hiroshi Inoue a écrit : >> In fact, the only text fields I have are memo in access, but I have very >> fex text fields. Most of them are character varying of length <= 254. > > What is the type of the field > 10000 bytes in PG ? > If it's the text type, you have to turn on the Text As LongVarchar option. Ok, good point, the type is text and the option was turned off. I'm changing this right now. Thanks !