Thread: analyzing postgresql performance for dbt-2

analyzing postgresql performance for dbt-2

From
markw@osdl.org
Date:
I'm running our DBT-2 workload against PostgreSQL 7.3.4 and I'm having
some trouble figuring out what I should be looking for when I'm trying
to tune the database.  I have results for a decent baseline, but when I
try to increase the load on the database, the performance drops.
Nothing in the graphs (in the links listed later) sticks out to me so
I'm wondering if there are other database statitics I should try to
collect. Any suggestions would be great and let me know if I can answer
any other questions.

Here are a pair of results where I just raise the load on the
database, where increasing the load increases the area of the database
touched in addition to increasing the transaction rate.  The overall
metric increases somewhat, but the response time for most of the
interactions also increases significantly:

http://developer.osdl.org/markw/dbt2-pgsql/158/ [baseline]
    - load of 100 warehouses
    - metric 1249.65

http://developer.osdl.org/markw/dbt2-pgsql/149/
    - load of 140 warehouses
    - metric 1323.90

Both of these runs had wal_buffers set to 8, checkpoint_segments 200,
and checkpoint_timeout 1800.

So far I've only tried various wal_buffers and checkpoint_segments
settings in the next set of results for a load of 140 warehouses.

http://developer.osdl.org/markw/dbt2-pgsql/148/
- metric 1279.26
- wal_buffers 8
- checkpoint_segments 100
- checkpoint_timeout 300

http://developer.osdl.org/markw/dbt2-pgsql/149/
- metric 1323.90
- wal_buffers 8
- checkpoint_segments 200
- checkpoint_timeout 1800

http://developer.osdl.org/markw/dbt2-pgsql/150/
- metric 1281.13
- wal_buffers 8
- checkpoint_segments 300
- checkpoint_timeout 1800

http://developer.osdl.org/markw/dbt2-pgsql/151/
- metric 1311.99
- wal_buffers 32
- checkpoint_segments 200
- checkpoint_timeout 1800

http://developer.osdl.org/markw/dbt2-pgsql/152/
- metric 1268.37
- wal_buffers 64
- checkpoint_segments 200
- checkpoint_timeout 1800

http://developer.osdl.org/markw/dbt2-pgsql/154/
- metric 1314.62
- wal_buffers 16
- checkpoint_segments 200
- checkpoint_timeout 1800


Thanks!

--
Mark Wong - - markw@osdl.org
Open Source Development Lab Inc - A non-profit corporation
12725 SW Millikan Way - Suite 400 - Beaverton, OR 97005
(503) 626-2455 x 32 (office)
(503) 626-2436      (fax)

Re: analyzing postgresql performance for dbt-2

From
Bruce Momjian
Date:
markw@osdl.org wrote:
> I'm running our DBT-2 workload against PostgreSQL 7.3.4 and I'm having
> some trouble figuring out what I should be looking for when I'm trying
> to tune the database.  I have results for a decent baseline, but when I
> try to increase the load on the database, the performance drops.
> Nothing in the graphs (in the links listed later) sticks out to me so
> I'm wondering if there are other database statitics I should try to
> collect. Any suggestions would be great and let me know if I can answer
> any other questions.
>
> Here are a pair of results where I just raise the load on the
> database, where increasing the load increases the area of the database
> touched in addition to increasing the transaction rate.  The overall
> metric increases somewhat, but the response time for most of the
> interactions also increases significantly:
>
> http://developer.osdl.org/markw/dbt2-pgsql/158/ [baseline]
>     - load of 100 warehouses
>     - metric 1249.65
>
> http://developer.osdl.org/markw/dbt2-pgsql/149/
>     - load of 140 warehouses
>     - metric 1323.90

I looked at these charts and they looked normal to me.  It looked like
your the load increased until your computer was saturated.  Is there
something I am missing?

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: analyzing postgresql performance for dbt-2

From
Mark Wong
Date:
On Tue, Oct 21, 2003 at 08:35:56PM -0400, Bruce Momjian wrote:
> markw@osdl.org wrote:
> > I'm running our DBT-2 workload against PostgreSQL 7.3.4 and I'm having
> > some trouble figuring out what I should be looking for when I'm trying
> > to tune the database.  I have results for a decent baseline, but when I
> > try to increase the load on the database, the performance drops.
> > Nothing in the graphs (in the links listed later) sticks out to me so
> > I'm wondering if there are other database statitics I should try to
> > collect. Any suggestions would be great and let me know if I can answer
> > any other questions.
> >
> > Here are a pair of results where I just raise the load on the
> > database, where increasing the load increases the area of the database
> > touched in addition to increasing the transaction rate.  The overall
> > metric increases somewhat, but the response time for most of the
> > interactions also increases significantly:
> >
> > http://developer.osdl.org/markw/dbt2-pgsql/158/ [baseline]
> >     - load of 100 warehouses
> >     - metric 1249.65
> >
> > http://developer.osdl.org/markw/dbt2-pgsql/149/
> >     - load of 140 warehouses
> >     - metric 1323.90
>
> I looked at these charts and they looked normal to me.  It looked like
> your the load increased until your computer was saturated.  Is there
> something I am missing?

I've run some i/o tests so I'm pretty sure I haven't saturated that.  And it
looks like I have almost 10% more processor time left.  I do agree that it
appears something might be saturated, I just don't know where to look...

Thanks,
Mark

Re: analyzing postgresql performance for dbt-2

From
Bruce Momjian
Date:
Mark Wong wrote:
> > > Here are a pair of results where I just raise the load on the
> > > database, where increasing the load increases the area of the database
> > > touched in addition to increasing the transaction rate.  The overall
> > > metric increases somewhat, but the response time for most of the
> > > interactions also increases significantly:
> > >
> > > http://developer.osdl.org/markw/dbt2-pgsql/158/ [baseline]
> > >     - load of 100 warehouses
> > >     - metric 1249.65
> > >
> > > http://developer.osdl.org/markw/dbt2-pgsql/149/
> > >     - load of 140 warehouses
> > >     - metric 1323.90
> >
> > I looked at these charts and they looked normal to me.  It looked like
> > your the load increased until your computer was saturated.  Is there
> > something I am missing?
>
> I've run some i/o tests so I'm pretty sure I haven't saturated that.  And it
> looks like I have almost 10% more processor time left.  I do agree that it
> appears something might be saturated, I just don't know where to look...

Could the 10% be context switching time, or is the I/O saturated?

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: analyzing postgresql performance for dbt-2

From
markw@osdl.org
Date:
On 26 Oct, Bruce Momjian wrote:
> Mark Wong wrote:
>> > > Here are a pair of results where I just raise the load on the
>> > > database, where increasing the load increases the area of the database
>> > > touched in addition to increasing the transaction rate.  The overall
>> > > metric increases somewhat, but the response time for most of the
>> > > interactions also increases significantly:
>> > >
>> > > http://developer.osdl.org/markw/dbt2-pgsql/158/ [baseline]
>> > >     - load of 100 warehouses
>> > >     - metric 1249.65
>> > >
>> > > http://developer.osdl.org/markw/dbt2-pgsql/149/
>> > >     - load of 140 warehouses
>> > >     - metric 1323.90
>> >
>> > I looked at these charts and they looked normal to me.  It looked like
>> > your the load increased until your computer was saturated.  Is there
>> > something I am missing?
>>
>> I've run some i/o tests so I'm pretty sure I haven't saturated that.  And it
>> looks like I have almost 10% more processor time left.  I do agree that it
>> appears something might be saturated, I just don't know where to look...
>
> Could the 10% be context switching time, or is the I/O saturated?

There are about 14,000 to 17,000 context switches/s according to the
vmstat output.  This is on a 1.5Ghz hyperthreaded Xeon processor.  I
don't know what I'm supposed to be able to expect in terms of context
switching.  I really doubt the i/o is saturated because I've run
disktest (part of the Linux Test Project suite) and saw much higher
throughput for various sequential/random read/write tests.

I'm starting to collect oprofile data (and will hopefully have some
results soon) to get an idea where the database is spending its time,
just in case that may have something to do with it.

Mark

Re: analyzing postgresql performance for dbt-2

From
markw@osdl.org
Date:
I've done a better controlled series of tests where I restore the
database before each test and have grabbed sar and oprofile data:

http://developer.osdl.org/markw/dbt2-pgsql/176/
    - load of 100 warehouses
    - metric 1234.52

http://developer.osdl.org/markw/dbt2-pgsql/177/
    - load of 120 warehouses
    - metric 1259.43

http://developer.osdl.org/markw/dbt2-pgsql/178/
    - load of 140 warehouses
    - metric 1244.33

For the most part our primary metric, and the vmstat and sar output look
fairly close for each run.  Here are a couple of things that I've found
to be considerably different from run 176 to 178:

- oprofile says postgresql calls to SearchCatCache increased ~ 20%

- readprofile says there are 50% more calls in the linux kernel to
  do_signaction (in kernel/signal.c)

Would these two things offer any insight to what might be throttling the
throughput?

Thanks,
Mark