Performance tuning on RedHat Enterprise Linux 3 - Mailing list pgsql-general

Executive summary: We just did a cutover from a RedHat 8.0 box to a RedHat
Enterprise Linux 3 box and we're seeing a lot more swapping on the new box
than we ever did on the old box ... this is killing performance ...

Background:

Old Box:
    RedHat 8.0
    2GB Memory
    Dual PIII 600MHz
    Postgres 7.3.4
    SHMMAX = 1073741824 (1 GB)
    shared_buffers = 65536 (roughly 0.5 GB)
    max_fsm_relations = 1000
    max_fsm_pages = 1000000
    vacuum_mem = 131072
    Roughly 25 - 30 connections open (mostly idle) at any given time
(connection pools)

New Box:
    RedHat Enterprise Linux ES 3
    2GB Memory
    Dual P4 Xeon 2.7 GHz
    Postgres 7.3.4
    SHMMAX = 1610612736 (1.5 GB)
    shared_buffers = 131072 (roughly 1GB)
    max_fsm_relations = 10000
    max_fsm_pages = 10000000
    sort_mem = 4096
    vacuum_mem = 262144
    Roughly 25 - 30 connections open (mostly idle) at any given time
(connection pools)

Both boxes are dedicated DB servers ... With the new configuration, we were
seeing swap rates of 1000-5000 KB/s (according to vmstat) ... with the old
configuration, we never saw any swapping ... I turned the shared_buffers and
sort_mem down on the new box to match the settings of the old box and found
that it reduced the swapping significantly (roughly 0-2000 KB/s) but didn't
eliminate it completely ... when looking at 'top' on the new box, the list
of postgres processes all seem to be indicating a different amount of memory
usage ... under the periods of heavy swapping, one or more of the postgres
processes would be way up there (between 500MB and 1000MB (which would
easily explain the swapping)) ... the question is: why aren't all of the
processes sharing the same pool of shared memory since I thought that's what
I'm doing when adjusting the shared_buffers property?

Here's an example of my 'top' (not under heavy load) demonstrating the
different memory usage by each postgres process ... I unfortunately don't
have the same snapshot data from the old configuration, but I seem to recall
that all of the postgres processes had

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
16966 postgres  15   0  107M 107M  105M S     0.0  5.3   0:39   1 postmaster
20198 postgres  15   0 40448  39M 37752 S     0.2  1.9   0:07   0 postmaster
18801 postgres  15   0 21932  21M 19616 S     0.0  1.0   0:01   0 postmaster
19210 postgres  16   0 21276  20M 19008 S     0.0  1.0   0:01   0 postmaster
19507 postgres  15   0 15504  14M 13580 S     0.0  0.7   0:00   3 postmaster
20308 postgres  15   0 12832  12M 11248 S     0.0  0.6   0:00   3 postmaster
20456 postgres  15   0 12500  12M 10920 S     0.0  0.6   0:00   1 postmaster
20403 postgres  15   0 11572  11M  9928 S     0.2  0.5   0:00   1 postmaster
20251 postgres  15   0 10796  10M  9260 S     0.0  0.5   0:00   0 postmaster
20398 postgres  15   0 10792  10M  9256 S     0.0  0.5   0:00   2 postmaster
20306 postgres  21   0  9100 8808  7796 S     0.0  0.4   0:00   1 postmaster
20425 postgres  16   0  9100 8808  7796 S     0.0  0.4   0:00   0 postmaster
20360 postgres  15   0  9096 8804  7792 S     0.0  0.4   0:00   3 postmaster
20383 postgres  21   0  9096 8804  7792 S     0.0  0.4   0:00   0 postmaster
20434 postgres  21   0  9096 8804  7792 S     0.0  0.4   0:00   1 postmaster
20305 postgres  15   0  9108 8796  7804 S     0.0  0.4   0:00   2 postmaster

Can anyone think of a reason as to why I'm seeing such heavy swapping?
According to Bruce Momjian's performance tuning guide, he recommends roughly
half the amount of physical RAM for the shared_buffers ... I tried turning
UP the shared_buffers even higher (to 180,000 i believe; roughly 1.5GB) and
that seemed to make the problem even worse ...

Thanks in advance,
Dave