Thread: pg_dump issue
Good morning,
I have identical postgres installations running on identical machines. Dual Core AMD Opteron(tm) Processor 870 , 16GB RAM, Red Hat Linux 3.2.3-20 and 120GB worth of disk space on two drives.
Recently, I have noticed that my nightly backups take longer on one machine than on the other. I back up five (5) databases totaling 8.6GB in size. On Prod001 the backups take app. 7 minutes, on Prod002 the backups take app. 26 minutes! Quite a discrepancy. I checked myself than checked with our Engineering staff and have been assured that the machines are identical hardware wise, CPU, disk, etc.
Question; has anyone run into a similar issue? Here is the command I use for the nightly backup on both machines:
pg_dump -F c -f $DB.backup.$DATE $DB
Kind of scratching my head on this one....
Thank you,
Tim McElroy
"mcelroy, tim" <tim.mcelroy@bostonstock.com> writes:
> I have identical postgres installations running on identical machines. Dual
> Core AMD Opteron(tm) Processor 870 , 16GB RAM, Red Hat Linux 3.2.3-20 and
> 120GB worth of disk space on two drives.
> Recently, I have noticed that my nightly backups take longer on one machine
> than on the other. I back up five (5) databases totaling 8.6GB in size. On
> Prod001 the backups take app. 7 minutes, on Prod002 the backups take app. 26
> minutes! Quite a discrepancy.
Are the resulting backup files identical? Chasing down the reasons for
any diffs might yield some enlightenment.
One idea that comes to mind is that Prod002 is having performance
problems due to table bloat (maybe a missing vacuum cron job or
some such). A quick "du" on the two $PGDATA directories to check
for significant size differences would reveal this if so.
regards, tom lane
Thanks Tom. I have scheduled vacuums as follows and all have run without error.
Mon - Thu after-hours: vacuumdb -z -e -a -v On Fridays I add the -f option vacuumdb -z -e -a -v -f
The du . -h in $PGDATA showed PROD001 at 9.1G and Prod0002 at 8.8G so they're pretty much the same, one would think the smaller one should be faster. Yes, the backup files are identical in size. BTW - this is postgres 8.0.1. Stuck at this due to "that is the latest postgresql version certified by our vendor's application".
I'm hoping the Engineering staff can find something system related as I doubted and still doubt that it's a postgres issue.
Tim
-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Tuesday, May 30, 2006 11:16 AM
To: mcelroy, tim
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] pg_dump issue
"mcelroy, tim" <tim.mcelroy@bostonstock.com> writes:
> I have identical postgres installations running on identical machines. Dual
> Core AMD Opteron(tm) Processor 870 , 16GB RAM, Red Hat Linux 3.2.3-20 and
> 120GB worth of disk space on two drives.
> Recently, I have noticed that my nightly backups take longer on one machine
> than on the other. I back up five (5) databases totaling 8.6GB in size. On
> Prod001 the backups take app. 7 minutes, on Prod002 the backups take app. 26
> minutes! Quite a discrepancy.
Are the resulting backup files identical? Chasing down the reasons for
any diffs might yield some enlightenment.
One idea that comes to mind is that Prod002 is having performance
problems due to table bloat (maybe a missing vacuum cron job or
some such). A quick "du" on the two $PGDATA directories to check
for significant size differences would reveal this if so.
regards, tom lane
"mcelroy, tim" <tim.mcelroy@bostonstock.com> writes:
> The du . -h in $PGDATA showed PROD001 at 9.1G and Prod0002 at 8.8G so
> they're pretty much the same, one would think the smaller one should be
> faster. Yes, the backup files are identical in size.
Hmph. You should carry the "du" analysis down to the subdirectory
level, eg make sure that it's not a case of lots of pg_xlog bloat
balancing out bloat in a different area on the other system. But I
suspect you won't find anything.
> I'm hoping the Engineering staff can find something system related as I
> doubted and still doubt that it's a postgres issue.
I tend to agree. You might try watching "vmstat 1" output while taking
the dumps, so you could at least get a clue whether the problem is CPU
or I/O related ...
regards, tom lane
I did carry it down to the subdirectory level but only included the total for brevity. I'll paste the complete readout at the end of the email. I'll try the "vmstat 1" as you suggest next time the backups run. If the Eng staff finds anything I'll post the results and maybe save someone else some grief if they have the same issue. Thanks again for your input Tom.
Tim
PROD001 PROD002
220K ./global[PARA]4.0K ./pg_xlog/archive_status[PARA]529M ./pg_xlog[PARA]36K ./pg_clog[PARA]256K ./pg_subtrans[PARA]4.0K ./base/1/pgsql_tmp[PARA]4.8M ./base/1[PARA]4.8M ./base/17229[PARA]4.0K ./base/62878500/pgsql_tmp[PARA]4.8M ./base/62878500[PARA]5.5M ./base/1152695[PARA]4.0K ./base/67708567/pgsql_tmp[PARA]1.6G ./base/67708567[PARA]12K ./base/1157024/pgsql_tmp[PARA]6.3G ./base/1157024[PARA]4.0K ./base/1159370/pgsql_tmp[PARA]543M ./base/1159370[PARA]4.0K ./base/1157190/pgsql_tmp[PARA]164M ./base/1157190[PARA]4.0K ./base/1621391/pgsql_tmp[PARA]81M ./base/1621391[PARA]8.6G ./base[PARA]4.0K ./pg_tblspc[PARA]604K ./pg_log[PARA]9.1G . 220K ./global[PARA]4.0K ./pg_xlog/archive_status[PARA]529M ./pg_xlog[PARA]136K ./pg_clog[PARA]208K ./pg_subtrans[PARA]4.0K ./base/1/pgsql_tmp[PARA]4.9M ./base/1[PARA]4.8M ./base/17229[PARA]5.3M ./base/1274937[PARA]4.0K ./base/64257611/pgsql_tmp[PARA]1.6G ./base/64257611[PARA]4.0K ./base/71683200/pgsql_tmp[PARA]6.1G ./base/71683200[PARA]4.0K ./base/1281929/pgsql_tmp[PARA]478M ./base/1281929[PARA]4.0K ./base/58579022/pgsql_tmp[PARA]154M ./base/58579022[PARA]81M ./base/1773916[PARA]4.0K ./base/55667447/pgsql_tmp[PARA]4.8M ./base/55667447[PARA]8.3G ./base[PARA]4.0K ./pg_tblspc[PARA]588K ./pg_log[PARA]8.8G .
-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Tuesday, May 30, 2006 12:20 PM
To: mcelroy, tim
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] pg_dump issue
"mcelroy, tim" <tim.mcelroy@bostonstock.com> writes:
> The du . -h in $PGDATA showed PROD001 at 9.1G and Prod0002 at 8.8G so
> they're pretty much the same, one would think the smaller one should be
> faster. Yes, the backup files are identical in size.
Hmph. You should carry the "du" analysis down to the subdirectory
level, eg make sure that it's not a case of lots of pg_xlog bloat
balancing out bloat in a different area on the other system. But I
suspect you won't find anything.
> I'm hoping the Engineering staff can find something system related as I
> doubted and still doubt that it's a postgres issue.
I tend to agree. You might try watching "vmstat 1" output while taking
the dumps, so you could at least get a clue whether the problem is CPU
or I/O related ...
regards, tom lane
try to dump-restore your 'slow' database, this might help if your db or filesystem gets too fragmented. On Tue, 30 May 2006 10:31:08 -0400 "mcelroy, tim" <tim.mcelroy@bostonstock.com> wrote: > Good morning, > > I have identical postgres installations running on identical machines. Dual > Core AMD Opteron(tm) Processor 870 , 16GB RAM, Red Hat Linux 3.2.3-20 and > 120GB worth of disk space on two drives. > > Recently, I have noticed that my nightly backups take longer on one machine > than on the other. I back up five (5) databases totaling 8.6GB in size. On > Prod001 the backups take app. 7 minutes, on Prod002 the backups take app. 26 > minutes! Quite a discrepancy. I checked myself than checked with our > Engineering staff and have been assured that the machines are identical > hardware wise, CPU, disk, etc. > > Question; has anyone run into a similar issue? Here is the command I use > for the nightly backup on both machines: > > pg_dump -F c -f $DB.backup.$DATE $DB > > Kind of scratching my head on this one.... > > Thank you, > Tim McElroy > > -- Evgeny Gridasov Software Engineer I-Free, Russia