Thread: Restore of pg_dump taking a long time...
Hi, I'm currently restoring a fairly large DB from a pg_dump and it's taking about 12 hours to finish. The main part of this time is spent creating indexes. Is there anyway I can speed up the restore process, or do i just have to wait? I'm using postgres 8.1.3 on freebsd and the pg_dump is gzip'd. I'm restoring with the following command : "zcat <gzip'd db> | psql -U postgres <destination db>" Regards, Ruairi Carroll
"Ruairi" <rcarroll@bluemetrix.com> writes:
> I'm currently restoring a fairly large DB from a pg_dump and it's taking
> about 12 hours to finish. The main part of this time is spent creating
> indexes. Is there anyway I can speed up the restore process, or do i just
> have to wait?
There's not much you can do to improve the performance of a restore
already in progress. Next time you might want to think about kicking up
maintenance_work_mem before you start it; that's about the only thing
you can really do to speed up index builds.
(Actually, if there are lots of indexes yet to be built, increasing
the setting in postgresql.conf and SIGHUP'ing the postmaster could
be worth doing, as that should affect the builds yet to be done.)
regards, tom lane
I have found that the following steps have increased the speed of my restores:
1. dropdb <DBNAME>
2. createdb <DBANME>
3. Increase maintenance_work_mem as Tom mentioned. I do this at restore/runtime
4. Increase work_mem as Tom mentioned. I do this at restore/runtime
5. renice -20 -p <pid #> (on a Linux box) <= This step can be performed while the restore is running but you need root privilege
Tim
-----Original Message-----
From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Tom Lane
Sent: Wednesday, May 24, 2006 2:48 PM
To: Ruairi
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Restore of pg_dump taking a long time...
"Ruairi" <rcarroll@bluemetrix.com> writes:
> I'm currently restoring a fairly large DB from a pg_dump and it's taking
> about 12 hours to finish. The main part of this time is spent creating
> indexes. Is there anyway I can speed up the restore process, or do i just
> have to wait?
There's not much you can do to improve the performance of a restore
already in progress. Next time you might want to think about kicking up
maintenance_work_mem before you start it; that's about the only thing
you can really do to speed up index builds.
(Actually, if there are lots of indexes yet to be built, increasing
the setting in postgresql.conf and SIGHUP'ing the postmaster could
be worth doing, as that should affect the builds yet to be done.)
regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster
On Wed, 2006-05-24 at 13:56, mcelroy, tim wrote: > I have found that the following steps have increased the speed of my > restores: > 1. dropdb <DBNAME> > 2. createdb <DBANME> > 3. Increase maintenance_work_mem as Tom mentioned. I do this at > restore/runtime > 4. Increase work_mem as Tom mentioned. I do this at restore/runtime > 5. renice -20 -p <pid #> (on a Linux box) <= This step can be > performed while the restore is running but you need root privilege Note that if there's no other data in the database that you need to worry about, you can also restore with fsync off. I.e. if a database cluster could just be re-inited should a power failure occur. If you've got already important data you can't afford to lose in the cluster, then do not turn off fsync. Also, remember to turn it back on before you go into production.
"Todd A. Cook" <tcook@blackducksoftware.com> writes:
> I have found that increasing maintenance_work_mem can decrease index
> build speed on large tables:
You should probably re-measure when 8.2 comes out; we've fixed a number
of performance issues in the sorting code that might cause that.
regards, tom lane
Tom Lane wrote: > "Todd A. Cook" <tcook@blackducksoftware.com> writes: >> I have found that increasing maintenance_work_mem can decrease index >> build speed on large tables: > > You should probably re-measure when 8.2 comes out; we've fixed a number > of performance issues in the sorting code that might cause that. Thanks. I'll do that. If I can, I'll try it sooner on a build from CVS. -- todd
I would typically drop the indicies before taking the dump, and recreate in the new instance. I know the loads go alot faster that way. Perhaps the index creation would be the same. Tom Lane wrote: > "Ruairi" <rcarroll@bluemetrix.com> writes: >> I'm currently restoring a fairly large DB from a pg_dump and it's taking >> about 12 hours to finish. The main part of this time is spent creating >> indexes. Is there anyway I can speed up the restore process, or do i just >> have to wait? > > There's not much you can do to improve the performance of a restore > already in progress. Next time you might want to think about kicking up > maintenance_work_mem before you start it; that's about the only thing > you can really do to speed up index builds. > > (Actually, if there are lots of indexes yet to be built, increasing > the setting in postgresql.conf and SIGHUP'ing the postmaster could > be worth doing, as that should affect the builds yet to be done.) > > regards, tom lane > > ---------------------------(end of broadcast)--------------------------- > TIP 2: Don't 'kill -9' the postmaster -- ---------------------------------------------------------------------------- Naomi Walker Chief Information Officer Eldorado Computing, Inc. nwalker@eldocomp.com 602-604-3100 ---------------------------------------------------------------------------- What lies behind us and what lies before us are tiny matters compared to what lies within us. - William Morrow ---------------------------------------------------------------------------- -- CONFIDENTIALITY NOTICE -- Information transmitted by this e-mail is proprietary to MphasiS and/or its Customers and is intended for use only by theindividual or entity to which it is addressed, and may contain information that is privileged, confidential or exemptfrom disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwardedto you without proper authority, you are notified that any use or dissemination of this information in any manneris strictly prohibited. In such cases, please notify us immediately at mailmaster@mphasis.com and delete this mailfrom your records.
Tom Lane wrote: > > There's not much you can do to improve the performance of a restore > already in progress. Next time you might want to think about kicking up > maintenance_work_mem before you start it; that's about the only thing > you can really do to speed up index builds. I have found that increasing maintenance_work_mem can decrease index build speed on large tables: maintenance_work_mem 32768 262144 1048576 2097151 --------------------------------------------------------------- index 400M rows (min) 55 61 68 70 index 45M rows (min) 4.03 4.22 4.88 2.55 These timings were done on a machine with 8GB ram and postgresql 8.1.3 (from the RPMs on postgresql.org). Each time is an average of three runs; before each run, postgres was shut down and the filesystem with /var/lib/pgsql was remounted to clear the filesystem cache. I'm assuming that the times are increasing because there is less memory available for the filesystem cache. Similarly, I assume that the runs on the smaller table with maintenance_work_mem = 2097151 are faster because everything fits in that space. -- todd
On May 24, 2006, at 2:56 PM, Todd A. Cook wrote: > Tom Lane wrote: >> "Todd A. Cook" <tcook@blackducksoftware.com> writes: >>> I have found that increasing maintenance_work_mem can decrease index >>> build speed on large tables: >> You should probably re-measure when 8.2 comes out; we've fixed a >> number >> of performance issues in the sorting code that might cause that. > > Thanks. I'll do that. If I can, I'll try it sooner on a build > from CVS. If you'll be messing around with CVS, you might also want to try this patch: http://jim.nasby.net/misc/pgsqlcompression/compress-sort.patch (which was written by someone else). It hacks compression into the on- disk sort code, which has shown a 50% speed improvement on my machine. It should be fine to use for loading a database, but you wouldn't want to leave it in for serious use (IIRC there's some cases it flat-out doesn't handle right now). You could probably apply that to 8.1.4 as well if you wanted to; it should be fine for just loading the database. -- Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461