pg_dump incredibly slow dumping a single schema from a large db - Mailing list pgsql-general

From Mike Roest
Subject pg_dump incredibly slow dumping a single schema from a large db
Date
Msg-id CAE7ByhiroJbXUzAJMFE0mVhOOcyYLGONAj83dkrXFkG5-H-rZg@mail.gmail.com
Whole thread Raw
Responses Re: pg_dump incredibly slow dumping a single schema from a large db  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-general
Hey Everyone,
    I've got an interesting issue.  We're running postgres 9.1.1 linux x64 centos 5.8

aspdata=# select version();
                                                    version
---------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.1.1 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-52), 64-bit


We currently have 1 DB we use for multiple independent tenant schemas. The database size is current 56227005240 bytes as reported by pg_database_size.  

There are 557 schemas each with about 1300 objects (760 tables 520 views).

We are using pg_dump to do backups of a single schema with a total size of (5480448 bytes calculated with  SELECT sum(pg_relation_size(schemaname || '.' || tablename))::bigint FROM pg_tables WHERE schemaname ='miketest';)  

pg_dump -f /dumps/test.backup -Fc -n miketest aspdata

This dump is currently taking around 8 minutes.  While dumping the pg_dump process is using 100% of one core in the server (24 core machine).  Doing a -v pg_dump I found that the following stages are taking the majority of the time 

reading user_defined tables (2 minutes and 20 seconds)
reading dependency data (5 minutes and 30 seconds)

The size of the schema doesn't really seem to effect theses times are almost identical for a 700 meg schema as well (obviously the data dump portion takes longer with the bigger db)

During the reading user_defined tables the following query shows up for a 10-20 seconds then the pg_dump connection sits idle for the rest of the 2 minutes:

SELECT c.tableoid, c.oid, c.relname, c.relacl, c.relkind, c.relnamespace, (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = c.relowner) AS rolname, c.relchecks, c.relhastriggers, c.relhasindex, c.relhasrules, c.relhasoids, c.relfrozenxid, tc.oid AS
toid, tc.relfrozenxid AS tfrozenxid, c.relpersistence, CASE WHEN c.reloftype <> 0 THEN c.reloftype::pg_catalog.regtype ELSE NULL END AS reloftype, d.refobjid AS owning_tab, d.refobjsubid AS owning_col, (SELECT spcname FROM pg_tablespace
t WHERE t.oid = c.reltablespace) AS reltablespace, array_to_string(c.reloptions, ', ') AS reloptions, array_to_string(array(SELECT 'toast.' || x FROM unnest(tc.reloptions) x), ', ') AS toast_reloptions FROM pg_class c LEFT JOIN pg_depend
 d ON (c.relkind = 'S' AND d.classid = c.tableoid AND d.objid = c.oid AND d.objsubid = 0 AND d.refclassid = c.tableoid AND d.deptype = 'a') LEFT JOIN pg_class tc ON (c.reltoastrelid = tc.oid) WHERE c.relkind in ('r', 'S', 'v', 'c', 'f')
ORDER BY c.oid

During the reading dependency data the following queries show up for a few seconds then the connection sits idle for the rest of the 5.5 minutes:
 SELECT tableoid
, oid, typname, typnamespace, (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = typowner) AS rolname, typinput::oid AS typinput, typoutput::oid AS typoutput, typelem, typrelid, CASE WHEN typrelid = 0 THEN ' '::"char" ELSE (SELECT relk
ind FROM pg_class WHERE oid = typrelid) END AS typrelkind, typtype, typisdefined, typname[0] = '_' AND typelem != 0 AND (SELECT typarray FROM pg_type te WHERE oid = pg_type.typelem) = oid AS isarray FROM pg_type
SELECT classid,
 objid, refclassid, refobjid, deptype FROM pg_depend WHERE deptype != 'p' AND deptype != 'e' ORDER BY 1,2

Non production servers with less schemas don't seem to have any issue and perform the same dump in under 10 seconds on much lower classed hardware.

Server Specs:

2 x Intel Xeon X5650
32 Gigs of Ram
DELL Perc H700 Controller 
Data drive - 6XSAS2 15K in RAID10 FS: xfs
Log Drive - 2XSAS2 15K in RAID1 FS: xfs

There are 2 of these machine one master other slaved via streaming replication over gigabit network.


Thanks

pgsql-general by date:

Previous
From: Katherine Jeschke
Date:
Subject: Surge 2012 CFP is Open!
Next
From: Tom Lane
Date:
Subject: Re: pg_dump incredibly slow dumping a single schema from a large db