pgsql: Add ideas for concurrent pg_dump and pg_restore: < * pg_dump > * - Mailing list pgsql-committers

From momjian@postgresql.org (Bruce Momjian)
Subject pgsql: Add ideas for concurrent pg_dump and pg_restore: < * pg_dump > *
Date
Msg-id 20080304013332.DC7D1754108@cvs.postgresql.org
Whole thread Raw
List pgsql-committers
Log Message:
-----------
Add ideas for concurrent pg_dump and pg_restore:

< * pg_dump
> * pg_dump / pg_restore
>     o Allow pg_dump to utilize multiple CPUs and I/O channels by dumping
>       multiple objects simultaneously
>
>       The difficulty with this is getting multiple dump processes to
>       produce a single dump output file.
>       http://archives.postgresql.org/pgsql-hackers/2008-02/msg00205.php
>
>     o Allow pg_restore to utilize multiple CPUs and I/O channels by
>           restoring multiple objects simultaneously
>
>       This might require a pg_restore flag to indicate how many
>       simultaneous operations should be performed.  Only pg_dump's
>       -Fc format has the necessary dependency information.
>
>     o To better utilize resources, restore data, primary keys, and
>        indexes for a single table before restoring the next table
>
>       Hopefully this will allow the CPU-I/O load to be more uniform
>       for simultaneous restores.  The idea is to start data restores
>       for several objects, and once the first object is done, to move
>       on to its primary keys and indexes.  Over time, simultaneous
>       data loads and index builds will be running.
>
>     o To better utilize resources, allow pg_restore to check foreign
>       keys simultaneously, where possible
>     o Allow pg_restore to create all indexes of a table
>       concurrently, via a single heap scan
>
>       This requires a pg_dump -Fc file because that format contains
>           the required dependency information.
>       http://archives.postgresql.org/pgsql-general/2007-05/msg01274.php
>
>     o Allow pg_restore to load different parts of the COPY data
>       simultaneously
<   single heap scan, and have a restore of a pg_dump somehow use it
>   single heap scan, and have pg_restore use it
<   http://archives.postgresql.org/pgsql-general/2007-05/msg01274.php

Modified Files:
--------------
    pgsql/doc:
        TODO (r1.2251 -> r1.2252)
        (http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/doc/TODO?r1=1.2251&r2=1.2252)
    pgsql/doc/src/FAQ:
        TODO.html (r1.760 -> r1.761)
        (http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/doc/src/FAQ/TODO.html?r1=1.760&r2=1.761)

pgsql-committers by date:

Previous
From: nanardon@pgfoundry.org (User Nanardon)
Date:
Subject: pgrpm - pgrpm: - no promote epoch in version comparison (otherwise 4.4.8
Next
From: momjian@postgresql.org (Bruce Momjian)
Date:
Subject: pgsql: Add URL for: o Allow pg_restore to utilize multiple CPUs