pgsql: Add parallel pg_dump option. - Mailing list pgsql-committers

From Andrew Dunstan
Subject pgsql: Add parallel pg_dump option.
Date
Msg-id E1UJn1M-0001uh-Nw@gemulon.postgresql.org
Whole thread Raw
List pgsql-committers
Add parallel pg_dump option.

New infrastructure is added which creates a set number of workers
(threads on Windows, forked processes on Unix). Jobs are then
handed out to these workers by the master process as needed.
pg_restore is adjusted to use this new infrastructure in place of the
old setup which created a new worker for each step on the fly. Parallel
dumps acquire a snapshot clone in order to stay consistent, if
available.

The parallel option is selected by the -j / --jobs command line
parameter of pg_dump.

Joachim Wieland, lightly editorialized by Andrew Dunstan.

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/9e257a181cc1dc5e19eb5d770ce09cc98f470f5f

Modified Files
--------------
doc/src/sgml/backup.sgml              |   18 +
doc/src/sgml/perform.sgml             |    9 +
doc/src/sgml/ref/pg_dump.sgml         |   89 +++-
src/bin/pg_dump/Makefile              |    2 +-
src/bin/pg_dump/compress_io.c         |   10 +
src/bin/pg_dump/dumputils.c           |   86 ++-
src/bin/pg_dump/dumputils.h           |   13 +-
src/bin/pg_dump/parallel.c            | 1293 +++++++++++++++++++++++++++++++++
src/bin/pg_dump/parallel.h            |   85 +++
src/bin/pg_dump/pg_backup.h           |   11 +-
src/bin/pg_dump/pg_backup_archiver.c  |  735 +++++++------------
src/bin/pg_dump/pg_backup_archiver.h  |   35 +-
src/bin/pg_dump/pg_backup_custom.c    |   88 +++-
src/bin/pg_dump/pg_backup_db.c        |   20 +-
src/bin/pg_dump/pg_backup_directory.c |  264 +++++++-
src/bin/pg_dump/pg_backup_tar.c       |    8 +-
src/bin/pg_dump/pg_dump.c             |  681 ++++++++++--------
src/bin/pg_dump/pg_dump.h             |    3 +
src/bin/pg_dump/pg_dump_sort.c        |   92 +++-
src/bin/pg_dump/pg_dumpall.c          |   20 +-
src/bin/pg_dump/pg_restore.c          |   17 +-
src/tools/msvc/Mkvcbuild.pm           |    5 +
22 files changed, 2765 insertions(+), 819 deletions(-)


pgsql-committers by date:

Previous
From: Tom Lane
Date:
Subject: pgsql: Update time zone abbreviation lists for changes missed since 200
Next
From: Tom Lane
Date:
Subject: pgsql: Fix some unportable constructs in parallel pg_dump code.