Re: [PoC PATCH] Parallel dump to /dev/null - Mailing list pgsql-hackers

From Michael Banck
Subject Re: [PoC PATCH] Parallel dump to /dev/null
Date
Msg-id 20180228132840.GD13784@nighthawk.caipicrew.dd-dns.de
Whole thread Raw
In response to [PoC PATCH] Parallel dump to /dev/null  (Michael Banck <michael.banck@credativ.de>)
Responses Re: [PoC PATCH] Parallel dump to /dev/null
List pgsql-hackers
Hi,

On Thu, Feb 01, 2018 at 02:24:46PM +0100, Michael Banck wrote:
> dumping a database to /dev/null via pg_dump is (AFAIK) one recommended
> way to check for corruption.  However, dumping to /dev/null is currently
> not supported in directory mode which makes it not possible to dump to
> /dev/null in parallel.
> 
> I had a look at this, and it appears it would suffice to just override
> the few spots in pg_backup_directory.c which assemble filenames in the
> target directory to revert to '/dev/null' (or 'NUL' on Windows).
> 
> The attached proof-of-concept patch does that, and it seems to work; I'm
> getting some nice speedups on a 170 GB test database:
> 
> $ time pg_dump -Fc -Z0 -f /dev/null TESTDB
> real    32m45.120s
> [...]
> $ time pg_dump -Fd -j8 -Z0 -f /dev/null TESTDB
> real    9m28.357s

I have added this patch to the next commitfest:

https://commitfest.postgresql.org/17/1576/

I also tried to add a TAP test, but as this patch produces no dump
output, I had to exclude it from all restores tests and then got an
off-by-one between the number of tests that were expected vs. those that
were run. I've attached it if somebody wants to take a look, but I hope
with Stephen's refactoring of the pg_dump TAP tests, this might be
easier.


Michael

Attachment

pgsql-hackers by date:

Previous
From: Alexander Kuzmenkov
Date:
Subject: Re: [patch] BUG #15005: ANALYZE can make pg_class.reltuplesinaccurate.
Next
From: Amit Kapila
Date:
Subject: Re: Contention preventing locking