Re: Allowing parallel pg_restore from pipe - Mailing list pgsql-hackers

From Joachim Wieland
Subject Re: Allowing parallel pg_restore from pipe
Date
Msg-id CACw0+12denuxfkMk08BwhnTiXrYmy4z8pXK8yUZViQDDUXk4hg@mail.gmail.com
Whole thread Raw
In response to Re: Allowing parallel pg_restore from pipe  (Stefan Kaltenbrunner <stefan@kaltenbrunner.cc>)
Responses Re: Allowing parallel pg_restore from pipe
Re: Allowing parallel pg_restore from pipe
Re: Allowing parallel pg_restore from pipe
List pgsql-hackers
On Wed, Apr 24, 2013 at 4:05 PM, Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> wrote:
> What might make sense is something like pg_dump_restore which would have
> no intermediate storage at all, just pump the data etc from one source
> to another in parallel. But I pity the poor guy who has to write it :-)

hmm pretty sure that Joachims initial patch for parallel dump actually
had a PoC for something very similiar to that...

That's right, I implemented that as an own output format and named it "migrator" I think, which wouldn't write each stream to a file as the directory output format does but that instead pumps it back into a restore client.

Actually I think the logic was even reversed, it was a parallel restore that got the data from internally calling pg_dump functionality instead of from reading files... The neat thing about this approach was that the order was optimized and correct, i.e. largest tables start first and dependencies get resolved in the right order.

I could revisit that patch for 9.4 if enough people are interested.

Joachim

pgsql-hackers by date:

Previous
From: Simon Riggs
Date:
Subject: Re: Enabling Checksums
Next
From: Claudio Freire
Date:
Subject: Re: Allowing parallel pg_restore from pipe