Re: Adding pg_dump flag for parallel export to pipes - Mailing list pgsql-hackers

From Andrew Jackson
Subject Re: Adding pg_dump flag for parallel export to pipes
Date
Msg-id 175665500252.297107.3384642490431951825.pgcf@coridan.postgresql.org
Whole thread Raw
In response to Re: Adding pg_dump flag for parallel export to pipes  (Andrew Jackson <andrewjackson947@gmail.com>)
List pgsql-hackers
Hi,

Went ahead and experimented with your patch a bit. To answer my previous question this patch can be used to pipe
pg_dumpdirectly into pg_restore. This should absolutely be added as another use case to your list above as it is a well
knownlimitation that you can use pg_dump/psql to do buffered copy but only with a single process, while using
pg_dump/pg_restoreis capable of multiprocessed copy but it must be saved to disk in its entirety before the restore can
begin.This is extremely frustrating when dealing with large databases where you don't want multiple copies saved on
diskand because it's not as fast as it can be. With this patch you can get the best of both worlds. 
 

 Example dump
```bash
pg_dump --jobs=4 -Fd "${connection_str}" --pipe-command="mkfifo dumpdir/%f; cat >> dumpdir/%f"
```

Example restore run in different process
```bash
pg_restore --jobs=4 -Fd --dbname="${another_connection_str}" ./dumpdir
```
Thanks,
Andrew Jackson

pgsql-hackers by date:

Previous
From: Mihail Nikalayeu
Date:
Subject: Re: Adding REPACK [concurrently]
Next
From: Arseniy Mukhin
Date:
Subject: Use streaming read I/O in BRIN vacuuming