Re: Warn when parallel restoring a custom dump without data offsets - Mailing list pgsql-hackers

From Justin Pryzby
Subject Re: Warn when parallel restoring a custom dump without data offsets
Date
Msg-id 20200622184048.GX17995@telsasoft.com
Whole thread Raw
In response to Re: Warn when parallel restoring a custom dump without data offsets  (Justin Pryzby <pryzby@telsasoft.com>)
List pgsql-hackers
Adding Jim, since he ask about helping with perl.

You can read the history of the patch here:

https://commitfest.postgresql.org/28/2568/
https://www.postgresql.org/message-id/flat/CALBH9DDuJ+scZc4MEvw5uO-=vRyR2=QF9+Yh=3hPEnKHWfS81A@mail.gmail.com

Some context:

David is adding a test case for a bugfix he made.
We want to test parallel pg_restore of a pg_dump created with a nonseekable FD.
I suggested to make the patch smaller by creating a nonseekable FD using a pipe.

The postgres test code is using IPC::Run, but currently passing a single
argument.  I think we want to change that to pass *multiple* arguments, so we
can use '>' and/or '|'.  I have a patch which partially works, and another patch
which I didn't try very hard to make work, since the first part already took me
a long time..

You'll want to start with a git checkout and do:
time ./configure --enable-tap-tests ...

And apply David's patches from the above thread.  The goal is to make a test
case that fails without his patch and passes with it.  And maybe apply my
patches if they're useful.

I've been running the pg_dump checks like this:
time make -C src/bin/pg_dump check

On Sun, Jun 21, 2020 at 02:42:25PM -0500, Justin Pryzby wrote:
> On Sun, Jun 21, 2020 at 03:18:58PM -0400, David Gilman wrote:
> > Thank you for taking a stab at the perl thing. I took the question to
> > StackOverflow, I haven't yet looped back to try their suggestion but I
> > think there is hope by messing with the Perl references.
> > https://stackoverflow.com/questions/62086173/using-the-right-perl-array-references-with-ipcrun
> 
> I finally got this to work using IPC:Run's '>' redirection operator, but it
> seems like that opens a file which *is* seekable, so doesn't work for this
> purpose.  Since "cat" isn't portable (duh), the next best thing seems to be to
> pipe to perl -pe '' >ouput.  Otherwise maybe your way of adding an
> --disable-seeking option is best.
> 
> See if you can do anything with the attached.

[fixed patch I sent to David offlist]

[and another patch which doesn't work yet]

-- 
Justin

PS. The patches are named *.txt so that the patch tester doesn't try to test
them, as they're known to be incomplete.

Attachment

pgsql-hackers by date:

Previous
From: Alvaro Herrera
Date:
Subject: Re: Asymmetry in opening and closing indices for partition routing
Next
From: Surafel Temesgen
Date:
Subject: Decomposing xml into table