Re: [PATCH] Initial progress reporting for COPY command - Mailing list pgsql-hackers

From Josef Šimánek
Subject Re: [PATCH] Initial progress reporting for COPY command
Date
Msg-id CAFp7QwqZqTd3t7C6UUmgj+HEbYdc+HTo1Xs35yC11uMkduKzJQ@mail.gmail.com
Whole thread Raw
In response to Re: [PATCH] Initial progress reporting for COPY command  (Michael Paquier <michael@paquier.xyz>)
List pgsql-hackers


po 15. 6. 2020 v 2:18 odesílatel Michael Paquier <michael@paquier.xyz> napsal:
Hi Josef,

On Sun, Jun 14, 2020 at 02:32:33PM +0200, Josef Šimánek wrote:
> Hello, as proposed by Pavel Stěhule and discussed on local czech PostgreSQL
> maillist (
> https://groups.google.com/d/msgid/postgresql-cz/CAFj8pRCZ42CBCa1bPHr7htffSV%2BNAcgcHHG0dVqOog4bsu2LFw%40mail.gmail.com?utm_medium=email&utm_source=footer),
> I have prepared an initial patch for COPY command progress reporting.

Sounds like a good idea to me.

Great. I will continue working on this.
 
> I have not found any tests for progress reporting. Are there any? It would
> need two backends running (one running COPY, one checking output of report
> view). Is there any similar test I can inspire at? In theory, it should be
> possible to use dblink_send_query to run async COPY command in the
> background.

We don't have any tests in core.  I think that making deterministic
test cases is rather tricky here as long as we don't have a more
advanced testing framework that allows is to lock certain code paths
and keep around an expected state until a second session comes around
and looks at the progress catalog (even that would need adding more
code to core to mark the extra point looked at).  So I think that it is
fine to not focus on that for this feature.  The important parts are
the choice of the progress points and the data sent to MyProc, and
both should be chosen wisely.

Thanks for the info. I'm focusing exactly at looking for right spots to report the progress. I'll attach new patch with better places and supporting more options of reporting (including STDIN, STDOUT) soon and also I'll try to add it to commitfest.
 

> My initial (attached) patch also doesn't introduce documentation for this
> system view. I can add that later once this patch is finalized (if that
> happens).

You may want to add it to the next commit fest:
https://commitfest.postgresql.org/28/
Documentation is necessary, and having some would ease reviews.
--
Michael

pgsql-hackers by date:

Previous
From: Tomas Vondra
Date:
Subject: Re: suggest to rename enable_incrementalsort
Next
From: Josef Šimánek
Date:
Subject: Re: [PATCH] Initial progress reporting for COPY command