Postgres Pro
Downloads
Home   >   mailing lists

Thread: Gauging progress of COPY?

Gauging progress of COPY?

From
Wells Oliver
Date:
Is there some clever way to gauge the progress of a COPY command? i.e. you know it has 10m rows, but you can't SELECT COUNT(*).

--

Re: Gauging progress of COPY?

From
Tom Lane
Date:
Wells Oliver <wells.oliver@gmail.com> writes:
> Is there some clever way to gauge the progress of a COPY command? i.e. you
> know it has 10m rows, but you can't SELECT COUNT(*).

If it's a COPY IN (to a table) you could monitor growth of the
physical size of the table, cf pg_relation_size().  This isn't
terribly satisfactory if you're not clear on how much space each
row will take, but it's something.

            regards, tom lane

Re: Gauging progress of COPY?

From
Ron
Date:
On 1/20/22 12:00 PM, Wells Oliver wrote:
> Is there some clever way to gauge the progress of a COPY command? i.e. you 
> know it has 10m rows, but you can't SELECT COUNT(*).

Use a specific tool like pg_bulkload.

-- 
Angular momentum makes the world go 'round.

Re: Gauging progress of COPY?

From
Stephen Frost
Date:
Greetings,

* Ron (ronljohnsonjr@gmail.com) wrote:
> On 1/20/22 12:00 PM, Wells Oliver wrote:
> >Is there some clever way to gauge the progress of a COPY command? i.e. you
> >know it has 10m rows, but you can't SELECT COUNT(*).
>
> Use a specific tool like pg_bulkload.

Or maybe just use pv if you're doing it through psql or such.

Thanks,

Stephen
Attachment

Re: Gauging progress of COPY?

From
Magnus Hagander
Date:


On Thu, Jan 20, 2022, 19:01 Wells Oliver <wells.oliver@gmail.com> wrote:
Is there some clever way to gauge the progress of a COPY command? i.e. you know it has 10m rows, but you can't SELECT COUNT(*).


If you can open a second connection, you can just query pg_stat_progress_copy. 

/Magnus 

Re: Gauging progress of COPY?

From
Greg Spiegelberg
Date:
On Sat, Jan 22, 2022, 3:09 AM Magnus Hagander <magnus@hagander.net> wrote:


On Thu, Jan 20, 2022, 19:01 Wells Oliver <wells.oliver@gmail.com> wrote:
Is there some clever way to gauge the progress of a COPY command? i.e. you know it has 10m rows, but you can't SELECT COUNT(*).


If you can open a second connection, you can just query pg_stat_progress_copy. 

If COPY FROM STDIN the client can provide feedback since it is sending line by line. 
If size known...1%...5%...
If number of rows known... 10000... 20000...

I would think even a semi capable bash script could do it. 

-Greg