Re: postgres_fdw: Use COPY to speed up batch inserts - Mailing list pgsql-hackers

From Kirill Reshke
Subject Re: postgres_fdw: Use COPY to speed up batch inserts
Date
Msg-id CALdSSPhi0ca0zEz3qTBYeCg-J=R72uc4dZF-_0XDf9U3q1RDPw@mail.gmail.com
Whole thread
In response to postgres_fdw: Use COPY to speed up batch inserts  ("Matheus Alcantara" <matheusssilv97@gmail.com>)
Responses Re: blkreftable.c needs more sanity checks
List pgsql-hackers
On Thu, 10 Sept 2026 at 01:10, I <reshkekirill@gmail.com> wrote:

>

>
> By the way I noticed that batch_size is practically always no more
> than 1000 in batch insert callback (even though configured to a bigger
> value in relation options.). I didn't look for the exact reason.
>

I did actually take a fast look:

```
   /*
* Send the buffered tuples to the FDW in batches of at most
* batch_size, as we do for ExecForeignBatchInsert. Each call is a
* self-contained COPY operation.
*
* COPY provides no RETURNING, so this path is only usable when
* there are no AFTER ROW triggers that would need the stored rows.
*/
while (sent < nused)
{
int size = (batch_size < nused - sent) ? batch_size : (nused - sent);

resultRelInfo->ri_FdwRoutine->ExecForeignBatchCopy(estate,
  resultRelInfo,
  &slots[sent],
  size);

sent += size;

/* Update the row counter and progress of the COPY command */
*processed += size;
pgstat_progress_update_param(PROGRESS_COPY_TUPLES_PROCESSED,
*processed);
}
```

In this place, big batch_size (1e5-1e6) is always capped with 1000 by
CopyMultiInsertBuffer logic. At least for me this is the case. Maybe
this is another place for adjusting in sight of this feature.



-- 
Best regards,
Kirill Reshke



pgsql-hackers by date:

Previous
From: Andrew Dunstan
Date:
Subject: Re: pg_get_*_ddl() needs a redesign
Next
From: Nathan Bossart
Date:
Subject: Re: Speed up COPY FROM text/CSV parsing using SIMD