Thread: pg_restore mostly idle on restoring a large number of tables

On 7/13/23 02:41, Boris Sagadin wrote:
> Hi,
>
> restoring a 1.5TB database with about 800k tables on i3.4xlarge AWS 
> instace, PgSQL V12.15 on Ubuntu.
>
> Running pg_restore with -j 16, I noticed the pg_restore is busy for an 
> hour or so with IO at 80%+ and then most of processes start idling and 
> only a few doing some work, disk IO at 1-2%, pg_stat_activity is mostly 
> idle, same goes for CPU, and this state proceeds for further 6 hours, disk 
> space increases very slowly.
>
> I thought because of a lot of small tables, number of workers should be 
> increased to increase parallel efficiency, so I tried with -j 128. The 
> situation was somewhat better, but most of the workers start idling, again 
> disk IO lowers to about 4% util, CPU util goes to about 4%, too.
>
> Stracing workers produces the perpetual read call on most pg_restore workers:
>
> # strace -p 59567
> strace: Process 59567 attached
> read(3,
>
> With only about 10 or so (out of 128) workers doing some actual work:
>
> strace -p 59367 -e sendto
> strace: Process 59367 attached
> sendto(4, "Q\0\0\0\263ALTER TABLE ONLY raw.space0"..., 180, MSG_NOSIGNAL, 
> NULL, 0) = 180
> sendto(4, "Q\0\0\0.TRUNCATE TABLE ONLY raw.spa"..., 47, MSG_NOSIGNAL, 
> NULL, 0) = 47
> sendto(4, "Q\0\0\0\205COPY raw.space000448117 (da"..., 134, MSG_NOSIGNAL, 
> NULL, 0) = sendto(4, "Q\0\0\0\vCOMMIT\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
> sendto(4, "Q\0\0\0pCREATE INDEX idx_space00044"..., 113, MSG_NOSIGNAL, 
> NULL, 0) = 113
> sendto(4, "Q\0\0\0\263ALTER TABLE ONLY raw.space0"..., 180, MSG_NOSIGNAL, 
> NULL, 0) = 180
> ...
> .. some lines omitted
>
> I would think that all workers would proceed with creating indexes and 
> doing some useful work until the restore is finished completely?
>
> Most of the tables are very small, 2 indexes per table and without any 
> foreign references etc., we have a multi tenant environment.

I always run pg_dump and pg_restore with "--verbose", and redirect stdout & 
stderr to a log file.  Then I "tail -f" that log file. It'll tell you what's 
happening.

Also, iotop is quite useful.


-- 
Born in Arizona, moved to Babylonia.

Boris Sagadin <boris@infosplet.com> writes:
> restoring a 1.5TB database with about 800k tables on i3.4xlarge AWS
> instace, PgSQL V12.15 on Ubuntu.

> Running pg_restore with -j 16, I noticed the pg_restore is busy for an hour
> or so with IO at 80%+ and then most of processes start idling and only a
> few doing some work, disk IO at 1-2%, pg_stat_activity is mostly idle, same
> goes for CPU, and this state proceeds for further 6 hours, disk space
> increases very slowly.

Yeah, this seems like a performance bug in pg_restore, per analysis at
[1].  Until somebody gets around to doing something about that, I'd
counsel not bothering with parallelized restore for this many tables.

If the tables are mostly small, it'd likely be more useful to use
--single-transaction mode; though you'd have to make sure that
max_locks_per_transaction is set high enough to allow creating all
those tables in one transaction.

            regards, tom lane

[1] https://www.postgresql.org/message-id/3612876.1689443232%40sss.pgh.pa.us