Postgres Pro
Downloads
Home   >   mailing lists

Re: speed up full table scan using psql - Mailing list pgsql-general

From Rob Sargent
Subject Re: speed up full table scan using psql
Date
Msg-id 67ffdd7a-8870-c560-d5e4-c53f237f6ae3@gmail.com
Whole thread Raw
In response to speed up full table scan using psql  (Lian Jiang <jiangok2006@gmail.com>)
List pgsql-general
Tree view
On 5/30/23 22:25, Lian Jiang wrote:
hi,

I am using psql to periodically dump the postgres tables into json files which are imported into snowflake. For large tables (e.g. 70M rows), it takes hours for psql to complete. Using spark to read the postgres table seems not to work as the postgres read only replication is the bottleneck so spark cluster never uses >1 worker node and the working node timeout or out of memory.

Will vertical scaling the postgres db speed up psql? Or any thread related parameter of psql can help? Thanks for any hints.

Regards
Lian
Have you looked into COPY command?  Or CopyManager java class?

pgsql-general by date:

Previous
From: Lian Jiang
Date:
Subject: speed up full table scan using psql
Next
From: David Rowley
Date:
Subject: Re: speed up full table scan using psql