On 13. Jul 2020, at 16:23, Henrique Montenegro <typoon@gmail.com> wrote:
[...]
* Insert the data from the `users` table into the `users_no_dups` table
``` insert into users_no_dups ( created_ts, user_id, name, url ) ( select created_ts, user_id, name, url from users ) on conflict do nothing ```
How do you check contraints here? Is this enforced with UK/PK?
Running the above loop worked fine for about 12 hours. Each file was taking about 30 seconds to be processed. About 4 seconds to create the `users` table and have the CSV data loaded into it and anything between 20 and 30 seconds to insert the data from `users` into `users_no_dups`.
Do you see anything suspicious in the logs, i.e. something in the realms of running out of transaction IDs?
[...]
Recreating the table now isn't really providing any improvements. I tried recreating it with a `fillfactor` of `10`, but it was taking too long and too much space (the table had 300GB with the fillfactor set to 30; with it set to 10 it went up to almost 1TB).
To me it sounds like the UK/PK is getting too much to write. A possible solution could be to start partitioning the table.
Swarm64 AS Parkveien 41 B | 0258 Oslo | Norway Registered at Brønnøysundregistrene in Norway under Org.-Number 911 662 787 CEO/Geschäftsführer (Daglig Leder): Thomas Richter; Chairman/Vorsitzender (Styrets Leder): Dr. Sverre Munck
Swarm64 AS Zweigstelle Hive Ullsteinstr. 120 | 12109 Berlin | Germany Registered at Amtsgericht Charlottenburg - HRB 154382 B
