Thread: Please recommend me the best bulk-delete option
Hi All,
I'm using PostgreSQL 8.1.4. I've 3 tables: one being the core (table1), others are dependents (table2,table3). I
inserted70000 records in table1 and appropriate related records in other 2 tables. As I'd used CASCADE, I could able to
deletethe related records using DELETE FROM table1; It works fine when the records are minimal in my current PostgreSQL
version.When I've a huge volume of records, it tries to delete all but there is no sign of deletion progress for many
hours!Whereas, bulk import, does in few minutes. I wish to do bulk-delete in reasonable minutes. I tried TRUNCATE also.
Like,TRUNCATE table3, table2,table1; No change in performance though. It just takes more time, and no sign of
completion!From the net, I got few options, like, deleting all constraints and then recreating the same would be fine.
But,no query seems to be successfully run over 'table1' when it's loaded more data!
Please recommend me the best solutions to delete all the records in minutes.
CREATE TABLE table1(
t1_id SERIAL PRIMARY KEY,
disp_name TEXT NOT NULL DEFAULT '',
last_updated TIMESTAMP NOT NULL DEFAULT current_timestamp,
UNIQUE(disp_name)
) WITHOUT OIDS;
CREATE UNIQUE INDEX disp_name_index on table1(upper(disp_name));
CREATE TABLE table2 (
t2_id SERIAL PRIMARY KEY,
t1_id INTEGER REFERENCES table1 ON DELETE CASCADE,
type TEXT
) WITHOUT OIDS;
CREATE TABLE table3 (
t3_id SERIAL PRIMARY KEY,
t1_id INTEGER REFERENCES table1 ON DELETE CASCADE,
config_key TEXT,
config_value TEXT
) WITHOUT OIDS;
Regards,
Siva.
::DISCLAIMER::
-----------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates. Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of
this message without the prior written consent of the author of this e-mail is strictly prohibited. If you have
received this email in error please delete it and notify the sender immediately. Before opening any mail and
attachments please check them for viruses and defect.
-----------------------------------------------------------------------------------------------------------------------
-----Original Message-----
From: pgsql-general-owner@postgresql.org
[mailto:pgsql-general-owner@postgresql.org] On Behalf Of Siva Palanisamy
Sent: Thursday, November 17, 2011 1:04 PM
To: pgsql-general@postgresql.org
Subject: [GENERAL] Please recommend me the best bulk-delete option
Hi All,
I'm using PostgreSQL 8.1.4. I've 3 tables: one being the core (table1),
others are dependents (table2,table3). I inserted 70000 records in table1
and appropriate related records in other 2 tables. As I'd used CASCADE, I
could able to delete the related records using DELETE FROM table1; It works
fine when the records are minimal in my current PostgreSQL version. When
I've a huge volume of records, it tries to delete all but there is no sign
of deletion progress for many hours! Whereas, bulk import, does in few
minutes. I wish to do bulk-delete in reasonable minutes. I tried TRUNCATE
also. Like, TRUNCATE table3, table2,table1; No change in performance though.
It just takes more time, and no sign of completion! From the net, I got few
options, like, deleting all constraints and then recreating the same would
be fine. But, no query seems to be successfully run over 'table1' when it's
loaded more data!
Please recommend me the best solutions to delete all the records in minutes.
CREATE TABLE table1(
t1_id SERIAL PRIMARY KEY,
disp_name TEXT NOT NULL DEFAULT '',
last_updated TIMESTAMP NOT NULL DEFAULT current_timestamp,
UNIQUE(disp_name)
) WITHOUT OIDS;
CREATE UNIQUE INDEX disp_name_index on table1(upper(disp_name));
CREATE TABLE table2 (
t2_id SERIAL PRIMARY KEY,
t1_id INTEGER REFERENCES table1 ON DELETE CASCADE,
type TEXT
) WITHOUT OIDS;
CREATE TABLE table3 (
t3_id SERIAL PRIMARY KEY,
t1_id INTEGER REFERENCES table1 ON DELETE CASCADE,
config_key TEXT,
config_value TEXT
) WITHOUT OIDS;
Regards,
Siva.
---------------------------------------------------------------
The fastest you can do is:
TRUNCATE table3;
TRUNCATE table2;
TRUNCATE table1;
(do them separately to avoid any possible re-ordering - I am unsure whether
it is allowed but given your comments it may be)
Well, not totally true since:
DROP DATABASE ...;
CREATE TABLE table1;
CREATE TABLE table2;
CREATE TABLE table3;
Is possibly faster...
Anyway, if you execute the three TRUNCATEs in the proper order, thus
avoiding any kind of cascade, you should get maximum performance possible on
your UNSUPPORTED VERSION of PostgreSQL.
David J.
Hi. On 17 Listopad 2011, 19:03, Siva Palanisamy wrote: > Hi All, > > I'm using PostgreSQL 8.1.4. I've 3 tables: one being the core (table1), That's a bit old - update to 8.1.23 (or to a never version, if possible). > others are dependents (table2,table3). I inserted 70000 records in table1 > and appropriate related records in other 2 tables. As I'd used CASCADE, I > could able to delete the related records using DELETE FROM table1; It > works fine when the records are minimal in my current PostgreSQL version. > When I've a huge volume of records, it tries to delete all but there is no > sign of deletion progress for many hours! Whereas, bulk import, does in > few minutes. I wish to do bulk-delete in reasonable minutes. I tried > TRUNCATE also. Like, TRUNCATE table3, table2,table1; No change in > performance though. It just takes more time, and no sign of completion! > From the net, I got few options, like, deleting all constraints and then > recreating the same would be fine. But, no query seems to be successfully > run over 'table1' when it's loaded more data! > Please recommend me the best solutions to delete all the records in > minutes. "TRUNCATE table1 CASCADE" should be quite fast. Have you analyzed the tables after loading the data? That might be one cause. Post explain analyze of the queries, or at least explain if it takes very long to finish. And post some basic system stats collected when running them (iostat, vmstat). Tomas
On 17 Listopad 2011, 19:26, David Johnston wrote: > Anyway, if you execute the three TRUNCATEs in the proper order, thus > avoiding any kind of cascade, you should get maximum performance possible > on your UNSUPPORTED VERSION of PostgreSQL. AFAIK cascade with TRUNCATE means 'truncate the depending tables first'. That's quite different from cascade with DELETE where the db has to actually find the records, and I don't think there's a significant performance impact. Tomas