Re: Out of swap space & memory - Mailing list pgsql-general

Mike, thanks so much for your reply. I'm sorry for not showing you my SQL. I
didn't show it because I couldn't manage to boil it down to something
reproducible that everyone could try. But here's what it was:

drop table octanenights;
CREATE TABLE octanenights (member_id varchar(100), campaign_id varchar(100),
catalog_type varchar(100), pushed int, delivered int, clicks int, opened
int, month varchar(100), type1 int, type2 int, type3 int, type4 int, type5
int);

copy octanenights from
'/home/kevin/octanenights/proc/uberfile/uberfile1.txt' null as '';
copy octanenights from
'/home/kevin/octanenights/proc/uberfile/uberfile2.txt' null as '';
copy octanenights from
'/home/kevin/octanenights/proc/uberfile/uberfile3.txt' null as '';

select * from octanenights limit 10;
alter table octanenights rename to octanenights_raw;

-- de-dup the table
select member_id, campaign_id, catalog_type, pushed, delivered, clicks,
opened, month, type1, type2, type3, type4, type5
into octanenights
from octanenights_raw
group by member_id, campaign_id, catalog_type, pushed, delivered, clicks,
         opened, month, type1, type2, type3, type4, type5;

Let me tell you a little about octanenights. It's a file of about 36,000,000
rows, each describing an e-mail sent. Unfortunately, there are duplicate
records scattered throughout the table, which I do not care about. One might
suggest that I could've used uniq from the command line for this, but the
data were not sorted originally and the duplicate records may be scattered
anywhere in the table. The objective in the final line is to de-dup the
table and place it into octanenights, leaving the original in
octanenights_raw in case I ever need to refer back to it.

MS SQL Server, with as much RAM and less clock speed, de-dups the table in
about six minutes. The de-duped version has about 26,000,000 rows. The final
line is where Postgres gobbles up all my swap and RAM and then conks out
completely.

Am I doing something wrong? Maybe there was a better way to approach this
problem? I'd be open to suggestions of any kind, since I'm still very, very
new to the world of optimizing Postgres.

Kevin

-----Original Message-----
From: pgsql-general-owner@postgresql.org
[mailto:pgsql-general-owner@postgresql.org] On Behalf Of Manfred Koizar
Sent: Tuesday, August 03, 2004 3:04 AM
To: Kevin Bartz
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] Out of swap space & memory

On Mon, 2 Aug 2004 20:08:45 -0700, "Kevin Bartz"
<kbartz@loyaltymatrix.com> wrote:
>is there any way I can run this query?

What query?  You didn't show us your SQL.

Servus
 Manfred

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
      subscribe-nomail command to majordomo@postgresql.org so that your
      message can get through to the mailing list cleanly