Re: Parallel index build during COPY - Mailing list pgsql-hackers

From Jim Nasby
Subject Re: Parallel index build during COPY
Date
Msg-id 8B479C0F-0D00-4DE1-BF1A-D71FB2D41C09@pervasive.com
Whole thread Raw
In response to Re: Parallel index build during COPY  (Toru SHIMOGAKI <shimogaki.toru@oss.ntt.co.jp>)
List pgsql-hackers
On Jun 15, 2006, at 9:45 PM, Toru SHIMOGAKI wrote:

> NTT has some ideas about index creation during a large amount of
> data loading. Our approach is the following: index tuples are
> created at the same time as heap tuples and added into heapsort. In
> addition, we use old index tuples as sorted list if the target
> table has already data. It is not necessary for data loader to sort
> all the index tuples including old ones. After only new index
> tuples are sorted, both sorted lists are merged and the whole index
> is built. It can save both CPU resources and disk accesses
> dramatically, especially if the target table has already so many
> tuples.
> This approach needs to acquire a table lock, which is unlike COPY's
> lock mode, so we have developed it as another bulk load tool. We
> will talk about it in PostgreSQL Anniversary Conference at Toronto.
> Thank you for Josh’s coordination.

So does that mean you're able to do all that without hacking the back-
end? Impressive. :)

I look forward to hearing about it.
--
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461




pgsql-hackers by date:

Previous
From: paolo romano
Date:
Subject: Re: MultiXacts & WAL
Next
From: Jim Nasby
Date:
Subject: Re: postgresql and process titles