Re: An idea for parallelizing COPY within one backend - Mailing list pgsql-hackers

From Florian G. Pflug
Subject Re: An idea for parallelizing COPY within one backend
Date
Msg-id 47C5881E.2040205@phlo.org
Whole thread Raw
In response to Re: An idea for parallelizing COPY within one backend  (Brian Hurt <bhurt@janestcapital.com>)
Responses Re: An idea for parallelizing COPY within one backend  ("Tom Dunstan" <pgsql@tomd.cc>)
Re: An idea for parallelizing COPY within one backend  (Andrew Dunstan <andrew@dunslane.net>)
List pgsql-hackers
Brian Hurt wrote:
> Tom Lane wrote:
>> "Florian G. Pflug" <fgp@phlo.org> writes:
>>> ...
>>> Neither the "dealer", nor the "workers" would need access to the either
>>> the shared memory or the disk, thereby not messing with the "one backend
>>> is one transaction is one session" dogma.
>>> ...
>> Unfortunately, this idea has far too narrow a view of what a datatype
>> input function might do.  Just for starters, consider "enum" input,
>> which certainly requires catalog access.  We have also explicitly
>> acknowledged the idea that datatype I/O functions might try to store
>> typmod-related data in some special catalog somewhere.

> Would it be possible to determine when the copy is starting that this 
> case holds, and not use the parallel parsing idea in those cases?

In theory, yes. In pratice, I don't want to be the one who has to answer 
to an angry user who just suffered a major drop in COPY performance 
after adding an ENUM column to his table.

I was thinking more along the line of letting a datatype specify a
function "void* ioprepare(typmod)" which returns some opaque object
specifying all that the input and output function needs to know.
We could than establish the rule that input/output functions may not 
access the catalog, and instead pass them a pointer to that opaque object.

All pretty pie-in-the-sky at the moment, though...

regards, Florian Pflug



pgsql-hackers by date:

Previous
From: "Heikki Linnakangas"
Date:
Subject: Re: An idea for parallelizing COPY within one backend
Next
From: Paul van den Bogaard
Date:
Subject: Re: Proposed changes to DTrace probe implementation