Re: An idea for parallelizing COPY within one backend - Mailing list pgsql-hackers

From Florian G. Pflug
Subject Re: An idea for parallelizing COPY within one backend
Date
Msg-id 47C56F7E.3030301@phlo.org
Whole thread Raw
In response to Re: An idea for parallelizing COPY within one backend  (Dimitri Fontaine <dfontaine@hi-media.com>)
Responses Re: An idea for parallelizing COPY within one backend  ("A.M." <agentm@themactionfaction.com>)
List pgsql-hackers
Dimitri Fontaine wrote:
> Of course, the backends still have to parse the input given by pgloader, which 
> only pre-processes data. I'm not sure having the client prepare the data some 
> more (binary format or whatever) is a wise idea, as you mentionned and wrt 
> Tom's follow-up. But maybe I'm all wrong, so I'm all ears!

As far as I understand, pgloader starts N threads or processes that open 
up N individual connections to the server. In that case, moving then 
text->binary conversion from the backend into the loader won't give any
additional performace I'd say.

The reason that I'd love some within-one-backend solution is that I'd 
allow you to utilize more than one CPU for a restore within a *single* 
transaction. This is something that a client-side solution won't be able 
to deliver, unless major changes to the architecture of postgres happen 
first...

regards, Florian Pflug



pgsql-hackers by date:

Previous
From: Alvaro Herrera
Date:
Subject: Re: Required make version
Next
From: cristianopintado@gmail.com
Date:
Subject: select avanced