Re: Libpq COPY optimization - Mailing list pgsql-hackers

From Tom Lane
Subject Re: Libpq COPY optimization
Date
Msg-id 19325.1137107547@sss.pgh.pa.us
Whole thread Raw
In response to Re: Libpq COPY optimization  ("Alon Goldshuv" <agoldshuv@greenplum.com>)
Responses Re: Libpq COPY optimization  (Bruce Momjian <pgman@candle.pha.pa.us>)
List pgsql-hackers
"Alon Goldshuv" <agoldshuv@greenplum.com> writes:
> Please help me understand this better. It appears to me that when the
> client->backend pipe fills up, pqSendSome() consumes any incoming
> NOTICE/WARNING messages before waiting, which should prevent deadlock.

Hm, I had forgotten that the low-level pqSendSome routine does that.
That makes the PQconsumeInput call in PQputCopyData redundant (or
almost; see below).  The parseInput call is still needed, because it's
there to pull NOTICE messages out of the input buffer and get rid of
them, rather than possibly having the input buffer grow to exceed
memory.  But when there's nothing for it to do, parseInput is cheap
enough that there's no real need to bypass it.

In short, if you just remove the PQconsumeInput call I think you'll find
that it does what you want.

The only case where it's helpful to have it there is if there's a
incomplete message in the input buffer, as parseInput isn't quite so
fast if it has to determine that the message is incomplete.  Without
the PQconsumeInput call, the incomplete-message state could persist
for a long time, and you'd pay the parseInput overhead each time through
PQputCopyData.  However, that's certainly not the normal situation,
so I think we could leave that case slightly pessimal.  It's certainly
true that that path in parseInput is a lot faster than a kernel call,
so it'd still be better than it is now.
        regards, tom lane


pgsql-hackers by date:

Previous
From: David Fetter
Date:
Subject: Contrib Schemas
Next
From: Mark Kirkwood
Date:
Subject: Re: Contrib Schemas