Re: Binary COPY IN size reduction - Mailing list pgsql-hackers

From Tom Lane
Subject Re: Binary COPY IN size reduction
Date
Msg-id 23116.1587737963@sss.pgh.pa.us
Whole thread Raw
In response to Binary COPY IN size reduction  (Lőrinc Pap <lorinc@gradle.com>)
Responses Re: Binary COPY IN size reduction  (Lőrinc Pap <lorinc@gradle.com>)
List pgsql-hackers
=?UTF-8?Q?L=C5=91rinc_Pap?= <lorinc@gradle.com> writes:
> We've switched recently from TEXT based COPY to the BINARY one.
> We've noticed a slight performance increase, mostly because we don't need
> to escape the content anymore.
> Unfortunately the binary protocol's output ended up being slightly bigger
> than the text one (e.g. for one payload it's *373MB* now, was *356MB* before)
> ...
> By skipping the column count and sizes for every row, in our example this
> change would reduce the payload to *332MB* (most of our payload is binary,
> lightweight structures consisting of numbers only could see a >*2x*
> decrease in size).

TBH, that amount of gain does not seem to be worth the enormous
compatibility costs of introducing a new COPY data format.  What you
propose also makes the format a great deal less robust (readers are
less able to detect errors), which has other costs.  I'd vote no.

            regards, tom lane



pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: proposal - plpgsql - all plpgsql auto variables should be constant
Next
From: Justin Pryzby
Date:
Subject: Re: [PATCH'] Variables assigned with values that is never used.