Re: New Copy Formats - avro/orc/parquet - Mailing list pgsql-general

From Tom Lane
Subject Re: New Copy Formats - avro/orc/parquet
Date
Msg-id 18464.1518389293@sss.pgh.pa.us
Whole thread Raw
In response to Re: New Copy Formats - avro/orc/parquet  (Andres Freund <andres@anarazel.de>)
Responses Re: New Copy Formats - avro/orc/parquet  (Andres Freund <andres@anarazel.de>)
Re: New Copy Formats - avro/orc/parquet  (Magnus Hagander <magnus@hagander.net>)
List pgsql-general
Andres Freund <andres@anarazel.de> writes:
> So, I think making COPY extensible would be quite beneficial. I'm
> however quite doubtful that we want to add core code to handle all of
> the above. I think we should make the COPY input/output formatting
> extensible by extensions.

+1.  I can't see carrying code for these formats in-core, but I've
no objection to making it possible for someone else to maintain them.

> I imagine we'd have callbacks for
> - start copy in / out
> - output row, with a an array of values/nulls
> - parse row, with a input buffer as argument, returning values / nulls arrays
> - finish copy in / out

Also something to allow absorbing format-specific options, if the
precedent of CSV is anything to go by.  (Any such patch should manage
to turn COPY-CSV into an extension, at least so far as copy.c is
concerned, even if we don't package it as one.)

            regards, tom lane


pgsql-general by date:

Previous
From: Nicolas Paris
Date:
Subject: Re: New Copy Formats - avro/orc/parquet
Next
From: Andres Freund
Date:
Subject: Re: New Copy Formats - avro/orc/parquet