Re: Make COPY extendable in order to support Parquet and other formats - Mailing list pgsql-hackers

From Aleksander Alekseev
Subject Re: Make COPY extendable in order to support Parquet and other formats
Date
Msg-id CAJ7c6TNACaX1aVh-b9Z8n7aWoxAg2vK22TduZOP-9vkEk9_3Pw@mail.gmail.com
Whole thread Raw
In response to Re: Make COPY extendable in order to support Parquet and other formats  (Andrew Dunstan <andrew@dunslane.net>)
List pgsql-hackers
Hi Andrew,

> > IMO decent COPY FROM / TO STDIN support is crucial, because otherwise you
> > can't do COPY from/to a client. Which would make the feature unusable for
> > anybody not superuser, including just about all users of hosted PG.
> >
>
> +1
>
> Note that Parquet puts the metadata at the end of each file, which makes
> it nice to write but somewhat unfriendly for streaming readers, which
> would have to accumulate the whole file in order to process it.

It's not necessarily that bad since data is divided into pages, each
page can be processed separately. However personally I have limited
experience with Parquet at this point. Some experimentation is
required. I will keep in mind the requirement regarding COPY FROM / TO
STDIN.

-- 
Best regards,
Aleksander Alekseev



pgsql-hackers by date:

Previous
From: Robert Haas
Date:
Subject: Re: making relfilenodes 56 bits
Next
From: "Imseih (AWS), Sami"
Date:
Subject: Re: [BUG] Panic due to incorrect missingContrecPtr after promotion