COPY TO STDOUT Apache Arrow support - Mailing list pgsql-hackers

From Adam Lippai
Subject COPY TO STDOUT Apache Arrow support
Date
Msg-id CAGrfaBVyfm0wPzXVqm0=h5uArYh9N_ij+sVpUtDHqkB=VyB3jw@mail.gmail.com
Whole thread Raw
Responses Re: COPY TO STDOUT Apache Arrow support  (Adam Lippai <adam@rigo.sk>)
List pgsql-hackers
Hi,

would it be possible to add Apache Arrow streaming format to the copy backend + frontend?
The use case is fetching (or storing) tens or hundreds of millions of rows for client side data science purposes (Pandas, Apache Arrow compute kernels, Parquet conversion etc). It looks like the serialization overhead when using the postgresql wire format can be significant.

Best regards,
Adam Lippai

pgsql-hackers by date:

Previous
From: "David G. Johnston"
Date:
Subject: Re: Add version and data directory to initdb output
Next
From: Alvaro Herrera
Date:
Subject: Re: typos