Postgres Pro
Downloads
Home   >   mailing lists

Re: export to parquet - Mailing list pgsql-general

From George Woodring
Subject Re: export to parquet
Date
Msg-id CACi+J=QB8g3SLXHSob2eh1BBqyxYNvf07qvP+gqA-knw4PcHwQ@mail.gmail.com
Whole thread Raw
In response to export to parquet  (Scott Ribe <scott_ribe@elevated-dev.com>)
List pgsql-general
Tree view
I don't know how many hoops you want to jump through, we use AWS and Athena to create them.
  • Export table as JSON
  • Put on AWS S3
  • Create JSON table in Athena
  • Use the JSON table to create a parquet table
The parquet files will be in S3 as well after the parquet table is created.  If you are interested I can share the AWS CLI commands we use.

George Woodring
iGLASS Networks
www.iglass.net


On Wed, Aug 26, 2020 at 3:00 PM Scott Ribe <scott_ribe@elevated-dev.com> wrote:
I have no Hadoop, no HDFS. Just looking for the easiest way to export some PG tables into Parquet format for testing--need to determine what kind of space reduction we can get before deciding whether to look into it more.

Any suggestions on particular tools? (PG 12, Linux)


--
Scott Ribe
scott_ribe@elevated-dev.com
https://www.linkedin.com/in/scottribe/





pgsql-general by date:

Previous
From: Scott Ribe
Date:
Subject: Re: export to parquet
Next
From: Tom Lane
Date:
Subject: Re: Finding description pg_description