ETL tool for json and postgresql - Mailing list pgsql-admin

From Jayson Hreczuck
Subject ETL tool for json and postgresql
Date
Msg-id CAHg7=h48Bag0iJ=U3zsRO1bnnTO5v4=FTTgh--wY2d73cFE6HQ@mail.gmail.com
Whole thread Raw
List pgsql-admin
Good Morning,

This is probably not the best place to ask this, but I'd figure I would give it a shot.

We are slowly moving over to a hybrid nosql model, and many of our large sets of data are being stored in json format, specifically using the jsonb data type.

The issue is that the etl tool that I've developed to parse this data into a datawarehouse is very manual and pretty cumbersome - essentially using the jsonb postgresql query commands and cron or pgagent.  It's unable to effectively traverse the element levels, and really isn't very good at handling dynamic changes.

I am wondering what people use for a tool to do this work.  We are a smaller shop and do not need a large solution.  Would prefer something that integrates well with postgresql and can be easily modified to change data mappings, and possibly support additional filetypes outside of json.

The end result would be a table or set of tables that can be ingested by a visualization tool like Tableau or PowerBI.

Thanks!
-Jayson

--


Jayson Hreczuck | Principal Database Administrator

C: +1 617.571.9861130 Turner St, Ste 530, Waltham, MA 02453

pgsql-admin by date:

Previous
From: Priancka Chatz
Date:
Subject: Re: auto vacuum restart
Next
From: Laurenz Albe
Date:
Subject: Re: PostgreSQL community vs PostgreSQL EDB comparison and Difference document needed!!!!