ETL tool for json and postgresql

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

ETL tool for json and postgresql

Jayson Hreczuck
Good Morning,

This is probably not the best place to ask this, but I'd figure I would give it a shot.

We are slowly moving over to a hybrid nosql model, and many of our large sets of data are being stored in json format, specifically using the jsonb data type.

The issue is that the etl tool that I've developed to parse this data into a datawarehouse is very manual and pretty cumbersome - essentially using the jsonb postgresql query commands and cron or pgagent.  It's unable to effectively traverse the element levels, and really isn't very good at handling dynamic changes.

I am wondering what people use for a tool to do this work.  We are a smaller shop and do not need a large solution.  Would prefer something that integrates well with postgresql and can be easily modified to change data mappings, and possibly support additional filetypes outside of json.

The end result would be a table or set of tables that can be ingested by a visualization tool like Tableau or PowerBI.



Jayson Hreczuck | Principal Database Administrator

C: +1 617.571.9861

130 Turner St, Ste 530, Waltham, MA 02453