Hi Zhihong,
Hi,
I searched for Postgres support in Apache Spark.
I found Spark doc related to JDBC.
I wonder if the community is aware of Spark connector for Postgres (hopefully open source) where predicate involving jsonb columns can be pushed down.
JDBC driver is indeed the best driver which can be used if you have to persist your Spark dataframes in PostgreSQL, IMO.
It's a connector which supports just pure SQL (no mapping between your Scala/Java classes and the DB schema for instance, despite ORM frameworks like Hibernate), so it works at a lower level allowing you to use directly the queries you would use to handle jsonb data.
Maybe you need to use pure json's between Spark objects and the DB, but the communication using JDBC driver can be completely based on jsonb.
Giuseppe.
