albertwang wrote:
> Hi Kevin,
>
> How is your question / suggestion going?
>
> I met the same problem. Our company uses PostgreSQL as the OLTP system. I am
> working on an ETL project which reads data from a PostgreSQL db, transforms
> the data and then load the data into another PostgreSQL db. The ETL tool
> uses JDBC.
>
> My problem is that the ETL won't start to transform the data before the
> whole data result set has been loaded into memory.
>
> I would like to do the data transforming and loading while extracting data.
How about instead of using the JDBC intermediary, extract the data into a local
file/memory db, then transform and load from it to the destination PostgreSQL
db. Set up three seperate parallel processes to perform the task.
danap.
>
> Do you know if there is any progress reagrding the streaming result set? As
> I know, MySQL does support it.
>
> Thank you.
>
> Regards,
> Albert
