We are developing a data warehouse of which the integration layer will start with over 100 TB of data. There are not many entities though we probably can partition and foremost we should use inheritance for the lab results. I just was wondering if PostgreSQL was able to cope with. In case it depends on the modelling kind, we have not yet decided between classic erd, anchor modelling and data vault.
Does someone have experience with such a set up?
There are a significant number of issues that happen when you try to put that much data in PostgrSQL. Remember there is a hard limit of 32TB per table.
I currently help administer an analytics environment where 400TB of data is pre-aggregated into 32TB of ready-to-serve metrics. We generally try to keep our initial point of entry databases to under 20TB of possible.
Nonetheless it is quite possible either using distributed add-ons like Citus, forks like Postgres-XL, or (as we did at Adjust) a mixture of data consolidation and application-level sharding.
As a plug, I expect to deliver talks about this in various places. Stay tuned ;-)
Kind regards Thiemo
--
Best Wishes,
Chris Travers
Efficito: Hosted Accounting and ERP. Robust and Flexible. No vendor lock-in.
