Re: Postgres and data warehouses - Mailing list pgsql-general

From Brent Tubbs
Subject Re: Postgres and data warehouses
Date
Msg-id CAAOBOv9jynL7=1GarS+STRczLfguQk25s7U9LuOExKO=HAk74A@mail.gmail.com
Whole thread Raw
In response to Re: Postgres and data warehouses  (Jim Nasby <Jim.Nasby@BlueTreble.com>)
List pgsql-general
The logical replication features in 9.4 look very promising as an alternative to the binary replication that requires an identical schema on both ends. There's no reason that the thing you're replicating to couldn't be something like RabbitMQ or Storm.

I've been playing with a JSON-based decoder plugin so Postgres could emit changes to non-SQL systems like this.  See https://bitbucket.org/btubbs/decoder_json/src.  It should be still considered experimental and only used by people unafraid of hacking on C at this point.  More eyeballs would be welcome.

I blogged about more options on streaming updates from Postgres at http://btubbs.com/streaming-updates-from-postgres.html.

On Tue, Mar 10, 2015 at 8:11 AM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:
On 3/8/15 11:02 PM, Rob Sargent wrote:
I expected the latter but … some proprietors like to brag :)

Standing down.

On Mar 8, 2015, at 9:56 PM, John R Pierce <pierce@hogranch.com> wrote:

On 3/8/2015 8:24 PM, Rob Sargent wrote:
I strongly suspect many would like further details on your implementation. A user conference session well recorded perhaps?

the details are proprietary and quite specific to our workload and requirements.

You could build something similar with Skytools; it's what Skype actually created it for. They ran many "real time" datamarts/warehouses using it.

The big difference between it and Slony is that Skytools is modular, so it's designed to allow for this kind of flexibility.
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

pgsql-general by date:

Previous
From: Tomas Vondra
Date:
Subject: Re: Benchmarking partitioning triggers and rules
Next
From: Tom Lane
Date:
Subject: Re: PostgreSQL-related legal question