Hi Bob:
On Sat, Jul 5, 2014 at 3:34 PM, Bob Jolliffe <bobjolliffe@gmail.com> wrote:
> Though currently I am thinking to address the problem of generating deltas
> at the application level rather than to use postgresql features which are
> largely optimized for a slightly different set of circumstances and
> requirements.
Yeah, special purpose can always be more efficient. Specially when you
can tune for your special circunstances, like non full replication,
and replicating only once a day, and you probably can minimizae the
changesets knowing data dependencies.
> Impressive what can be done witha 2400 baud modem when you set your mind to
> it. Fortunately this days are mostly behind us :-)
Not so behind. Then you did that kind of thing for 100Mb data sets,
now you can use the same algorithms for 1Tb data sets. Line speed have
gone up more than data set sizes, but this advantage has been eaten by
customer delay expectations ( in my case, now they want at most 30 min
delay, not 1440 ).
Not knowing your exact data structure or size I cannot recommend
anything, but I've been abble to make really good deltas using the
same algorithms that were used on tapes, dump the table, sort it, diff
with last days, the throughput of current machines is amazing, you can
easily achieve full disk speed.
Regards.
Francisco Olarte.
