Faster compression, again - Mailing list pgsql-hackers

From Daniel Farina
Subject Faster compression, again
Date
Msg-id CAAZKuFb59sABSa7gCG0vnVnGb-mJCUBBbrKiyPraNXHnis7KMw@mail.gmail.com
Whole thread Raw
Responses Re: Faster compression, again  ("ktm@rice.edu" <ktm@rice.edu>)
Re: Faster compression, again  (Merlin Moncure <mmoncure@gmail.com>)
Re: Faster compression, again  (Daniel Farina <daniel@heroku.com>)
Re: Faster compression, again  (Simon Riggs <simon@2ndQuadrant.com>)
Re: Faster compression, again  (Huchev <hugochevrain@gmail.com>)
List pgsql-hackers
For 9.3 at a minimum.

The topic of LZO became mired in doubts about:

* Potential Patents
* The author's intention for the implementation to be GPL

Since then, Google released "Snappy," also an LZ77-class
implementation, and it has been ported to C (recently, and with some
quirks, like no LICENSE file...yet, although it is linked from the
original Snappy project).  The original Snappy (C++) has a BSD license
and a patent grant (which shields you from Google, at least).  Do we
want to investigate a very-fast compression algorithm inclusion again
in the 9.3 cycle?

I've been using the similar implementation "LZO" for WAL archiving and
it is a significant savings (not as much as pg_lesslog, but also less
invasive).  It is also fast enough that even if one were not to uproot
TOAST's compression that it would probably be very close to a complete
win for protocol traffic, whereas SSL's standardized zlib can
definitely be a drag in some cases.

This idea resurfaces often, but the reason why I wrote in about it is
because I have a table which I categorized as "small" but was, in
fact, 1.5MB, which made transferring it somewhat slow over a remote
link.  zlib compression takes it down to about 550K and lzo (similar,
but not identical) 880K.  If we're curious how it affects replication
traffic, I could probably gather statistics on LZO-compressed WAL
traffic, of which we have a pretty huge amount captured.

-- 
fdr


pgsql-hackers by date:

Previous
From: Marco Nenciarini
Date:
Subject: Re: [PATCH] Support for foreign keys with arrays
Next
From: "ktm@rice.edu"
Date:
Subject: Re: Faster compression, again