Re: Compression and on-disk sorting - Mailing list pgsql-hackers

From Martijn van Oosterhout
Subject Re: Compression and on-disk sorting
Date
Msg-id 20060517084559.GC15180@svana.org
Whole thread Raw
In response to Re: Compression and on-disk sorting  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-hackers
On Wed, May 17, 2006 at 12:03:15AM -0400, Tom Lane wrote:
> AFAICS the only sane choice here is to use
> src/backend/utils/adt/pg_lzcompress.c, on the grounds that (1) it's
> already in the backend, and (2) data compression in general is such a
> minefield of patents that we'd be foolish to expose ourselves in more
> than one direction.

Unfortunatly, the interface provided by pg_lzcompress.c is probably
insufficient for this purpose. You want to be able to compress tuples
as they get inserted and start a new block once the output reaches a
certain size. pg_lzcompress.c only has the options compress-whole-block
and decompress-whole-block.

zlib allows you to compress as the data comes along, keeping an eye on
the output buffer while you do it. For an initial test, using zlib
directly would probably be easier. If it works out we can look into
alternatives.

Have a nice day,
--
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

pgsql-hackers by date:

Previous
From: Martijn van Oosterhout
Date:
Subject: Re: Compression and on-disk sorting
Next
From: Andrew Piskorski
Date:
Subject: Re: Compression and on-disk sorting