Re: [PATCH] Compression and on-disk sorting - Mailing list pgsql-patches

From Simon Riggs
Subject Re: [PATCH] Compression and on-disk sorting
Date
Msg-id 1147948476.2646.426.camel@localhost.localdomain
Whole thread Raw
In response to Re: [PATCH] Compression and on-disk sorting  (Martijn van Oosterhout <kleptog@svana.org>)
Responses Re: [PATCH] Compression and on-disk sorting  (Martijn van Oosterhout <kleptog@svana.org>)
List pgsql-patches
On Thu, 2006-05-18 at 10:31 +0200, Martijn van Oosterhout wrote:
> On Wed, May 17, 2006 at 06:38:47PM +0100, Simon Riggs wrote:
> > > - Each tape is compressed as one long compressed stream. Currently no
> > > seeking is allowed, so only sorts, no joins! (As tom said, quick and
> > > dirty numbers). This should show this possibility in its best light
> > > but if we want to support seeking we're going to need to change that.
> > > Maybe no compression on the last pass?
> >
> > We should be able to do this without significant loss of compression by
> > redefining the lts block size to be 32k. That's the size of the
> > look-back window anyhow, so compressing the whole stream doesn't get us
> > much more.
>
> The major problem is looking back costs significantly more with
> compression. If you need to look back into the previous compressed
> block, you need to decompress the whole previous block. The simple
> solution would be to keep a buffer of the last 32KB. Another posibility
> would be to have a limit of 32KB of uncompressed data per block and
> just remember the whole previous block.

Just do a Z_FULL_FLUSH when you hit end of block. That way all blocks
will be independent of each other and you can rewind as much as you
like. We can choose the block size to be 32KB or even 64KB, there's no
dependency there, just memory allocation. It should be pretty simple to
make the block size variable at run time, so we can select it according
to how many files and how much memory we have.

--
  Simon Riggs
  EnterpriseDB   http://www.enterprisedb.com


pgsql-patches by date:

Previous
From: Martijn van Oosterhout
Date:
Subject: Re: [PATCH] Compression and on-disk sorting
Next
From: Martijn van Oosterhout
Date:
Subject: Re: [PATCH] Compression and on-disk sorting