Re: [HACKERS][Proposal] LZ4 Compressed Storage Manager - Mailing list pgsql-hackers

From Bruce Momjian
Subject Re: [HACKERS][Proposal] LZ4 Compressed Storage Manager
Date
Msg-id 20190411161825.ut44vjkh4zie6sgo@momjian.us
Whole thread Raw
In response to [HACKERS][Proposal] LZ4 Compressed Storage Manager  (Николай Петров <nik.petrov.ua@yandex.ru>)
Responses Re: [HACKERS][Proposal] LZ4 Compressed Storage Manager
List pgsql-hackers
On Sun, Mar 31, 2019 at 05:25:51PM +0300, Николай Петров wrote:
> Why it should be implemented on Storage Manager level instead of usage
> Pluggable storage API [9]?
>   - From my perspective view Storage Manager level implementation 
>     allows to focus on proper I/O operations and compression. 
>     It allows to write much more simple realization. It's because of 
>     Pluggable storage API force you to implement more complex 
>     interfaces. To be honest, I am really hesitating about this point, 
>     especially because of Pluggable storage API allows to create 
>     extension without core code modification and it potentially allows 
>     to use more perfective compression algorithms (Table Access Manager
>     allows you to get more information about storing data). 
> 
> I would like to implement a proof of concept 
> and have a couple of questions:
>   - your opinion about necessity of this feature 
>     (Compressed Storage Manager)
>   - Is it good idea to implement DB compression on Storage Manager 
>     level? Perhaps it is better to use Pluggable storage API.
>   - Is there any reason to refuse this proposal?
>   - Are there any circumstances what didn't allow to implement 
>     Compressed Storage Manager?

Stepping back a bit, there are several levels of compression:

1.  single field
2.  across all fields in a row
3.  across rows on a single page
4.  across all rows in a table
5.  across tables in a database

We currently do #1 with TOAST, and your approach would allow the first
three.  #4 feels like it is getting near the features of columnar
storage.  I think it is unclear if adding #2 and #3 produce enough of a
benefit to warrant special storage, given the complexity and overhead of
implementing it.

I do think the Pluggable storage API is the right approach, and, if you
are going to go that route, adding #4 compression seems very worthwhile.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +



pgsql-hackers by date:

Previous
From: Peter Billen
Date:
Subject: Re: serializable transaction: exclude constraint violation (backed byGIST index) instead of ssi conflict
Next
From: Pavel Stehule
Date:
Subject: Re: [HACKERS][Proposal] LZ4 Compressed Storage Manager