Re: Optimizing pglz compressor - Mailing list pgsql-hackers
From | Andres Freund |
---|---|
Subject | Re: Optimizing pglz compressor |
Date | |
Msg-id | 20130306174217.GG4970@alap2.anarazel.de Whole thread Raw |
In response to | Re: Optimizing pglz compressor (Merlin Moncure <mmoncure@gmail.com>) |
List | pgsql-hackers |
On 2013-03-06 11:31:06 -0600, Merlin Moncure wrote: > On Wed, Mar 6, 2013 at 10:53 AM, Andres Freund <andres@2ndquadrant.com> wrote: > > On 2013-03-06 09:36:19 -0600, Merlin Moncure wrote: > >> On Wed, Mar 6, 2013 at 8:32 AM, Joachim Wieland <joe@mcknight.de> wrote: > >> > On Tue, Mar 5, 2013 at 8:32 AM, Heikki Linnakangas > >> > <hlinnakangas@vmware.com> wrote: > >> >> With these tweaks, I was able to make pglz-based delta encoding perform > >> >> roughly as well as Amit's patch. > >> > > >> > Out of curiosity, do we know how pglz compares with other algorithms, e.g. lz4 ? > >> > >> This has been a subject of much recent discussion. It compares very > >> poorly, but installing a new compressor tends to be problematic due to > >> patent concerns (something which I disagree with but it's there). All > >> that said, Heikki's proposed changes seem to be low risk and quite > >> fast. > > > > Imo the licensing part is by far the smaller one. The interesting part > > is making a compatible change to the way toast compression works that > > supports multiple compression schemes. Afaics nobody has done that work. > > After that the choice of to-be-integrated compression schemes needs to > > be discussed, sure. > > That wasn't the conversation I remember. lz4 completely smokes pglz > (Heikki's changes notwithstanding) and is bsd licensed so AFAICS there > it no reason to support multiple compression technologies except for > migration purposes (and even if you did want to, a plausible way to do > that was identified). Well, we need to permanently support at least two technologies - otherwise we will break pg_upgrade. And there is absolutely no reason to believe nothing better than lz4 will come along in the future so just supporting two seems like a bad idea. And sure, there are rough ideas on how to support different compression schemes in toast, but nobody has implemented anything afaics. It would be possible to reuse what I proposed for indirect toast tuples for that purpose although I am not sure whether I like using varatt_external's in multiple types as indicated by varatt1_be.va_len_1be (renamed to va_type or such) apointing to different types. Which means that an uncompressible datum would potentially have multiple representations. > ...but that's a separate topic for another day. Heikki's proposal > seems like a win either way. Yes. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
pgsql-hackers by date: