Re: libpq compression - Mailing list pgsql-hackers
| From | Craig Ringer |
|---|---|
| Subject | Re: libpq compression |
| Date | |
| Msg-id | CAMsr+YEmb6sAd3jzXJu+_sseOTrv7JLa+HDDWQBNv15+7PYfdQ@mail.gmail.com Whole thread Raw |
| In response to | Re: libpq compression (Andrew Dunstan <andrew.dunstan@2ndquadrant.com>) |
| List | pgsql-hackers |
On 15 May 2018 at 21:36, Andrew Dunstan <andrew.dunstan@2ndquadrant.com> wrote:
> To use zstd compression, Postgres should be configured with --with-zstd. Otherwise compression will use zlib unless it is disabled by --without-zlib option.
> I have added compression=on/off parameter to connection string and -Z option to psql and pgbench utilities.
I'm a bit confused why there was no reply to this. I mean, it wasn't sent on
1st April, the patch still can be applied on top of the master branch and looks
like it even works.
I assume the main concern her is that it's implemented in a rather not
extensible way. Also, if I understand correctly, it compresses the data stream
in both direction server <-> client, not sure if there is any value in
compressing what a client sends to a server. But still I'm wondering why it
didn't start at least a discussion about how it can be implemented. Do I miss
something?
Implementation of libpq compression will be included in next release of PgProEE.
Looks like community is not so interested in this patch. Frankly speaking I do not understand why.
I'm definitely very interested, and simply missed the post.
I'll talk with some team mates as we're doing some PG12 planning now.
Yes, this patch is not extensible: it can use either zlib either zstd. Unfortunately internal Postgres compression pglz doesn't provide streaming API.
May be it is good idea to combine it with Ildus patch (custom compression methods): https://commitfest.postgresql.org/18/1294/
Given the history of issues with attempting custom/pluggable compression for toast etc, I really wouldn't want to couple those up.
pglz wouldn't make much sense for protocol compression anyway, except maybe for fast local links where it was worth a slight compression overhead but not the cpu needed for gzip. I don't think it's too exciting. zlib/gzip is likely the sweet spot for the reasonable future for protocol compression, or a heck of a lot better than what we have, anyway.
We should make sure the protocol part is extensible, but the implementation doesn't need to be pluggable.
In this case it will be possible to use any custom compression algorithm. But we need to design and implement streaming API for pglz and other compressors.
I'm sure there is plenty of interest in this. However, you guys need to understand where we are in the development cycle. We're trying to wrap up Postgres 11, which was feature frozen before this patch ever landed. So it's material for Postgres 12. That means it will probably need to wait a little while before it gets attention. It doesn't mean nobody is interested.
I disagree with Dmitry about compressing in both directions - I can think of plenty of good cases where we would want to compress traffic from the client.
Agreed. The most obvious case being COPY, but there's also big bytea values, etc.
pgsql-hackers by date: