Re: libpq compression - Mailing list pgsql-hackers

From Daniil Zakhlystov
Subject Re: libpq compression
Date
Msg-id 62527092-16BD-479F-B503-FA527AF3B0C2@yandex-team.ru
Whole thread Raw
In response to Re: libpq compression  (Konstantin Knizhnik <k.knizhnik@postgrespro.ru>)
Responses Re: libpq compression
List pgsql-hackers
Hi!

I’ve fixed an issue with compression level parsing in this PR https://github.com/postgrespro/libpq_compression/pull/4

Also, did a couple of pgbenchmarks to measure database resource usage with different compression levels.

Firstly, I measured the bidirectional compression scenario, i.e. database had to do both compression and decompression:

Database setup:
pgbench "host=xxx dbname=xxx  port=5432 user=xxx” -i -s 500

Test run:
pgbench "host=xxx dbname=xxx  port=5432 user=xxx compression=zstd:(1/3/5/7/9/11/13/15/17/19/20)" --builtin tpcb-like -t
50--jobs=64 --client=700 

When using bidirectional compression, Postgres resource usage correlates with the selected compression level. For
example,here is the Postgresql application memory usage: 

No compression - 1.2 GiB

ZSTD
zstd:1 - 1.4 GiB
zstd:7 - 4.0 GiB
zstd:13 - 17.7 GiB
zstd:19 - 56.3 GiB
                                    
zstd:20 - 109.8 GiB - did not succeed
zstd:21, zstd:22  > 140 GiB
Postgres process crashes (out of memory)

ZLIB
zlib:1 - 1.35 GiB
zlib:5 - 1.35 GiB
zlib:9 - 1.35 GiB

Full report with CPU/Memory/Network consumption graph is available here:
https://docs.google.com/document/d/1qakHcsabZhV70GfSEOjFxmlUDBe21p7DRoPrDPAjKNg

Then, I’ve disabled the compression for the backend and decompression for the frontend
and measured the resource usage for single-directional compression scenario (frontend compression only, backend
decompressiononly): 

ZSTD
For all ZSTD compression levels, database host resource usage was roughly the same, except the Committed Memory
(Committed_AS):

no compression - 44.4 GiB
zstd:1 - 45.0 GiB
zstd:3 - 46.1 GiB
zstd:5 - 46.1 GiB
zstd:7 - 46.0 GiB
zstd:9 - 46.0 GiB
zstd:11 - 47.4 GiB
zstd:13 - 47.4 GiB
zstd:15 - 47.4 GiB
zstd:17 - 50.3 GiB
zstd:19 - 50.1 GiB
zstd:20 - 66.8 GiB
zstd:21 - 88.7 GiB
zstd:22 - 123.9 GiB

ZLIB
For all ZLIB compression level, database host resource usage was roughly the same.

Full report with CPU/Memory/Network consumption graph is available here:
https://docs.google.com/document/d/1gI7c3_YvcL5-PzeK65P0pIY-4BI9KBDwlfPpGhYxrNg

To sum up, there is actually almost no difference when decompressing the different compression levels, except the
Committed_ASsize.  

—
Daniil Zakhlystov




pgsql-hackers by date:

Previous
From: "Hou, Zhijie"
Date:
Subject: RE: Consider Parallelism While Planning For REFRESH MATERIALIZED VIEW
Next
From: Peter Smith
Date:
Subject: Re: Single transaction in the tablesync worker?