Compression of full-page-writes - Mailing list pgsql-hackers
From | Fujii Masao |
---|---|
Subject | Compression of full-page-writes |
Date | |
Msg-id | CAHGQGwGqG8e9YN0fNCUZqTTT=hNr7Ly516kfT5ffqf4pp1qnHg@mail.gmail.com Whole thread Raw |
Responses |
Re: Compression of full-page-writes
(Satoshi Nagayasu <snaga@uptime.jp>)
Re: Compression of full-page-writes (Peter Geoghegan <pg@heroku.com>) Re: Compression of full-page-writes (Amit Kapila <amit.kapila16@gmail.com>) Re: Compression of full-page-writes (KONDO Mitsumasa <kondo.mitsumasa@lab.ntt.co.jp>) Re: Compression of full-page-writes (Nikhil Sontakke <nikkhils@gmail.com>) Re: Compression of full-page-writes (Michael Paquier <michael.paquier@gmail.com>) Re: Compression of full-page-writes (Heikki Linnakangas <hlinnakangas@vmware.com>) Re: Compression of full-page-writes (Robert Haas <robertmhaas@gmail.com>) Re: Compression of full-page-writes (Fujii Masao <masao.fujii@gmail.com>) Re: Compression of full-page-writes (Simon Riggs <simon@2ndQuadrant.com>) Re: Compression of full-page-writes (Rahila Syed <rahilasyed.90@gmail.com>) |
List | pgsql-hackers |
Hi, Attached patch adds new GUC parameter 'compress_backup_block'. When this parameter is enabled, the server just compresses FPW (full-page-writes) in WAL by using pglz_compress() before inserting it to the WAL buffers. Then, the compressed FPW is decompressed in recovery. This is very simple patch. The purpose of this patch is the reduction of WAL size. Under heavy write load, the server needs to write a large amount of WAL and this is likely to be a bottleneck. What's the worse is, in replication, a large amount of WAL would have harmful effect on not only WAL writing in the master, but also WAL streaming and WAL writing in the standby. Also we would need to spend more money on the storage to store such a large data. I'd like to alleviate such harmful situations by reducing WAL size. My idea is very simple, just compress FPW because FPW is a big part of WAL. I used pglz_compress() as a compression method, but you might think that other method is better. We can add something like FPW-compression-hook for that later. The patch adds new GUC parameter, but I'm thinking to merge it to full_page_writes parameter to avoid increasing the number of GUC. That is, I'm thinking to change full_page_writes so that it can accept new value 'compress'. I measured how much WAL this patch can reduce, by using pgbench. * Server spec CPU: 8core, Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz Mem: 16GB Disk: 500GB SSD Samsung 840 * Benchmark pgbench -c 32 -j 4 -T 900 -M prepared scaling factor: 100 checkpoint_segments = 1024 checkpoint_timeout = 5min (every checkpoint during benchmark were triggered by checkpoint_timeout) * Result [tps] 1386.8 (compress_backup_block = off) 1627.7 (compress_backup_block = on) [the amount of WAL generated during running pgbench] 4302 MB (compress_backup_block = off) 1521 MB (compress_backup_block = on) At least in my test, the patch could reduce the WAL size to one-third! The patch is WIP yet. But I'd like to hear the opinions about this idea before completing it, and then add the patch to next CF if okay. Regards, -- Fujii Masao
Attachment
pgsql-hackers by date: