Re: Teach pg_receivewal to use lz4 compression - Mailing list pgsql-hackers

From Jian Guo
Subject Re: Teach pg_receivewal to use lz4 compression
Date
Msg-id 163133656252.1167.2818066389662879524.pgcf@coridan.postgresql.org
Whole thread Raw
In response to Re: Teach pg_receivewal to use lz4 compression  (gkokolatos@pm.me)
Responses Re: Teach pg_receivewal to use lz4 compression
List pgsql-hackers
@@ -250,14 +302,18 @@ FindStreamingStart(uint32 *tli)
         /*
          * Check that the segment has the right size, if it's supposed to be
          * completed.  For non-compressed segments just check the on-disk size
-         * and see if it matches a completed segment. For compressed segments,
-         * look at the last 4 bytes of the compressed file, which is where the
-         * uncompressed size is located for gz files with a size lower than
-         * 4GB, and then compare it to the size of a completed segment. The 4
-         * last bytes correspond to the ISIZE member according to
+         * and see if it matches a completed segment. For zlib compressed
+         * segments, look at the last 4 bytes of the compressed file, which is
+         * where the uncompressed size is located for gz files with a size lower
+         * than 4GB, and then compare it to the size of a completed segment.
+         * The 4 last bytes correspond to the ISIZE member according to
          * http://www.zlib.org/rfc-gzip.html.
+         *
+         * For lz4 compressed segments read the header using the exposed API and
+         * compare the uncompressed file size, stored in
+         * LZ4F_frameInfo_t{.contentSize}, to that of a completed segment.
          */
-        if (!ispartial && !iscompress)
+        if (!ispartial && wal_compression_method == COMPRESSION_NONE)
         {
             struct stat statbuf;
             char        fullpath[MAXPGPATH * 2];
@@ -276,7 +332,7 @@ FindStreamingStart(uint32 *tli)
                 continue;
             }
         }
-        else if (!ispartial && iscompress)
+        else if (!ispartial && wal_compression_method == COMPRESSION_ZLIB)
         {
             int            fd;
             char        buf[4];
@@ -322,6 +378,70 @@ FindStreamingStart(uint32 *tli)
                 continue;
             }
         }
+        else if (!ispartial && compression_method == COMPRESSION_LZ4)
+        {
+#ifdef HAVE_LIBLZ4
+            int            fd;
+            int            r;
+            size_t        consumed_len = LZ4F_HEADER_SIZE_MAX;
+            char        buf[LZ4F_HEADER_SIZE_MAX];
+            char        fullpath[MAXPGPATH * 2];
+            LZ4F_frameInfo_t frame_info = { 0 };
+            LZ4F_decompressionContext_t ctx = NULL;
+
+            snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+            fd = open(fullpath, O_RDONLY | PG_BINARY, 0);

Should close the fd before exit or abort.

pgsql-hackers by date:

Previous
From: Dilip Kumar
Date:
Subject: Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints
Next
From: Michael Paquier
Date:
Subject: Re: strange case of "if ((a & b))"