Re: More issues with pg_verify_checksums and checksum verificationin base backups - Mailing list pgsql-hackers

Mmm. It took too long time than expected because I was repeatedly
teased by git..

At Wed, 24 Oct 2018 14:31:37 +0900, Michael Paquier <michael@paquier.xyz> wrote in <20181024053137.GL1658@paquier.xyz>
> On Sun, Oct 21, 2018 at 08:56:32PM -0400, Stephen Frost wrote:
> > All of this pie-in-the-sky about what pluggable storage might have is
> > just hand-waving, in my opinion, and not worth much more than that.  I
> > hope (and suspect..) that the actual pluggable storage that's being
> > worked on doesn't do any of this "just drop a file somewhere" because
> > there's a lot of downsides to it- and if it did, it wouldn't be much
> > more than what we can do with an FDW, so why go through and add it?
> 
> Well, there is no point in enforcing a rule that something is forbidden
> if if was never implied and never documented (the rule here is to be
> able to drop custom files into global/, base/ or pg_tblspc.).  Postgres
> is highly-customizable, I would prefer if features in core are designed
> so as we keep things extensible, the checksum verification for base
> backup on the contrary restricts things.
> 
> So, do we have other opinions about this thread?

I believe of freedom for anyone of dropping any files into
anywhere in $PGDATA if he thinks it sefe for the time
being. Changes in core sometimes breaks some extensions and they
used to be 'fixed' in the manner or claim for a replacement to
core. This is the same for changes of (undocumented) APIs. I
think things have been going around in this manner for years and
I don't think core side is unable to define a definite border of
what extensions are allowed to do since we are not knowledgeable
of all extensions that will be created in future or that have
created.

So I'm +1 for the Michael's current patch as (I think) we can't
make visible or large changes.


That said, I agree with Stephen's concern on the point we could
omit requried files in future, but on the other hand I don't want
random files are simply rejected.

So I propose to sort files into roughly three categories. One is
files we know of but to skip. Second is files we know of and need
checksum verification. The last is the files we just don't know of.

In the attached PoC (checksum_)find_file_type categorizes a file
into 6 (or 7) categories.

ENTRY_TO_IGNORE: to be always ignored, for "." and "..".
FILE_TO_SKIP:    we know of and to skip. files in the black list.
DIR_TO_SKIP:     directories same to the above.
DIR_TO_SCAN:     we know any file to scan may be in it.
HEAP_TO_SCAN:     we know it has checksums in heap format.
FILE_UNKNOWN:    we don't know of.
(STAT_FAILED:    stat filed for the file)

Among these types, what are related to the discussion above are
FILE_TO_SKIP, HEAP_TO_SCAN and FILE_UNKNOWN. Others are for
controlling search loop in pg_verify_checksums.

Based on this categorization, pg_verify_checksum's result is shown as:

> Checksum scan completed
> Data checksum version: 1
> Files scanned:  1095
> Blocks scanned: 2976
> Bad checksums:  0
+ Files skipped: 8
+ Unknown files skipped: 1
+ Skipped directories: 1

(Data checksum version is bonded with heap checksums..)

If this sort of change is acceptable for us, I believe it
comforms everyone here's wishes. If skipped unknown is not zero
on a buildfarm animal, it is a sign of something is forgotten.

The second attached (also PoC) is common'ize the file sorter. The
function is moved to src/common/file_checksums.c and both
pg_verify_checksums.c and basebackup.c uses it.  With
log_min_messages=debug1, you will see the following message for
unkown files during basebackup.

> DEBUG:  checksum verification was skipped for unknown file: ./base/hoge

This changed one behavior of the tool. -r now can take
'dbid/relid' addition to just 'relid'.  I think we could have
.verify_checksum.exlucde file so that extensions can declare
files.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center
From 187d84a0ffc94fb9d5c9c0f6708227cc8f47fa3c Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horiguchi.kyotaro@lab.ntt.co.jp>
Date: Thu, 25 Oct 2018 16:48:47 +0900
Subject: [PATCH 1/2] Make pg_verify_checksums conscious of unknown files

---
 src/bin/pg_verify_checksums/pg_verify_checksums.c | 225 +++++++++++++++++-----
 1 file changed, 179 insertions(+), 46 deletions(-)

diff --git a/src/bin/pg_verify_checksums/pg_verify_checksums.c b/src/bin/pg_verify_checksums/pg_verify_checksums.c
index f0e09bea20..4b527913c1 100644
--- a/src/bin/pg_verify_checksums/pg_verify_checksums.c
+++ b/src/bin/pg_verify_checksums/pg_verify_checksums.c
@@ -26,6 +26,9 @@
 static int64 files = 0;
 static int64 blocks = 0;
 static int64 badblocks = 0;
+static int64 skipped_known = 0;
+static int64 skipped_unknown = 0;
+static int64 skipped_dirs = 0;
 static ControlFileData *ControlFile;
 
 static char *only_relfilenode = NULL;
@@ -33,6 +36,46 @@ static bool verbose = false;
 
 static const char *progname;
 
+/* struct for checksum verification paremter*/
+typedef struct
+{
+    union
+    {
+        struct
+        {
+            BlockNumber    segmentno;
+        } heap_param;
+    } params;
+} checksum_scan_context;
+
+/* enum for return value of find_file_type */
+typedef enum
+{
+    ENTRY_TO_IGNORE,
+    DIR_TO_SCAN,
+    HEAP_TO_SCAN,
+    FILE_TO_SKIP,
+    DIR_TO_SKIP,
+    FILE_UNKNOWN
+} checksum_file_types;
+
+/* black (explisit exclusion) list for checksum verification */
+static const char *const checksum_known_to_skip[] = {
+    "pg_control",
+     "pg_internal.init",
+    "pg_filenode.map",
+    "PG_VERSION",
+    "config_exec_params",
+    "config_exec_params.new",
+    "pgsql_tmp",                    /* this is a directory */
+    NULL
+};
+
+static checksum_file_types find_file_type(const char *fn,
+                                          const char *relfilenode,
+                                          checksum_scan_context *ctx);
+
+
 static void
 usage(void)
 {
@@ -116,11 +159,12 @@ isRelFileName(const char *fn)
 }
 
 static void
-scan_file(const char *fn, BlockNumber segmentno)
+scan_heap_file(const char *fn, checksum_scan_context *ctx)
 {
     PGAlignedBlock buf;
     PageHeader    header = (PageHeader) buf.data;
     int            f;
+    BlockNumber segmentno = ctx->params.heap_param.segmentno;
     BlockNumber blockno;
 
     f = open(fn, O_RDONLY | PG_BINARY, 0);
@@ -187,63 +231,147 @@ scan_directory(const char *basedir, const char *subdir)
     while ((de = readdir(dir)) != NULL)
     {
         char        fn[MAXPGPATH];
-        struct stat st;
-
-        if (!isRelFileName(de->d_name))
-            continue;
+        checksum_scan_context ctx;
 
         snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name);
-        if (lstat(fn, &st) < 0)
+        switch (find_file_type(fn, only_relfilenode, &ctx))
         {
-            fprintf(stderr, _("%s: could not stat file \"%s\": %s\n"),
-                    progname, fn, strerror(errno));
-            exit(1);
+        case ENTRY_TO_IGNORE:
+            continue;        /* ignore completely silently */
+        case FILE_TO_SKIP:
+            if (verbose)
+                fprintf(stderr, "skipped file: %s/%s/%s\n",
+                        basedir, subdir, de->d_name);
+            skipped_known++;
+            continue;
+        case DIR_TO_SKIP:
+            if (verbose)
+                fprintf(stderr, "skipped directory: %s/%s/%s\n",
+                        basedir, subdir, de->d_name);
+            skipped_dirs++;
+            continue;
+        case FILE_UNKNOWN:
+            if (verbose)
+                fprintf(stderr, "skipped unknown file: %s/%s/%s\n",
+                        basedir, subdir, de->d_name);
+            skipped_unknown++;
+            continue;
+        case HEAP_TO_SCAN:
+            scan_heap_file(fn, &ctx);
+            break;
+        case DIR_TO_SCAN:
+            scan_directory(path, de->d_name);
+            break;
         }
-        if (S_ISREG(st.st_mode))
+    }
+
+    closedir(dir);
+}
+
+/*
+ * find_file_type: identify what to do on a file
+ *
+ * fn is a file path in full path or relative down from the current directory.
+ * relfilenode is filter string of file. Only specified files of node number or
+ * databaseid/filenodenum will be verified checksum.
+ * ctx is the parameter needed for following checksum scan.
+ */
+static checksum_file_types
+find_file_type(const char *fn, const char *relfilenode,
+               checksum_scan_context *ctx)
+{
+    struct stat st;
+    char        fnonly[MAXPGPATH];
+    const char *fname;
+    char       *forkpath;
+    char       *segmentpath;
+    const char *const *p;
+    bool        is_subdir = false;
+
+    /* find file name the full path */
+    fname = strrchr(fn, '/');
+    if (fname)
+        fname++;
+    else
+        fname = fn;
+
+    if (strcmp(fname, ".") == 0 ||
+        strcmp(fname, "..") == 0)
+        return ENTRY_TO_IGNORE;
+
+    if (lstat(fn, &st) < 0)
+    {
+        fprintf(stderr, _("%s: could not stat file \"%s\": %s\n"),
+                progname, fn, strerror(errno));
+        exit(1);
+    }
+
+#ifndef WIN32
+    if (S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode))
+#else
+    if (S_ISDIR(st.st_mode) || pgwin32_is_junction(fn))
+#endif
+        is_subdir = true;
+
+    /* exluded by blacklist */
+
+    for (p = checksum_known_to_skip ; *p ; p++)
+    {
+        if (strcmp(*p, fname) != 0)
+            continue;
+
+        if (!is_subdir)
+            return FILE_TO_SKIP;
+        else
+            return DIR_TO_SKIP;
+    }
+
+    if (is_subdir)
+        return DIR_TO_SCAN;
+
+    /* we now know only of relfiles */
+    if (isRelFileName(fname))
+    {
+        /* copy the path so that we can scribble on it */
+        strlcpy(fnonly, fn, sizeof(fnonly));
+        ctx->params.heap_param.segmentno = 0;
+        segmentpath = strchr(fnonly, '.');
+
+        /* make sure that the dot is in the last segment in the path  */
+        if (segmentpath != NULL && strchr(segmentpath, '/') == NULL)
         {
-            char        fnonly[MAXPGPATH];
-            char       *forkpath,
-                       *segmentpath;
-            BlockNumber segmentno = 0;
+            *segmentpath++ = '\0';
+            ctx->params.heap_param.segmentno = atoi(segmentpath);
 
-            /*
-             * Cut off at the segment boundary (".") to get the segment number
-             * in order to mix it into the checksum. Then also cut off at the
-             * fork boundary, to get the relfilenode the file belongs to for
-             * filtering.
-             */
-            strlcpy(fnonly, de->d_name, sizeof(fnonly));
-            segmentpath = strchr(fnonly, '.');
-            if (segmentpath != NULL)
-            {
-                *segmentpath++ = '\0';
-                segmentno = atoi(segmentpath);
-                if (segmentno == 0)
-                {
-                    fprintf(stderr, _("%s: invalid segment number %d in file name \"%s\"\n"),
-                            progname, segmentno, fn);
-                    exit(1);
-                }
-            }
+            /* something's wrong, treat it as unknown file  */
+            if (ctx->params.heap_param.segmentno == 0)
+                return FILE_UNKNOWN;
+        }
+    
+        if (only_relfilenode)
+        {
+            char *p;
 
-            forkpath = strchr(fnonly, '_');
-            if (forkpath != NULL)
+            /* find file suffix if any */
+            forkpath = strrchr(fnonly, '_');
+
+            /* the underscore must be in the last segment in the path */
+            if (forkpath != NULL && strchr(forkpath, '/') == NULL)
                 *forkpath++ = '\0';
 
-            if (only_relfilenode && strcmp(only_relfilenode, fnonly) != 0)
+            /* make a tail match with only_relfilenode */
+            p = fnonly + strlen(fnonly) - strlen(relfilenode);
+            if (fnonly > p ||                     /* cannot match*/
+                (fnonly < p && *(p-1) != '/') || /* avoid false match */
+                strcmp(relfilenode, p) != 0)
                 /* Relfilenode not to be included */
-                continue;
-
-            scan_file(fn, segmentno);
+                return FILE_TO_SKIP;
         }
-#ifndef WIN32
-        else if (S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode))
-#else
-        else if (S_ISDIR(st.st_mode) || pgwin32_is_junction(fn))
-#endif
-            scan_directory(path, de->d_name);
+
+        return HEAP_TO_SCAN;
     }
-    closedir(dir);
+
+    return FILE_UNKNOWN;
 }
 
 int
@@ -359,6 +487,11 @@ main(int argc, char *argv[])
     printf(_("Files scanned:  %s\n"), psprintf(INT64_FORMAT, files));
     printf(_("Blocks scanned: %s\n"), psprintf(INT64_FORMAT, blocks));
     printf(_("Bad checksums:  %s\n"), psprintf(INT64_FORMAT, badblocks));
+    printf(_("Files skipped: %s\n"),  psprintf(INT64_FORMAT, skipped_known));
+    printf(_("Unknown files skipped: %s\n"),
+           psprintf(INT64_FORMAT, skipped_unknown));
+    printf(_("Skipped directories: %s\n"),
+           psprintf(INT64_FORMAT, skipped_dirs));
 
     if (badblocks > 0)
         return 1;
-- 
2.16.3

From 87baf73f56f688f4532ef78f6684934a47be3ba2 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horiguchi.kyotaro@lab.ntt.co.jp>
Date: Thu, 25 Oct 2018 16:49:46 +0900
Subject: [PATCH 2/2] Common'ize file type checker for checksums

pg_verify_checksums.c and basebackup.c has the same notion of 'files
that have checksums'. This patch moves the core logic so that
src/common and the both files share the logic.
---
 src/backend/replication/basebackup.c              |  43 +++--
 src/bin/pg_verify_checksums/Makefile              |   3 +-
 src/bin/pg_verify_checksums/pg_verify_checksums.c | 220 +---------------------
 src/common/Makefile                               |   3 +-
 src/common/file_checksums.c                       | 197 +++++++++++++++++++
 src/include/common/file_checksums.h               |  42 +++++
 6 files changed, 273 insertions(+), 235 deletions(-)
 create mode 100644 src/common/file_checksums.c
 create mode 100644 src/include/common/file_checksums.h

diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index b20f6c379c..4ebc969f3d 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -19,6 +19,7 @@
 #include "access/xlog_internal.h"    /* for pg_start/stop_backup */
 #include "catalog/pg_type.h"
 #include "common/file_perm.h"
+#include "common/file_checksums.h"
 #include "lib/stringinfo.h"
 #include "libpq/libpq.h"
 #include "libpq/pqformat.h"
@@ -187,18 +188,6 @@ static const char *excludeFiles[] =
     NULL
 };
 
-/*
- * List of files excluded from checksum validation.
- */
-static const char *const noChecksumFiles[] = {
-    "pg_control",
-    "pg_filenode.map",
-    "pg_internal.init",
-    "PG_VERSION",
-    NULL,
-};
-
-
 /*
  * Called when ERROR or FATAL happens in perform_base_backup() after
  * we have started the backup - make sure we end it!
@@ -1321,22 +1310,36 @@ sendDir(const char *path, int basepathlen, bool sizeonly, List *tablespaces,
 static bool
 is_checksummed_file(const char *fullpath, const char *filename)
 {
-    const char *const *f;
+    checksum_scan_context ctx;
 
     /* Check that the file is in a tablespace */
     if (strncmp(fullpath, "./global/", 9) == 0 ||
         strncmp(fullpath, "./base/", 7) == 0 ||
         strncmp(fullpath, "/", 1) == 0)
     {
-        /* Compare file against noChecksumFiles skiplist */
-        for (f = noChecksumFiles; *f; f++)
-            if (strcmp(*f, filename) == 0)
-                return false;
+        /* check if the file has checksums */
+        switch (checksum_find_file_type(fullpath, NULL, &ctx))
+        {
+        case HEAP_TO_SCAN:
+            return true;
+        case STAT_FAILED:
+            ereport(ERROR,
+                    (errcode_for_file_access(),
+                     errmsg("failed to stat \"%s\": %m",
+                            fullpath)));
 
-        return true;
+        case ENTRY_TO_IGNORE:
+        case FILE_TO_SKIP:
+        case DIR_TO_SKIP:
+        case DIR_TO_SCAN:
+            break;
+        case FILE_UNKNOWN:
+            elog(DEBUG1, "checksum verification was skipped for unknown file: %s", fullpath);
+            break;
+        }
     }
-    else
-        return false;
+
+    return false;
 }
 
 /*****
diff --git a/src/bin/pg_verify_checksums/Makefile b/src/bin/pg_verify_checksums/Makefile
index cfe4ab1b8b..3d0a9baf24 100644
--- a/src/bin/pg_verify_checksums/Makefile
+++ b/src/bin/pg_verify_checksums/Makefile
@@ -15,7 +15,8 @@ subdir = src/bin/pg_verify_checksums
 top_builddir = ../../..
 include $(top_builddir)/src/Makefile.global
 
-OBJS= pg_verify_checksums.o $(WIN32RES)
+OBJS= pg_verify_checksums.o $(top_builddir)/src/common/file_checksums.o \
+    $(WIN32RES)
 
 all: pg_verify_checksums
 
diff --git a/src/bin/pg_verify_checksums/pg_verify_checksums.c b/src/bin/pg_verify_checksums/pg_verify_checksums.c
index 4b527913c1..dc2143ea65 100644
--- a/src/bin/pg_verify_checksums/pg_verify_checksums.c
+++ b/src/bin/pg_verify_checksums/pg_verify_checksums.c
@@ -15,7 +15,7 @@
 
 #include "catalog/pg_control.h"
 #include "common/controldata_utils.h"
-#include "common/relpath.h"
+#include "common/file_checksums.h"
 #include "getopt_long.h"
 #include "pg_getopt.h"
 #include "storage/bufpage.h"
@@ -36,46 +36,6 @@ static bool verbose = false;
 
 static const char *progname;
 
-/* struct for checksum verification paremter*/
-typedef struct
-{
-    union
-    {
-        struct
-        {
-            BlockNumber    segmentno;
-        } heap_param;
-    } params;
-} checksum_scan_context;
-
-/* enum for return value of find_file_type */
-typedef enum
-{
-    ENTRY_TO_IGNORE,
-    DIR_TO_SCAN,
-    HEAP_TO_SCAN,
-    FILE_TO_SKIP,
-    DIR_TO_SKIP,
-    FILE_UNKNOWN
-} checksum_file_types;
-
-/* black (explisit exclusion) list for checksum verification */
-static const char *const checksum_known_to_skip[] = {
-    "pg_control",
-     "pg_internal.init",
-    "pg_filenode.map",
-    "PG_VERSION",
-    "config_exec_params",
-    "config_exec_params.new",
-    "pgsql_tmp",                    /* this is a directory */
-    NULL
-};
-
-static checksum_file_types find_file_type(const char *fn,
-                                          const char *relfilenode,
-                                          checksum_scan_context *ctx);
-
-
 static void
 usage(void)
 {
@@ -93,71 +53,6 @@ usage(void)
     printf(_("Report bugs to <pgsql-bugs@postgresql.org>.\n"));
 }
 
-/*
- * isRelFileName
- *
- * Check if the given file name is authorized for checksum verification.
- */
-static bool
-isRelFileName(const char *fn)
-{
-    int            pos;
-
-    /*----------
-     * Only files including data checksums are authorized for verification.
-     * This is guessed based on the file name by reverse-engineering
-     * GetRelationPath() so make sure to update both code paths if any
-     * updates are done.  The following file name formats are allowed:
-     * <digits>
-     * <digits>.<segment>
-     * <digits>_<forkname>
-     * <digits>_<forkname>.<segment>
-     *
-     * Note that temporary files, beginning with 't', are also skipped.
-     *
-     *----------
-     */
-
-    /* A non-empty string of digits should follow */
-    for (pos = 0; isdigit((unsigned char) fn[pos]); ++pos)
-        ;
-    /* leave if no digits */
-    if (pos == 0)
-        return false;
-    /* good to go if only digits */
-    if (fn[pos] == '\0')
-        return true;
-
-    /* Authorized fork files can be scanned */
-    if (fn[pos] == '_')
-    {
-        int            forkchar = forkname_chars(&fn[pos + 1], NULL);
-
-        if (forkchar <= 0)
-            return false;
-
-        pos += forkchar + 1;
-    }
-
-    /* Check for an optional segment number */
-    if (fn[pos] == '.')
-    {
-        int            segchar;
-
-        for (segchar = 1; isdigit((unsigned char) fn[pos + segchar]); ++segchar)
-            ;
-
-        if (segchar <= 1)
-            return false;
-        pos += segchar;
-    }
-
-    /* Now this should be the end */
-    if (fn[pos] != '\0')
-        return false;
-    return true;
-}
-
 static void
 scan_heap_file(const char *fn, checksum_scan_context *ctx)
 {
@@ -234,7 +129,7 @@ scan_directory(const char *basedir, const char *subdir)
         checksum_scan_context ctx;
 
         snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name);
-        switch (find_file_type(fn, only_relfilenode, &ctx))
+        switch (checksum_find_file_type(fn, only_relfilenode, &ctx))
         {
         case ENTRY_TO_IGNORE:
             continue;        /* ignore completely silently */
@@ -262,118 +157,16 @@ scan_directory(const char *basedir, const char *subdir)
         case DIR_TO_SCAN:
             scan_directory(path, de->d_name);
             break;
+        case STAT_FAILED:
+            fprintf(stderr, _("%s: could not stat file \"%s\": %s\n"),
+                    progname, fn, strerror(errno));
+            exit(1);
         }
     }
 
     closedir(dir);
 }
 
-/*
- * find_file_type: identify what to do on a file
- *
- * fn is a file path in full path or relative down from the current directory.
- * relfilenode is filter string of file. Only specified files of node number or
- * databaseid/filenodenum will be verified checksum.
- * ctx is the parameter needed for following checksum scan.
- */
-static checksum_file_types
-find_file_type(const char *fn, const char *relfilenode,
-               checksum_scan_context *ctx)
-{
-    struct stat st;
-    char        fnonly[MAXPGPATH];
-    const char *fname;
-    char       *forkpath;
-    char       *segmentpath;
-    const char *const *p;
-    bool        is_subdir = false;
-
-    /* find file name the full path */
-    fname = strrchr(fn, '/');
-    if (fname)
-        fname++;
-    else
-        fname = fn;
-
-    if (strcmp(fname, ".") == 0 ||
-        strcmp(fname, "..") == 0)
-        return ENTRY_TO_IGNORE;
-
-    if (lstat(fn, &st) < 0)
-    {
-        fprintf(stderr, _("%s: could not stat file \"%s\": %s\n"),
-                progname, fn, strerror(errno));
-        exit(1);
-    }
-
-#ifndef WIN32
-    if (S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode))
-#else
-    if (S_ISDIR(st.st_mode) || pgwin32_is_junction(fn))
-#endif
-        is_subdir = true;
-
-    /* exluded by blacklist */
-
-    for (p = checksum_known_to_skip ; *p ; p++)
-    {
-        if (strcmp(*p, fname) != 0)
-            continue;
-
-        if (!is_subdir)
-            return FILE_TO_SKIP;
-        else
-            return DIR_TO_SKIP;
-    }
-
-    if (is_subdir)
-        return DIR_TO_SCAN;
-
-    /* we now know only of relfiles */
-    if (isRelFileName(fname))
-    {
-        /* copy the path so that we can scribble on it */
-        strlcpy(fnonly, fn, sizeof(fnonly));
-        ctx->params.heap_param.segmentno = 0;
-        segmentpath = strchr(fnonly, '.');
-
-        /* make sure that the dot is in the last segment in the path  */
-        if (segmentpath != NULL && strchr(segmentpath, '/') == NULL)
-        {
-            *segmentpath++ = '\0';
-            ctx->params.heap_param.segmentno = atoi(segmentpath);
-
-            /* something's wrong, treat it as unknown file  */
-            if (ctx->params.heap_param.segmentno == 0)
-                return FILE_UNKNOWN;
-        }
-    
-        if (only_relfilenode)
-        {
-            char *p;
-
-            /* find file suffix if any */
-            forkpath = strrchr(fnonly, '_');
-
-            /* the underscore must be in the last segment in the path */
-            if (forkpath != NULL && strchr(forkpath, '/') == NULL)
-                *forkpath++ = '\0';
-
-            /* make a tail match with only_relfilenode */
-            p = fnonly + strlen(fnonly) - strlen(relfilenode);
-            if (fnonly > p ||                     /* cannot match*/
-                (fnonly < p && *(p-1) != '/') || /* avoid false match */
-                strcmp(relfilenode, p) != 0)
-                /* Relfilenode not to be included */
-                return FILE_TO_SKIP;
-        }
-
-        return HEAP_TO_SCAN;
-    }
-
-    return FILE_UNKNOWN;
-}
-
 int
 main(int argc, char *argv[])
 {
@@ -397,6 +190,7 @@ main(int argc, char *argv[])
         if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
         {
             usage();
+
             exit(0);
         }
         if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)
diff --git a/src/common/Makefile b/src/common/Makefile
index ec8139f014..54b7c9f440 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -44,7 +44,8 @@ override CPPFLAGS += -DVAL_LIBS="\"$(LIBS)\""
 override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
 LIBS += $(PTHREAD_LIBS)
 
-OBJS_COMMON = base64.o config_info.o controldata_utils.o exec.o file_perm.o \
+OBJS_COMMON = base64.o config_info.o controldata_utils.o exec.o \
+    file_checksums.o file_perm.o \
     ip.o keywords.o link-canary.o md5.o pg_lzcompress.o \
     pgfnames.o psprintf.o relpath.o \
     rmtree.o saslprep.o scram-common.o string.o unicode_norm.o \
diff --git a/src/common/file_checksums.c b/src/common/file_checksums.c
new file mode 100644
index 0000000000..f83bb52c1d
--- /dev/null
+++ b/src/common/file_checksums.c
@@ -0,0 +1,197 @@
+/*-------------------------------------------------------------------------
+ * file_checksums.c
+ *        checksumming files
+ *
+ * This implements Unicode normalization, per the documentation at
+ * http://www.unicode.org/reports/tr15/.
+ *
+ * Portions Copyright (c) 2018, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ *      src/common/file_checksums.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include <sys/stat.h>
+
+#include "c.h"
+#include "common/file_checksums.h"
+#include "common/relpath.h"
+
+/* black (explisit exclusion) list for checksum verification */
+static const char *const checksum_known_to_skip[] = {
+    "pg_control",
+     "pg_internal.init",
+    "pg_filenode.map",
+    "PG_VERSION",
+    "config_exec_params",
+    "config_exec_params.new",
+    "pgsql_tmp",        /* directory */
+    NULL
+};
+
+/*
+ * isRelFileName
+ *
+ * Check if the given file name is authorized for checksum verification.
+ */
+static bool
+isRelFileName(const char *fn)
+{
+    int            pos;
+
+    /*----------
+     * Only files including data checksums are authorized for verification.
+     * This is guessed based on the file name by reverse-engineering
+     * GetRelationPath() so make sure to update both code paths if any
+     * updates are done.  The following file name formats are allowed:
+     * <digits>
+     * <digits>.<segment>
+     * <digits>_<forkname>
+     * <digits>_<forkname>.<segment>
+     *
+     * Note that temporary files, beginning with 't', are also skipped.
+     *
+     *----------
+     */
+
+    /* A non-empty string of digits should follow */
+    for (pos = 0; isdigit((unsigned char) fn[pos]); ++pos)
+        ;
+    /* leave if no digits */
+    if (pos == 0)
+        return false;
+    /* good to go if only digits */
+    if (fn[pos] == '\0')
+        return true;
+
+    /* Authorized fork files can be scanned */
+    if (fn[pos] == '_')
+    {
+        int            forkchar = forkname_chars(&fn[pos + 1], NULL);
+
+        if (forkchar <= 0)
+            return false;
+
+        pos += forkchar + 1;
+    }
+
+    /* Check for an optional segment number */
+    if (fn[pos] == '.')
+    {
+        int            segchar;
+
+        for (segchar = 1; isdigit((unsigned char) fn[pos + segchar]); ++segchar)
+            ;
+
+        if (segchar <= 1)
+            return false;
+        pos += segchar;
+    }
+
+    /* Now this should be the end */
+    if (fn[pos] != '\0')
+        return false;
+    return true;
+}
+
+/*
+ * checksum_find_file_type: identify a file from the viewpoint of checksum
+ *
+ * fn is file name with full path to check
+ * relfilenode is relfilenode in string to exclude files other than that.
+ * ctx is the context to scan checksum, which contains parameters for scanners.
+ */
+checksum_file_types
+checksum_find_file_type(const char *fn,
+                        const char *relfilenode, checksum_scan_context *ctx)
+{
+    struct stat st;
+    char        fnonly[MAXPGPATH];
+    char       *fname;
+    char       *forkpath;
+    char       *segmentpath;
+    const char *const *p;
+    bool        is_subdir = false;
+
+    fname = strrchr(fn, '/');
+
+    if (fname == NULL)
+        return ENTRY_TO_IGNORE;
+
+    fname++;
+
+    if (strcmp(fname, ".") == 0 ||
+        strcmp(fname, "..") == 0)
+        return ENTRY_TO_IGNORE;
+
+    if (lstat(fn, &st) < 0)
+        return STAT_FAILED;
+
+#ifndef WIN32
+    if (S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode))
+#else
+    if (S_ISDIR(st.st_mode) || pgwin32_is_junction(fn))
+#endif
+        is_subdir = true;
+
+    /* exluded by blacklist */
+
+    for (p = checksum_known_to_skip ; *p ; p++)
+    {
+        if (strcmp(*p, fname) != 0)
+            continue;
+
+        if (is_subdir)
+            return DIR_TO_SKIP;
+
+        return FILE_TO_SKIP;
+    }
+
+    if (is_subdir)
+        return DIR_TO_SCAN;
+
+    /* we now know only of relfiles */
+    if (isRelFileName(fname))
+    {
+        /* copy the path so that we can scribble on it */
+        strlcpy(fnonly, fn, sizeof(fnonly));
+        ctx->params.heap_param.segmentno = 0;
+        segmentpath = strchr(fnonly, '.');
+
+        /* make sure that the dot is in the last segment in the path  */
+        if (segmentpath != NULL && strchr(segmentpath, '/') == NULL)
+        {
+            *segmentpath++ = '\0';
+            ctx->params.heap_param.segmentno = atoi(segmentpath);
+
+            /* something's wrong, treat it as unknown file  */
+            if (ctx->params.heap_param.segmentno == 0)
+                return FILE_UNKNOWN;
+        }
+    
+        if (relfilenode)
+        {
+            char *p;
+
+            /* find file suffix if any */
+            forkpath = strrchr(fnonly, '_');
+
+            /* the underscore must be in the last segment in the path */
+            if (forkpath != NULL && strchr(forkpath, '/') == NULL)
+                *forkpath++ = '\0';
+
+            /* make a tail match with only_relfilenode */
+            p = fnonly + strlen(fnonly) - strlen(relfilenode);
+            if (fnonly > p ||                     /* cannot match*/
+                (fnonly < p && *(p-1) != '/') || /* avoid false match */
+                strcmp(relfilenode, p) != 0)
+                /* Relfilenode not to be included */
+                return FILE_TO_SKIP;
+        }
+
+        return HEAP_TO_SCAN;
+    }
+
+    return FILE_UNKNOWN;
+}
diff --git a/src/include/common/file_checksums.h b/src/include/common/file_checksums.h
new file mode 100644
index 0000000000..3ead25c97f
--- /dev/null
+++ b/src/include/common/file_checksums.h
@@ -0,0 +1,42 @@
+/*
+ *    file_checksums.h
+ *        checksumming files
+ *
+ *    Copyright (c) 2018, PostgreSQL Global Development Group
+ *
+ *    src/include/common/file_checksums.h
+ */
+#ifndef FILE_CHECKSUMS_H
+#define FILE_CHECKSUMS_H
+
+#include "storage/block.h"
+
+/* struct for checksum verification paremter*/
+typedef struct
+{
+    union
+    {
+        struct
+        {
+            BlockNumber    segmentno;
+        } heap_param;
+    } params;
+} checksum_scan_context;
+
+/* enum for return value of find_file_type */
+typedef enum
+{
+    ENTRY_TO_IGNORE,
+    DIR_TO_SCAN,
+    HEAP_TO_SCAN,
+    FILE_TO_SKIP,
+    DIR_TO_SKIP,
+    FILE_UNKNOWN,
+    STAT_FAILED
+} checksum_file_types;
+
+checksum_file_types checksum_find_file_type(const char *fn,
+                                            const char *relfilenode,
+                                            checksum_scan_context *ctx);
+
+#endif                            /* FILE_CHECKSUMS_H */
-- 
2.16.3