Reducing buildfarm disk usage: remove temp installs when done - Mailing list pgsql-hackers

One of the biggest causes of buildfarm run failures is "out of disk
space".  That's not just because people are running buildfarm critters
on small slow machines; it's because "make check-world" is an enormous
space hog.  Some numbers from current HEAD:

clean source tree:        120MB
built source tree:        400MB
tree after make check-world:    3GB

(This is excluding ~250MB for one's git repo.)

The reason for all the bloat is the temporary install trees that we
create, which tend to eat up about 100MB apiece, and there are dozens
of them (eg, one per testable contrib module).  Those don't get removed
until the end of the test run, so the usage is cumulative.

The attached proposed patch removes each temp install tree as soon as
we're done with it, in the normal case where no error was detected.
This brings the peak space usage down from ~3GB to ~750MB.

To make things better in the buildfarm, we'd have to back-patch this into
all active branches, but I don't see any big problem with doing so.

Any objections?

            regards, tom lane

diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index caae3f0..ee3b80b 100644
*** a/src/test/regress/pg_regress.c
--- b/src/test/regress/pg_regress.c
*************** regression_main(int argc, char *argv[],
*** 2668,2673 ****
--- 2668,2686 ----
          stop_postmaster();
      }

+     /*
+      * If there were no errors, remove the temp installation immediately to
+      * conserve disk space.  (If there were errors, we leave the installation
+      * in place for possible manual investigation.)
+      */
+     if (temp_install && fail_count == 0 && fail_ignore_count == 0)
+     {
+         header(_("removing temporary installation"));
+         if (!rmtree(temp_install, true))
+             fprintf(stderr, _("\n%s: could not remove temp installation \"%s\": %s\n"),
+                     progname, temp_install, strerror(errno));
+     }
+
      fclose(logfile);

      /*

pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: Re: Better way of dealing with pgstat wait timeout during buildfarm runs?
Next
From: Michael Paquier
Date:
Subject: Re: Reducing buildfarm disk usage: remove temp installs when done