Thread: Occasional tablespace.sql failures in check-world -jnn

Hi,

For fairly obvious reasons I like to run check-world in parallel [1]. In
the last few months I've occasionally seen failures during that that I
cannot recall seeing before.

--- /home/andres/build/postgres/13-assert/vpath/src/test/regress/expected/tablespace.out        2020-12-07
18:41:23.079235588-0800
 
+++ /home/andres/build/postgres/13-assert/vpath/src/test/regress/results/tablespace.out 2020-12-07 18:42:01.892632468
-0800
@@ -209,496 +209,344 @@
 ERROR:  cannot specify default tablespace for partitioned relations
 CREATE TABLE testschema.dflt (a int PRIMARY KEY USING INDEX TABLESPACE regress_tblspace) PARTITION BY LIST (a);
 ERROR:  cannot specify default tablespace for partitioned relations
 -- but these work:
 CREATE TABLE testschema.dflt (a int PRIMARY KEY USING INDEX TABLESPACE regress_tblspace) PARTITION BY LIST (a)
TABLESPACEregress_tblspace;
 
 SET default_tablespace TO '';
 CREATE TABLE testschema.dflt2 (a int PRIMARY KEY) PARTITION BY LIST (a);
 DROP TABLE testschema.dflt, testschema.dflt2;
 -- check that default_tablespace doesn't affect ALTER TABLE index rebuilds
 CREATE TABLE testschema.test_default_tab(id bigint) TABLESPACE regress_tblspace;
+ERROR:  could not create directory "pg_tblspc/16387/PG_13_202007201/16384": No such file or directory
 INSERT INTO testschema.test_default_tab VALUES (1);

(many failures follow)


I suspect this is related to the pg_upgrade test and the main regression
test running at the same time. We have the following in src/test/regress/GNUMakefile

# Tablespace setup

.PHONY: tablespace-setup
tablespace-setup:
    echo $(realpath ./testtablespace) >> /tmp/tablespace.log
    rm -rf ./testtablespace
    mkdir ./testtablespace
...

which pg_upgrade triggers. Even though it, as far as I can tell, never
actually ends up putting any data in it:

# Send installcheck outputs to a private directory.  This avoids conflict when
# check-world runs pg_upgrade check concurrently with src/test/regress check.
# To retrieve interesting files after a run, use pattern tmp_check/*/*.diffs.
outputdir="$temp_root/regress"
EXTRA_REGRESS_OPTS="$EXTRA_REGRESS_OPTS --outputdir=$outputdir"
export EXTRA_REGRESS_OPTS
mkdir "$outputdir"
mkdir "$outputdir"/testtablespace

It's not clear to me why we have this logic in the makefile at all?
Somebody taught pg_regress to do so, but only on windows... See
convert_sourcefiles_in().


The other thing that confuses me is why I started getting that error in
*multiple* branches recently, even though I have used the parallel
check-world for ages.

Greetings,

Andres Freund

[1]: make -Otarget -j20 -s check-world && echo success || echo failed

On Tue, Dec 08, 2020 at 05:29:11PM -0800, Andres Freund wrote:
> I suspect this is related to the pg_upgrade test and the main regression
> test running at the same time. We have the following in
> src/test/regress/GNUMakefile.

Yes, this one is not completely new to -hackers.  See patch 0002 here
that slightly touched the topic by creating a specific makefile rule,
but I never got back to it as I never got annoyed by this problem:
https://www.postgresql.org/message-id/20200511.171354.514381788845037011.horikyota.ntt@gmail.com
What we have here is not a solution though...

> It's not clear to me why we have this logic in the makefile at all?
> Somebody taught pg_regress to do so, but only on windows... See
> convert_sourcefiles_in().

...  Because we may still introduce this problem again if some new
stuff uses src/test/pg_regress in a way similar to pg_upgrade,
triggering again tablespace-setup.  Something like the attached may be
enough, though I have not spent much time checking the surroundings,
Windows included.

> The other thing that confuses me is why I started getting that error in
> *multiple* branches recently, even though I have used the parallel
> check-world for ages.

Perhaps you have just increased -j lately or moved to a faster machine
where there are higher changes of collision?
--
Michael

On 2020-12-09 02:29, Andres Freund wrote:
> I suspect this is related to the pg_upgrade test and the main regression
> test running at the same time. We have the following in src/test/regress/GNUMakefile
> 
> # Tablespace setup
> 
> .PHONY: tablespace-setup
> tablespace-setup:
>     echo $(realpath ./testtablespace) >> /tmp/tablespace.log
>     rm -rf ./testtablespace
>     mkdir ./testtablespace
> ...
> 
> which pg_upgrade triggers. Even though it, as far as I can tell, never
> actually ends up putting any data in it:
> 
> # Send installcheck outputs to a private directory.  This avoids conflict when
> # check-world runs pg_upgrade check concurrently with src/test/regress check.
> # To retrieve interesting files after a run, use pattern tmp_check/*/*.diffs.
> outputdir="$temp_root/regress"
> EXTRA_REGRESS_OPTS="$EXTRA_REGRESS_OPTS --outputdir=$outputdir"
> export EXTRA_REGRESS_OPTS
> mkdir "$outputdir"
> mkdir "$outputdir"/testtablespace
> 
> It's not clear to me why we have this logic in the makefile at all?
> Somebody taught pg_regress to do so, but only on windows... See
> convert_sourcefiles_in().

I vaguely recall that this had something to do with SELinux (or 
something similar?), where it matters in what context you create a file 
or directory and then certain properties attach to it that are relevant 
to subsequent programs that run on it.  Again, vague.

On Fri, Jan 15, 2021 at 09:59:02AM +0100, Peter Eisentraut wrote:
> I vaguely recall that this had something to do with SELinux (or something
> similar?), where it matters in what context you create a file or directory
> and then certain properties attach to it that are relevant to subsequent
> programs that run on it.  Again, vague.

Hmm.  Does it?  sepgsql has some tests for tablespaces involving only
pg_default, so it does not seem that this applies in the context of
the regression tests.  The cleanup of testtablespace in GNUMakefile
comes from 2467394, as of June 2004, that introduced tablespaces.
--
Michael