Dear Andres,
While tracking BF failures related with pg_ugprade, I found the same failure has still happened [1] - [4].
According to the log, the output directory was remained even after the successful upgrade [5].
I analyzed and attached the fix patch, and below is my analysis... how do you think?
=====
lstat() seemed fail while doing the second try of rmtree(). This error message is
output from get_dirent_type().
Apart from pgunlink(), get_dirent_type() does not have an retry mechanism when
lstat()->_pglstat64() detects STATUS_DELETE_PENDING. Therefore, I think rmtree()
may not wait the file until it would be really removed, if the status is deceted
in the get_dirent_type().
One solution is to retry stat() or lstat() even in get_dirent_type(), like attached.
[1]: 2023-07-21 02:21:53 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2023-07-21%2002%3A21%3A53
[2]: 2023-10-21 13:39:15 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2023-10-21%2013%3A39%3A15
[3]: 2023-10-23 09:03:07 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2023-10-23%2009%3A03%3A07
[4]: 2023-10-27 23:06:17 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2023-10-27%2023%3A06%3A17
[5]
```
...
*Clusters are compatible*
pg_upgrade: warning: could not remove directory
"C:/tools/nmsys64/home/pgrunner/bf/root/HEAD/pgsql.build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20231027T234552.867/log":
Directorynot empty
pg_upgrade: warning: could not remove directory
"C:/tools/nmsys64/home/pgrunner/bf/root/HEAD/pgsql.build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20231027T234552.867":
Directorynot empty
pg_upgrade: warning: could not stat file
"C:/tools/nmsys64/home/pgrunner/bf/root/HEAD/pgsql.build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20231027T234552.867/log/pg_upgrade_internal.log":
Nosuch file or directory
pg_upgrade: warning: could not remove directory
"C:/tools/nmsys64/home/pgrunner/bf/root/HEAD/pgsql.build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20231027T234552.867/log":
Directorynot empty
pg_upgrade: warning: could not remove directory
"C:/tools/nmsys64/home/pgrunner/bf/root/HEAD/pgsql.build/testrun/pg_upgrade/002_pg_upgrade/data/t_002_pg_upgrade_new_node_data/pgdata/pg_upgrade_output.d/20231027T234552.867":
Directorynot empty
[23:46:07.585](17.106s) ok 12 - run of pg_upgrade --check for new instance
[23:46:07.587](0.002s) not ok 13 - pg_upgrade_output.d/ removed after pg_upgrade --check success
...
```
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
