Thread: [buildfarm-members] BuildFarm client release 4.19
I have released version 4.19 of the PostgreSQL Buildfarm client. It can be downloaded from <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz> Apart from some minor bug fixes, the following changes are made: * Include the script's path in @INC. That means you can usually run the script from anywhere rather than just its own directory. * Set TZ after "make check" is run. this makes for slightly faster initdb runs in later stages. * make TAP tests run with --timer * change default log_line_prefix in config file to use %p instead of %c. That makes it easier to tie log messages to other messages that mention a pid * Add a module to log running commands to a file as it runs and replace critical uses of `` with the new procedure. That means we have better traces if the buildfarm process crashes. * Improvements to TAP tests logging and coverage. Each test set (i.e each /t directory) is now logged separately. Coverage is extended to remaining test sets in src/test except SSL tests. * Add a module for testing ICU * change target URL in config file sample to use canonical name matching SSL certificate * default build_root to a directory in the same place that the script is found, and create it if it doesn't exist. * add a command-line flag to allow setting config settings * from-source path can now be relative * from-source doesn't need to specify a branch, if not clear it defaults to HEAD * from-source can now do vpath builds These changes mean that the client is more useful for testing development code, and also that testing config settings is much simpler. An initial test run on a fresh installation is now as simple as: cp buildfarm.conf.sample build-farm.conf ./run_build.pl --test --verbose To test development code, the from-source option is now much more flexible and friendly. For example, one might do something like: ./run_build.pl --from-source=/path/to/postgresql \ --config-set use_vpath=1 \ --config-set config_opts+=--enable-tap-tests \ --config-set locales+=en_US.utf8 If you run something like this against your development code, and it succeeds, you can be reasonably confident that if committed it won't sent the buildfarm all red. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: > I have released version 4.19 of the PostgreSQL Buildfarm client. It can > be downloaded from > <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz> Nice work! > * Improvements to TAP tests logging and coverage. Each test set (i.e > each /t directory) is now logged separately. Coverage is extended to > remaining test sets in src/test except SSL tests. I was going to plead with buildfarm owners who use --enable-tap-tests to update sooner rather than later, because this separate-logging behavior is going to make it a lot less painful to decipher failures during the TAP tests. However, in view of my results in https://www.postgresql.org/message-id/21358.1492622881%40sss.pgh.pa.us I can't really recommend that owners of slower machines update just yet. Hopefully we can do something to trim down the added runtime. > These changes mean that the client is more useful for testing > development code, and also that testing config settings is much simpler. > An initial test run on a fresh installation is now as simple as: > ... > To test development code, the from-source option is now much more > flexible and friendly. For example, one might do something like: These things are pretty cool, but maybe they need to be documented at https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto ? regards, tom lane
On 04/19/2017 01:38 PM, Tom Lane wrote: > Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes: >> I have released version 4.19 of the PostgreSQL Buildfarm client. It can >> be downloaded from >> <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz> > Nice work! > Thank you. >> * Improvements to TAP tests logging and coverage. Each test set (i.e >> each /t directory) is now logged separately. Coverage is extended to >> remaining test sets in src/test except SSL tests. > I was going to plead with buildfarm owners who use --enable-tap-tests to > update sooner rather than later, because this separate-logging behavior > is going to make it a lot less painful to decipher failures during the > TAP tests. However, in view of my results in > https://www.postgresql.org/message-id/21358.1492622881%40sss.pgh.pa.us > I can't really recommend that owners of slower machines update just yet. > Hopefully we can do something to trim down the added runtime. As I mentioned in reply to that thread, you can disable the extra TAP tests with this addition to the command line: --skip-steps=misc-check > >> These changes mean that the client is more useful for testing >> development code, and also that testing config settings is much simpler. >> An initial test run on a fresh installation is now as simple as: >> ... >> To test development code, the from-source option is now much more >> flexible and friendly. For example, one might do something like: > These things are pretty cool, but maybe they need to be documented at > https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto > ? > > Indeed. I am planning to. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2017-04-19 15:59, Andrew Dunstan wrote: > I have released version 4.19 of the PostgreSQL Buildfarm client. It can > be downloaded from > <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz> I don't know if it's only me or if others have noticed this also but I have the buildfarm client set up like this: /home/pgbf /home/pgbf/build-farm-4 /home/pgbf/buildfarm@ -> build-farm-4.19/ /home/pgbf/buildroot and then I have a cron job that does this: x y * * * cd /home/pgbf/buildfarm && ./run_build.pl REL9_6_STABLE and when it tries to run I get the following error: cannot create /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log: No such file or directory gmake is not GNU Make - please fix config file at ./run_build.pl line 343. if I manually create the directory loach.lastrun-logs the script then continues normally. I am no perl guru so I can't figure out really what is going on here. I think that the directory /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs needs to be created before the PGBuild/Utils.pm sub run_log tries to write to it. /Mikael
On 05/01/2017 02:13 PM, Mikael Kjellström wrote: > On 2017-04-19 15:59, Andrew Dunstan wrote: > > >> I have released version 4.19 of the PostgreSQL Buildfarm client. It can >> be downloaded from >> <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz> >> > > I don't know if it's only me or if others have noticed this also but I > have the buildfarm client set up like this: > > /home/pgbf > /home/pgbf/build-farm-4 > /home/pgbf/buildfarm@ -> build-farm-4.19/ > /home/pgbf/buildroot > > and then I have a cron job that does this: > > x y * * * cd /home/pgbf/buildfarm && ./run_build.pl REL9_6_STABLE > > and when it tries to run I get the following error: > > cannot create > /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log: > No such file or directory > gmake is not GNU Make - please fix config file at ./run_build.pl line > 343. > > if I manually create the directory loach.lastrun-logs the script then > continues normally. > > I am no perl guru so I can't figure out really what is going on here. > I think that the directory > /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs needs to be > created before the PGBuild/Utils.pm sub run_log tries to write to it. > I will check on it. Thanks for the report. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
=?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= <mikael.kjellstrom@mksoft.nu> writes:
> I don't know if it's only me or if others have noticed this also but I
> have the buildfarm client set up like this:
> ...
> and when it tries to run I get the following error:
> cannot create
> /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
> No such file or directory
> gmake is not GNU Make - please fix config file at ./run_build.pl line 343.
> if I manually create the directory loach.lastrun-logs the script then
> continues normally.
FWIW, I haven't seen that, but probably that's because the directory
already exists in each branch's working directory. I bet zapping the
whole working directory for a branch would reproduce it.
regards, tom lane
On 05/01/2017 02:13 PM, Mikael Kjellström wrote: > On 2017-04-19 15:59, Andrew Dunstan wrote: > > >> I have released version 4.19 of the PostgreSQL Buildfarm client. It can >> be downloaded from >> <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz> >> > > I don't know if it's only me or if others have noticed this also but I > have the buildfarm client set up like this: > > /home/pgbf > /home/pgbf/build-farm-4 > /home/pgbf/buildfarm@ -> build-farm-4.19/ > /home/pgbf/buildroot > > and then I have a cron job that does this: > > x y * * * cd /home/pgbf/buildfarm && ./run_build.pl REL9_6_STABLE > > and when it tries to run I get the following error: > > cannot create > /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log: > No such file or directory > gmake is not GNU Make - please fix config file at ./run_build.pl line > 343. > > if I manually create the directory loach.lastrun-logs the script then > continues normally. > > I am no perl guru so I can't figure out really what is going on here. > I think that the directory > /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs needs to be > created before the PGBuild/Utils.pm sub run_log tries to write to it. > > OK, that's a bug. Mea culpa. the quick fix is this patch: diff --git a/run_build.pl b/run_build.pl index aeb8966..822b4de 100755 --- a/run_build.pl +++ b/run_build.pl @@ -1008,7 +1008,8 @@ sub writelog sub check_make { - my @out = run_log("$make -v"); + # don't use run_log here - it's too early in the process + my @out = `$make -v 2>&1`; return undef unless ($? == 0 && grep {/GNU Make/} @out); return 'OK'; } cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2017-05-01 20:19, Tom Lane wrote: >> if I manually create the directory loach.lastrun-logs the script then >> continues normally. > > FWIW, I haven't seen that, but probably that's because the directory > already exists in each branch's working directory. I bet zapping the > whole working directory for a branch would reproduce it. Yes, zapping the whole working directory for each branch reproduces it for me every time. /Mikael
On 2017-05-01 20:25, Andrew Dunstan wrote:
> OK, that's a bug. Mea culpa.
>
> the quick fix is this patch:
>
>
> diff --git a/run_build.pl b/run_build.pl
> index aeb8966..822b4de 100755
> --- a/run_build.pl
> +++ b/run_build.pl
> @@ -1008,7 +1008,8 @@ sub writelog
>
> sub check_make
> {
> - my @out = run_log("$make -v");
> + # don't use run_log here - it's too early in the process
> + my @out = `$make -v 2>&1`;
> return undef unless ($? == 0 && grep {/GNU Make/} @out);
> return 'OK';
> }
Nope, that didn't do it.
If I zap REL9_6_STABLE and then run:
./run_build.pl REL9_6_STABLE
I get the following output:
cannot create
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
No such file or directory
cannot create
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
No such file or directory
cannot create
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
No such file or directory
Buildfarm member loach failed on REL9_6_STABLE stage pgsql-Git
/Mikael
On 2017-05-01 20:44, Mikael Kjellström wrote: > Nope, that didn't do it. Or well. It fixed the check_make bug but not the other bug with that the loach.lastrun-logs-directory isn't created before trying to write to it. /Mikael
On 05/01/2017 02:46 PM, Mikael Kjellström wrote: > > > On 2017-05-01 20:44, Mikael Kjellström wrote: >> Nope, that didn't do it. > > Or well. It fixed the check_make bug but not the other bug with that > the loach.lastrun-logs-directory isn't created before trying to write > to it. > OK, coming up with a more comprehensive fix. The obvious workaround for now is to create the directory and dont zap it or its parents. You should only have to do it once (per branch) cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2017-05-01 20:56, Andrew Dunstan wrote: > OK, coming up with a more comprehensive fix. Ok. > The obvious workaround for now is to create the directory and dont zap > it or its parents. You should only have to do it once (per branch) Yes, I know. That is what I have been doing so far. But if you want to rerun a branch from scratch it's much easier to just do: rm -rf buildroot/HEAD or rm -rf buildroot/REL9_6_STABLE and then the buildfarm script should do the right thing and create all the directories that it needs. /Mikael
On 05/01/2017 03:00 PM, Mikael Kjellström wrote:
>
> On 2017-05-01 20:56, Andrew Dunstan wrote:
>> OK, coming up with a more comprehensive fix.
>
> Ok.
>
>> The obvious workaround for now is to create the directory and dont zap
>> it or its parents. You should only have to do it once (per branch)
>
> Yes, I know. That is what I have been doing so far. But if you want
> to rerun a branch from scratch it's much easier to just do:
>
> rm -rf buildroot/HEAD
> or
> rm -rf buildroot/REL9_6_STABLE
>
> and then the buildfarm script should do the right thing and create all
> the directories that it needs.
>
Not sure I understand what "rerun a branch" from scratch means. If you
zap the branch directory you lose all its state. That's generally a bad
thing.
Anyway, this patch should fix it for all uses. It creates the directory
if it doesn't exist.
diff --git a/PGBuild/Utils.pm b/PGBuild/Utils.pm
index 91c1362..175eaa7 100644
--- a/PGBuild/Utils.pm
+++ b/PGBuild/Utils.pm
@@ -14,6 +14,8 @@ See accompanying License file for license details
use strict;
use warnings;
+use File::Path;
+
use Exporter ();
our (@ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
@@ -30,8 +32,9 @@ use vars qw($VERSION); $VERSION = 'REL_4.19';
sub run_log
{
my $command = shift;
- my $file=
-
"$main::branch_root/$main::st_prefix$main::logdirname/lastcomand.log";
+ my $filedir =
"$main::branch_root/$main::st_prefix$main::logdirname";
+ mkpath($filedir);
+ my $file= "$filedir/lastcomand.log";
unlink $file;
system("$command > $file 2>&1");
my @loglines;
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2017-05-01 21:10, Andrew Dunstan wrote:
> Not sure I understand what "rerun a branch" from scratch means. If you
> zap the branch directory you lose all its state. That's generally a bad
> thing.
I mean like the first time when you set up the buildfarm client /
branch. And it's not something I recomend doing all the time but some
times when the buildfarm client doesn't clean up after it self and there
are a lot of crap left it's "easier" to just zap everything and start
over to get it running again.
> Anyway, this patch should fix it for all uses. It creates the directory
> if it doesn't exist.
>
> diff --git a/PGBuild/Utils.pm b/PGBuild/Utils.pm
> index 91c1362..175eaa7 100644
> --- a/PGBuild/Utils.pm
> +++ b/PGBuild/Utils.pm
> @@ -14,6 +14,8 @@ See accompanying License file for license details
> use strict;
> use warnings;
>
> +use File::Path;
> +
> use Exporter ();
> our (@ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
>
> @@ -30,8 +32,9 @@ use vars qw($VERSION); $VERSION = 'REL_4.19';
> sub run_log
> {
> my $command = shift;
> - my $file=
> -
> "$main::branch_root/$main::st_prefix$main::logdirname/lastcomand.log";
> + my $filedir =
> "$main::branch_root/$main::st_prefix$main::logdirname";
> + mkpath($filedir);
> + my $file= "$filedir/lastcomand.log";
> unlink $file;
> system("$command > $file 2>&1");
> my @loglines;
Ok.
Thanks. Will try it out.
/Mikael
On 2017-05-01 21:19, Mikael Kjellström wrote: > Thanks. Will try it out. Just wanted to report that I've tried it and it works as expected. Thanks for the really fast fixes. /Mikael