Re: Speed up build on Windows by generating symbol definition inbatch - Mailing list pgsql-hackers

From Noah Misch
Subject Re: Speed up build on Windows by generating symbol definition inbatch
Date
Msg-id 20190407063110.GA1977607@rfd.leadboat.com
Whole thread Raw
In response to Speed up build on Windows by generating symbol definition in batch  (Peifeng Qiu <pqiu@pivotal.io>)
Responses Re: Speed up build on Windows by generating symbol definition in batch
List pgsql-hackers
On Sat, Mar 30, 2019 at 03:42:39PM +0900, Peifeng Qiu wrote:
> When I watched the whole build process with a task manager, I discovered
> that a lot of time was spent on generating export symbol definitions,
> without consuming much CPU or IO.
> The script that doing this is src/tools/msvc/gendef.pl, it enumerates the
> whole directory for ".obj" files and call dumpbin utility to generate
> ".sym" files one by one like this:
> 
> dumpbin /symbols /out:a.sym a.obj >NUL
> 
> Actually the dumpbin utility accepts a wildcard file name, so we can
> generate the export symbols of all ".obj" files in batch.
> 
> dumpbin /symbols /out:all.sym *.obj >NUL
> 
> This will avoid wasting time by creating and destroying dumpbin process
> repeatedly and can speed up the build process considerably.
> I've tested on my 4-core 8-thread Intel i7 CPU. I've set MSBFLAGS=/m to
> ensure it can utilize all CPU cores.
> Building without this patch takes about 370 seconds. Building with this
> patch takes about 200 seconds. That's almost 2x speed up.

I, too, get a strong improvement, from 201s to 149s.  I can confirm it yields
identical *.def files.  Thanks for identifying this improvement.

> -    my ($objfile, $symfile) = @_;
> -    my ($symvol, $symdirs, $symbase) = splitpath($symfile);
> -    my $tmpfile = catpath($symvol, $symdirs, "symbols.out");

You removed the last use of File::Spec::Functions, so remove its "use"
statement.

> -    system("dumpbin /symbols /out:$tmpfile $_ >NUL")
> -      && die "Could not call dumpbin";

This error handling was crude, but don't replace it with zero error handling.

> -    rename($tmpfile, $symfile);

Keep the use of a temporary file, too.

> +system("dumpbin /symbols /out:$symfile $ARGV[0]/*obj >NUL");

That should be *.obj, not *obj.



pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: "could not reattach to shared memory" on buildfarm member dory
Next
From: Noah Misch
Date:
Subject: Re: Augment every test postgresql.conf