Thread: Speed up build on Windows by generating symbol definition in batch
Hi, hackers.
Build process on Windows includes compiling all source into object files, linking them to binaries, and generating export symbol definitions, etc.
When I watched the whole build process with a task manager, I discovered that a lot of time was spent on generating export symbol definitions, without consuming much CPU or IO.
The script that doing this is src/tools/msvc/gendef.pl, it enumerates the whole directory for ".obj" files and call dumpbin utility to generate ".sym" files one by one like this:
dumpbin /symbols /out:a.sym a.obj >NUL
Actually the dumpbin utility accepts a wildcard file name, so we can generate the export symbols of all ".obj" files in batch.
dumpbin /symbols /out:all.sym *.obj >NUL
This will avoid wasting time by creating and destroying dumpbin process repeatedly and can speed up the build process considerably.
I've tested on my 4-core 8-thread Intel i7 CPU. I've set MSBFLAGS=/m to ensure it can utilize all CPU cores.
Building without this patch takes about 370 seconds. Building with this patch takes about 200 seconds. That's almost 2x speed up.
Best regards,
Peifeng Qiu
Attachment
On Sat, Mar 30, 2019 at 03:42:39PM +0900, Peifeng Qiu wrote:
> When I watched the whole build process with a task manager, I discovered
> that a lot of time was spent on generating export symbol definitions,
> without consuming much CPU or IO.
> The script that doing this is src/tools/msvc/gendef.pl, it enumerates the
> whole directory for ".obj" files and call dumpbin utility to generate
> ".sym" files one by one like this:
>
> dumpbin /symbols /out:a.sym a.obj >NUL
>
> Actually the dumpbin utility accepts a wildcard file name, so we can
> generate the export symbols of all ".obj" files in batch.
>
> dumpbin /symbols /out:all.sym *.obj >NUL
>
> This will avoid wasting time by creating and destroying dumpbin process
> repeatedly and can speed up the build process considerably.
> I've tested on my 4-core 8-thread Intel i7 CPU. I've set MSBFLAGS=/m to
> ensure it can utilize all CPU cores.
> Building without this patch takes about 370 seconds. Building with this
> patch takes about 200 seconds. That's almost 2x speed up.
I, too, get a strong improvement, from 201s to 149s. I can confirm it yields
identical *.def files. Thanks for identifying this improvement.
> - my ($objfile, $symfile) = @_;
> - my ($symvol, $symdirs, $symbase) = splitpath($symfile);
> - my $tmpfile = catpath($symvol, $symdirs, "symbols.out");
You removed the last use of File::Spec::Functions, so remove its "use"
statement.
> - system("dumpbin /symbols /out:$tmpfile $_ >NUL")
> - && die "Could not call dumpbin";
This error handling was crude, but don't replace it with zero error handling.
> - rename($tmpfile, $symfile);
Keep the use of a temporary file, too.
> +system("dumpbin /symbols /out:$symfile $ARGV[0]/*obj >NUL");
That should be *.obj, not *obj.
Thanks for reviewing!
I've updated the patch according to your comments.
Best regards,
Peifeng Qiu
On Sun, Apr 7, 2019 at 2:31 PM Noah Misch <noah@leadboat.com> wrote:
On Sat, Mar 30, 2019 at 03:42:39PM +0900, Peifeng Qiu wrote:
> When I watched the whole build process with a task manager, I discovered
> that a lot of time was spent on generating export symbol definitions,
> without consuming much CPU or IO.
> The script that doing this is src/tools/msvc/gendef.pl, it enumerates the
> whole directory for ".obj" files and call dumpbin utility to generate
> ".sym" files one by one like this:
>
> dumpbin /symbols /out:a.sym a.obj >NUL
>
> Actually the dumpbin utility accepts a wildcard file name, so we can
> generate the export symbols of all ".obj" files in batch.
>
> dumpbin /symbols /out:all.sym *.obj >NUL
>
> This will avoid wasting time by creating and destroying dumpbin process
> repeatedly and can speed up the build process considerably.
> I've tested on my 4-core 8-thread Intel i7 CPU. I've set MSBFLAGS=/m to
> ensure it can utilize all CPU cores.
> Building without this patch takes about 370 seconds. Building with this
> patch takes about 200 seconds. That's almost 2x speed up.
I, too, get a strong improvement, from 201s to 149s. I can confirm it yields
identical *.def files. Thanks for identifying this improvement.
> - my ($objfile, $symfile) = @_;
> - my ($symvol, $symdirs, $symbase) = splitpath($symfile);
> - my $tmpfile = catpath($symvol, $symdirs, "symbols.out");
You removed the last use of File::Spec::Functions, so remove its "use"
statement.
> - system("dumpbin /symbols /out:$tmpfile $_ >NUL")
> - && die "Could not call dumpbin";
This error handling was crude, but don't replace it with zero error handling.
> - rename($tmpfile, $symfile);
Keep the use of a temporary file, too.
> +system("dumpbin /symbols /out:$symfile $ARGV[0]/*obj >NUL");
That should be *.obj, not *obj.
Attachment
On Wed, Apr 10, 2019 at 02:27:26PM +0800, Peifeng Qiu wrote: > I've updated the patch according to your comments. Looks good. Thanks. I plan to push this on Saturday.
On Mon, 29 Apr 2019 at 14:50, Noah Misch <noah@leadboat.com> wrote: > > On Wed, Apr 10, 2019 at 02:27:26PM +0800, Peifeng Qiu wrote: > > I've updated the patch according to your comments. > > Looks good. Thanks. I plan to push this on Saturday. I didn't really look at the patch in detail, but on testing it on a windows machine with vs2017, it took built time from 5:00 to 4:10. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services