Thread: MSVC build system
I want to overhaul the MSVC build system somewhat and want to discuss my plans. The first thing I want to do is get rid of at least most of the .bat files in it altogether and make a single coherent perl-based system. The reasons are: . the XP_EXIT_FIX hack we built in to cater for the XP command processor doesn't actually work on XP - if it's turned on it fails too often (e.g. on build warnings) and if it's turned off if doesn't fail often enough (e.g. on build errors). . it's unnecessarily complex. For example, the vcregress.bat file calls a perl script to create a temporary .bat file which it then calls. . we need perl anyway, so why not write it all in perl? I venture to suggest that the majority of our developers are more at home in perl than in the arcane syntax of the Windows command processor, so we might be less reliant on a tiny group of people (Magnus and Dave, basically) to maintain the build system.Writing it all in perl would give us a chance to make the whole system more coherent and data driven. . the .bat files have serious (from my POV) limitations / errors anyway. e.g. vcregress.bat has a hardcoded temp port (a definite nono for a machine running buildfarm and a departure from the regression make file) and also does not at all honor the MAX_CONNECTIONS environment setting. It also hardcodes the use of the parallel schedule for installcheck, unlike the same named test on Unix. In the longer run I want to make the whole system more data driven, so that it's comparatively easy for someone to add stuff. The whole thing needs proper documentation for developers too. Right now, discovering where to add a new installation file, for example, requires you to dig into the code which can be rather bewildering, I suspect. Anyway, the first part of what I want to achieve is to replace the build.bat and vcregress.bat files with perl equivalents, with the errors/omissions fixed. Along the way, getregress.pl would become redundant. Apart from fixing the issue with using the systems "dir" command rather than using File::Find, which I will revisit, I think that's all I would do now, given how close we are to Beta. The rest can wait until after we release. I should add that some at least of this is my fault - Magnus showed me at least parts of the system and asked my advice, and I didn't pay close enough attention. It was only when (much later) I started to try to use it seriously that I understood how much work was still needed. Thoughts? cheers andrew
On Mon, 2007-08-27 at 11:00 -0400, Andrew Dunstan wrote: > In the longer run I want to make the whole system more data driven, so > that it's comparatively easy for someone to add stuff. I don't mean to hijack your thread, but I wonder if maintaining two separate build systems is the best approach in the long term. I think CMake[1] is an interesting alternative: it would allow us to generate both makefiles and MSVC .proj's from a single set of master build files. -Neil [1] http://www.cmake.org
Neil Conway wrote: > On Mon, 2007-08-27 at 11:00 -0400, Andrew Dunstan wrote: > >> In the longer run I want to make the whole system more data driven, so >> that it's comparatively easy for someone to add stuff. >> > > I don't mean to hijack your thread, but I wonder if maintaining two > separate build systems is the best approach in the long term. I think > CMake[1] is an interesting alternative: it would allow us to generate > both makefiles and MSVC .proj's from a single set of master build files. > > -Neil > > [1] http://www.cmake.org > > I would be more than happy if we had a single build system. Maybe some enterprising person would like to try to create such a system as a proof of concept. I count around 200 makefiles in our sources ;-) Of course, we'd need to know that cmake was pretty universally available too. cheers andrew
Andrew Dunstan wrote: > Neil Conway wrote: >> I don't mean to hijack your thread, but I wonder if maintaining two >> separate build systems is the best approach in the long term. I think >> CMake[1] is an interesting alternative: it would allow us to generate >> both makefiles and MSVC .proj's from a single set of master build files. > > I would be more than happy if we had a single build system. Maybe some > enterprising person would like to try to create such a system as a proof of > concept. I count around 200 makefiles in our sources ;-) > > Of course, we'd need to know that cmake was pretty universally available > too. That, or we create the makefiles in a fixed system and keep the Makefiles in CVS (though would be derived files). -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
Neil Conway wrote: > On Mon, 2007-08-27 at 11:00 -0400, Andrew Dunstan wrote: > >> In the longer run I want to make the whole system more data driven, so >> that it's comparatively easy for someone to add stuff. >> > > I don't mean to hijack your thread, but I wonder if maintaining two > separate build systems is the best approach in the long term. I think > CMake[1] is an interesting alternative: it would allow us to generate > both makefiles and MSVC .proj's from a single set of master build files. > To add my 2d worth to this: after working on a few very large projects that built on both Unix and Windows my preference is to use a single autotools-based build for both, with a script called cccl that translates cc-style arguments for Microsoft's cl compiler/linker tool chain (plus Cygwin for the command line utilities, gmake etc). We have a locally-enhanced version of cccl that's a bit more capable than the latest public version, I seem to remember. I've used cmake but don't particularly like it because most everyone has settled on autotools (for better or worse) as the way to build big C/C++ projects.
Alvaro Herrera wrote: > Andrew Dunstan wrote: > >> Neil Conway wrote: > >>> I don't mean to hijack your thread, but I wonder if maintaining two >>> separate build systems is the best approach in the long term. I think >>> CMake[1] is an interesting alternative: it would allow us to generate >>> both makefiles and MSVC .proj's from a single set of master build files. >> I would be more than happy if we had a single build system. Maybe some >> enterprising person would like to try to create such a system as a proof of >> concept. I count around 200 makefiles in our sources ;-) >> >> Of course, we'd need to know that cmake was pretty universally available >> too. > > That, or we create the makefiles in a fixed system and keep the > Makefiles in CVS (though would be derived files). IIRC, we previously looked into cmake and concluded it supported a lot fewer platforms than pgsql. However, if we can go by Alvaros suggestion and keep the makefiles as derived files, that could certainly work... //Magnus
David Boreham wrote: > Neil Conway wrote: >> On Mon, 2007-08-27 at 11:00 -0400, Andrew Dunstan wrote: >> >>> In the longer run I want to make the whole system more data driven, >>> so that it's comparatively easy for someone to add stuff. >>> >> >> I don't mean to hijack your thread, but I wonder if maintaining two >> separate build systems is the best approach in the long term. I think >> CMake[1] is an interesting alternative: it would allow us to generate >> both makefiles and MSVC .proj's from a single set of master build files. >> > To add my 2d worth to this: after working on a few very large > projects that built on both Unix and Windows my preference is > to use a single autotools-based build for both, with a script called cccl > that translates cc-style arguments for Microsoft's cl compiler/linker > tool chain (plus Cygwin for the command line utilities, gmake etc). > We have a locally-enhanced version of cccl that's a bit > more capable than the latest public version, I seem to remember. But that still requires you to have a full set of "unix style commandline tools" on your windows box in order to build, no? And if it doesn't generate project files and such, it won't be usable in Visual Studio, just the commandline compiler... //Magnus
Magnus Hagander wrote: > IIRC, we previously looked into cmake and concluded it supported a lot > fewer platforms than pgsql. > > However, if we can go by Alvaros suggestion and keep the makefiles as > derived files, that could certainly work... > > > But everyone would still need to learn it. Maybe we should think about this and look again after release. I still think that for now the fairly low impact changes I suggested would be best. cheers andrew
Andrew Dunstan wrote: > > > Magnus Hagander wrote: >> IIRC, we previously looked into cmake and concluded it supported a lot >> fewer platforms than pgsql. >> >> However, if we can go by Alvaros suggestion and keep the makefiles as >> derived files, that could certainly work... >> >> >> > > But everyone would still need to learn it. > > Maybe we should think about this and look again after release. I still > think that for now the fairly low impact changes I suggested would be best. Agreed. //Magnus
Magnus Hagander <magnus@hagander.net> writes:
> Alvaro Herrera wrote:
>> That, or we create the makefiles in a fixed system and keep the
>> Makefiles in CVS (though would be derived files).
> IIRC, we previously looked into cmake and concluded it supported a lot
> fewer platforms than pgsql.
> However, if we can go by Alvaros suggestion and keep the makefiles as
> derived files, that could certainly work...
Not really, as it still disenfranchises developers who don't have or
know how to use cmake (or whatever tool you select). This is not like
bison or flex, which you can avoid learning and still be able to work on
many interesting parts of Postgres. If you can't work with the build
system then you can't even add a new source file, and that's a pretty
crippling restriction.
The situation as we have it right now is not satisfactory on this
point either. The lingua franca is gmake, which is fine for all the
systems we've historically supported, but it disenfranchises would-be
developers who use MSVC; they likely don't know gmake or even have it
installed.
I've never worked with cmake, but the info on their home page sounds
like it would work on all the systems we are interested in. I think
the $64 question is whether we can make it sit up and do all the tricks
that are in our Makefiles now. In any case, the conversion cost would
be pretty darn sizable --- not only the effort from a few people to do
the initial conversion, but the distributed costs of all developers
having to learn cmake. I'm not sure we want to go there ... not yet
anyway ...
regards, tom lane
Tom Lane wrote: > Magnus Hagander <magnus@hagander.net> writes: >> Alvaro Herrera wrote: >>> That, or we create the makefiles in a fixed system and keep the >>> Makefiles in CVS (though would be derived files). > >> IIRC, we previously looked into cmake and concluded it supported a lot >> fewer platforms than pgsql. > >> However, if we can go by Alvaros suggestion and keep the makefiles as >> derived files, that could certainly work... > > Not really, as it still disenfranchises developers who don't have or > know how to use cmake (or whatever tool you select). This is not like > bison or flex, which you can avoid learning and still be able to work on > many interesting parts of Postgres. If you can't work with the build > system then you can't even add a new source file, and that's a pretty > crippling restriction. It would require developer education, absoulutely, but it would fix the cross platform problem, which is what I was referring to. > I've never worked with cmake, but the info on their home page sounds > like it would work on all the systems we are interested in. It does look pretty good on that - probably they've fixed the ones that were missing when it was discussed before. Or my memory just sucks, that's also a clear possibility. > I think > the $64 question is whether we can make it sit up and do all the tricks > that are in our Makefiles now. Having never used it, I can't comment on that. > In any case, the conversion cost would > be pretty darn sizable --- not only the effort from a few people to do > the initial conversion, but the distributed costs of all developers > having to learn cmake. I'm not sure we want to go there ... not yet > anyway ... Oh yes, it's a huge change. I think we should go with the current method a bit longer to see how it holds up before making such a decision. //Magnus
Magnus Hagander wrote: > David Boreham wrote: > >> To add my 2d worth to this: after working on a few very large >> projects that built on both Unix and Windows my preference is >> to use a single autotools-based build for both, with a script called cccl >> that translates cc-style arguments for Microsoft's cl compiler/linker >> tool chain (plus Cygwin for the command line utilities, gmake etc). >> We have a locally-enhanced version of cccl that's a bit >> more capable than the latest public version, I seem to remember. >> > > But that still requires you to have a full set of "unix style > commandline tools" on your windows box in order to build, no? And if it > doesn't generate project files and such, it won't be usable in Visual > Studio, just the commandline compiler... > Correct on both counts, and for me neither is a problem. I can't imagine using a Windows box for software development without Unix tools installed, and I've never found building huge pieces of software using an IDE to be a useful thing to do. As I said, just my 2d worth...
Andrew Dunstan wrote:
>
> I want to overhaul the MSVC build system somewhat and want to discuss
> my plans.
>
> [snip]. Apart from fixing the issue with using the systems "dir"
> command rather than using File::Find, which I will revisit, I think
> that's all I would do now, given how close we are to Beta. [snip]
Hi,
please find attached the patch I submitted for the "systems 'dir'"-issue
a couple of month ago. The patch is against current CVS head (as of
2007-09-14). You might find it useful.
hth,
-Hannes
*** ../pgsql-cvshead/src/tools/msvc/Install.pm Fri Sep 14 20:56:16 2007
--- src/tools/msvc/Install.pm Fri Sep 14 21:30:09 2007
***************
*** 10,15 ****
--- 10,18 ----
use Carp;
use File::Basename;
use File::Copy;
+ use File::Find;
+ use File::Glob;
+ use File::Spec;
use Exporter;
our (@ISA,@EXPORT_OK);
***************
*** 103,108 ****
--- 106,146 ----
print "\n";
}
+ sub FindFiles
+ {
+ my $spec = shift;
+ my $nonrecursive = shift;
+ my $pat = basename($spec);
+ my $dir = dirname($spec);
+
+ if ($dir eq '') { $dir = '.'; }
+
+ -d $dir || croak "Could not list directory $dir: $!\n";
+
+ if ($nonrecursive)
+ {
+ return glob($spec);
+ }
+
+ # borrowed from File::DosGlob
+ # escape regex metachars but not glob chars
+ $pat =~ s:([].+^\-\${}[|]):\\$1:g;
+ # and convert DOS-style wildcards to regex
+ $pat =~ s/\*/.*/g;
+ $pat =~ s/\?/.?/g;
+
+ $pat = '^' . $pat . '\z';
+
+ my @res;
+ find(
+ {
+ wanted => sub { /$pat/s && push (@res, File::Spec->canonpath($File::Find::name)); }
+ },
+ $dir
+ );
+ return @res;
+ }
+
sub CopySetOfFiles
{
my $what = shift;
***************
*** 110,131 ****
my $target = shift;
my $silent = shift;
my $norecurse = shift;
- my $D;
- my $subdirs = $norecurse?'':'/s';
print "Copying $what" unless ($silent);
! open($D, "dir /b $subdirs $spec |") || croak "Could not list $spec\n";
! while (<$D>)
{
- chomp;
next if /regress/; # Skip temporary install in regression subdir
next if /ecpg.test/; # Skip temporary install in regression subdir
! my $tgt = $target . basename($_);
print ".";
! my $src = $norecurse?(dirname($spec) . '/' . $_):$_;
! copy($src, $tgt) || croak "Could not copy $src: $!\n";
}
! close($D);
print "\n";
}
--- 148,166 ----
my $target = shift;
my $silent = shift;
my $norecurse = shift;
print "Copying $what" unless ($silent);
!
! foreach (FindFiles($spec, $norecurse))
{
next if /regress/; # Skip temporary install in regression subdir
next if /ecpg.test/; # Skip temporary install in regression subdir
! my $src = $_;
! my $tgt = $target . basename($src);
print ".";
! copy($src, $tgt) || croak "Could not copy $src to $tgt: $!\n";
}
!
print "\n";
}
***************
*** 415,439 ****
{
my $target = shift;
my $nlspath = shift;
- my $D;
print "Installing NLS files...";
EnsureDirectories($target, "share/locale");
! open($D,"dir /b /s nls.mk|") || croak "Could not list nls.mk\n";
! while (<$D>)
{
- chomp;
s/nls.mk/po/;
my $dir = $_;
next unless ($dir =~ /([^\\]+)\\po$/);
my $prgm = $1;
$prgm = 'postgres' if ($prgm eq 'backend');
- my $E;
- open($E,"dir /b $dir\\*.po|") || croak "Could not list contents of $_\n";
! while (<$E>)
{
- chomp;
my $lang;
next unless /^(.*)\.po/;
$lang = $1;
--- 450,469 ----
{
my $target = shift;
my $nlspath = shift;
print "Installing NLS files...";
EnsureDirectories($target, "share/locale");
!
! foreach (FindFiles("nls.mk"))
{
s/nls.mk/po/;
my $dir = $_;
next unless ($dir =~ /([^\\]+)\\po$/);
my $prgm = $1;
$prgm = 'postgres' if ($prgm eq 'backend');
! foreach (FindFiles("$dir\\*.po", 1))
{
my $lang;
next unless /^(.*)\.po/;
$lang = $1;
***************
*** 445,453 ****
&& croak("Could not run msgfmt on $dir\\$_");
print ".";
}
- close($E);
}
! close($D);
print "\n";
}
--- 475,482 ----
&& croak("Could not run msgfmt on $dir\\$_");
print ".";
}
}
!
print "\n";
}
Added to TODO for Win32:
o Convert MSVC build system to remove most batch files
http://archives.postgresql.org/pgsql-hackers/2007-08/msg00961.php
---------------------------------------------------------------------------
Andrew Dunstan wrote:
>
> I want to overhaul the MSVC build system somewhat and want to discuss my
> plans.
>
> The first thing I want to do is get rid of at least most of the .bat
> files in it altogether and make a single coherent perl-based system. The
> reasons are:
>
> . the XP_EXIT_FIX hack we built in to cater for the XP command processor
> doesn't actually work on XP - if it's turned on it fails too often (e.g.
> on build warnings) and if it's turned off if doesn't fail often enough
> (e.g. on build errors).
> . it's unnecessarily complex. For example, the vcregress.bat file calls
> a perl script to create a temporary .bat file which it then calls.
> . we need perl anyway, so why not write it all in perl? I venture to
> suggest that the majority of our developers are more at home in perl
> than in the arcane syntax of the Windows command processor, so we might
> be less reliant on a tiny group of people (Magnus and Dave, basically)
> to maintain the build system.Writing it all in perl would give us a
> chance to make the whole system more coherent and data driven.
> . the .bat files have serious (from my POV) limitations / errors anyway.
> e.g. vcregress.bat has a hardcoded temp port (a definite nono for a
> machine running buildfarm and a departure from the regression make file)
> and also does not at all honor the MAX_CONNECTIONS environment setting.
> It also hardcodes the use of the parallel schedule for installcheck,
> unlike the same named test on Unix.
>
> In the longer run I want to make the whole system more data driven, so
> that it's comparatively easy for someone to add stuff.
>
> The whole thing needs proper documentation for developers too. Right
> now, discovering where to add a new installation file, for example,
> requires you to dig into the code which can be rather bewildering, I
> suspect.
>
> Anyway, the first part of what I want to achieve is to replace the
> build.bat and vcregress.bat files with perl equivalents, with the
> errors/omissions fixed. Along the way, getregress.pl would become
> redundant. Apart from fixing the issue with using the systems "dir"
> command rather than using File::Find, which I will revisit, I think
> that's all I would do now, given how close we are to Beta. The rest can
> wait until after we release.
>
> I should add that some at least of this is my fault - Magnus showed me
> at least parts of the system and asked my advice, and I didn't pay close
> enough attention. It was only when (much later) I started to try to use
> it seriously that I understood how much work was still needed.
>
> Thoughts?
>
> cheers
>
> andrew
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
> choose an index scan if your joining column's datatypes do not
> match
-- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB
http://postgres.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +