Thread: Re: [COMMITTERS] pgsql: Append major version number and for libraries soname major

On Tuesday 16 December 2008 16:53:26 Tom Lane wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > Dave Page wrote:
> >> Any eta on a fix for this? My internal builds are failing as well as
> >> red_bat. (and yes, the other 2 MSVC buildfarm members are currently
> >> waiting for Dell to get hold of a new motherboard for their box).
> >
> > I think the maintenance of the MSVC build system is the job of the,
> > well, maintainer of the MSVC build system, whoever that may be.  In
> > other words, I have no ETA for you from me.  I'd be glad, however, to
> > provide information to the maintainers.
>
> I do not think this is an appropriate attitude for a committer to take.
> You are responsible for what you commit.  If you don't have the
> knowledge to fix something, or the resources to test it, okay ... but
> you then need to be proactive about getting someone else to fix/test it.
> Or else revert the broken patch.

I would like to have this clarified, as I keep running afoul of it.

We originally did the Windows port using the MinGW build system because we did 
not want to maintain another build system.  When people starting on the MSVC 
build system, I was given assurances that they would maintain it and we would 
not have to worry about it.  I specifically recall discussions about that in 
Toronto in 2006.

Now who are those maintainers and what are they doing about it?

I am not hostile against the MSVC system, even though I have no interest in it 
(and I am unfamiliar with the payoff).  I have done a lot of portability work 
for crazy platforms that I have no stake in.  But those platforms and build 
systems used standard tools, had publically available documentation, and in 
the extreme cases had a box that one could log into to verify the work.  None 
of this is available here.  I don't think the progress of this project can 
depend on proprietary tools and systems in a few hands.

Help?!?

Peter Eisentraut <peter_e@gmx.net> writes:
> On Tuesday 16 December 2008 16:53:26 Tom Lane wrote:
>> I do not think this is an appropriate attitude for a committer to take.

> I would like to have this clarified, as I keep running afoul of it.

My two cents: I don't expect you to fix the MSVC scripts if you are
uninterested in doing so.  But it would be appropriate to post a note
to -hackers when you make a build system change that is going to need
to be reflected in those scripts.  The people who do care about MSVC
should not have to find that out by watching the buildfarm and then
trying to reverse-engineer the cause of a failure.
        regards, tom lane

Tom Lane wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
>> On Tuesday 16 December 2008 16:53:26 Tom Lane wrote:
>>> I do not think this is an appropriate attitude for a committer to take.
> 
>> I would like to have this clarified, as I keep running afoul of it.
> 
> My two cents: I don't expect you to fix the MSVC scripts if you are
> uninterested in doing so.  But it would be appropriate to post a note
> to -hackers when you make a build system change that is going to need
> to be reflected in those scripts.  The people who do care about MSVC
> should not have to find that out by watching the buildfarm and then
> trying to reverse-engineer the cause of a failure.

As one of said people, I think that's perfectly reasonable. It'd make it
a lot easier to get a heads-up, or just the version of the patch right
before a commit to add it to. Some people do that, some don't - it would
be muchos helpful if all did.

I don't expect unix hackers to always patch the msvc system as needed -
given that you don't have a way to test it, and no experience how those
tools work. (I know Tom has made a number of smaller fixes in them, but
it's mostly been around the areas of smaller changes, not actually
adding new stuff, for exmaple)

If I had warning this time, for example, I could've said that I for one
can't fix that for a while, because my msvc build system is all geared
up around the git server (because it's so much easier to get the stuff
in and out of the VM when cross-deveoping than with CVS and the crippled
commandline toolchain on windows). And since that one is currently not
working properly, I need to find a different way to get the code in
there unless someone can fix it :-(

Hopefully someone else can pick it up this time around? If not, I'll do
it as soon as I get things back up and working in my env.

//Magnus

Exactly what type of changes affec the MSVC build system?

---------------------------------------------------------------------------

Magnus Hagander wrote:
> Tom Lane wrote:
> > Peter Eisentraut <peter_e@gmx.net> writes:
> >> On Tuesday 16 December 2008 16:53:26 Tom Lane wrote:
> >>> I do not think this is an appropriate attitude for a committer to take.
> > 
> >> I would like to have this clarified, as I keep running afoul of it.
> > 
> > My two cents: I don't expect you to fix the MSVC scripts if you are
> > uninterested in doing so.  But it would be appropriate to post a note
> > to -hackers when you make a build system change that is going to need
> > to be reflected in those scripts.  The people who do care about MSVC
> > should not have to find that out by watching the buildfarm and then
> > trying to reverse-engineer the cause of a failure.
> 
> As one of said people, I think that's perfectly reasonable. It'd make it
> a lot easier to get a heads-up, or just the version of the patch right
> before a commit to add it to. Some people do that, some don't - it would
> be muchos helpful if all did.
> 
> I don't expect unix hackers to always patch the msvc system as needed -
> given that you don't have a way to test it, and no experience how those
> tools work. (I know Tom has made a number of smaller fixes in them, but
> it's mostly been around the areas of smaller changes, not actually
> adding new stuff, for exmaple)
> 
> If I had warning this time, for example, I could've said that I for one
> can't fix that for a while, because my msvc build system is all geared
> up around the git server (because it's so much easier to get the stuff
> in and out of the VM when cross-deveoping than with CVS and the crippled
> commandline toolchain on windows). And since that one is currently not
> working properly, I need to find a different way to get the code in
> there unless someone can fix it :-(
> 
> Hopefully someone else can pick it up this time around? If not, I'll do
> it as soon as I get things back up and working in my env.
> 
> //Magnus
> 
> 
> -- 
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Anything the system doesn't pick up automatically. That means most new
build targets (but it will pick up a contrib automatically, and a
conversion proc, for example). Also if modifications are made to the
scripts that run (like gen_fmgr.sh).

Just adding new files to exisitng makefiles, or adding a new subdir that
adds more files to an existing target, should require no changes.

//Magnus

Bruce Momjian wrote:
> Exactly what type of changes affec the MSVC build system?
> 
> ---------------------------------------------------------------------------
> 
> Magnus Hagander wrote:
>> Tom Lane wrote:
>>> Peter Eisentraut <peter_e@gmx.net> writes:
>>>> On Tuesday 16 December 2008 16:53:26 Tom Lane wrote:
>>>>> I do not think this is an appropriate attitude for a committer to take.
>>>> I would like to have this clarified, as I keep running afoul of it.
>>> My two cents: I don't expect you to fix the MSVC scripts if you are
>>> uninterested in doing so.  But it would be appropriate to post a note
>>> to -hackers when you make a build system change that is going to need
>>> to be reflected in those scripts.  The people who do care about MSVC
>>> should not have to find that out by watching the buildfarm and then
>>> trying to reverse-engineer the cause of a failure.
>> As one of said people, I think that's perfectly reasonable. It'd make it
>> a lot easier to get a heads-up, or just the version of the patch right
>> before a commit to add it to. Some people do that, some don't - it would
>> be muchos helpful if all did.
>>
>> I don't expect unix hackers to always patch the msvc system as needed -
>> given that you don't have a way to test it, and no experience how those
>> tools work. (I know Tom has made a number of smaller fixes in them, but
>> it's mostly been around the areas of smaller changes, not actually
>> adding new stuff, for exmaple)
>>
>> If I had warning this time, for example, I could've said that I for one
>> can't fix that for a while, because my msvc build system is all geared
>> up around the git server (because it's so much easier to get the stuff
>> in and out of the VM when cross-deveoping than with CVS and the crippled
>> commandline toolchain on windows). And since that one is currently not
>> working properly, I need to find a different way to get the code in
>> there unless someone can fix it :-(
>>
>> Hopefully someone else can pick it up this time around? If not, I'll do
>> it as soon as I get things back up and working in my env.
>>
>> //Magnus
>>
>>
>> -- 
>> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-hackers
> 

Magnus Hagander <magnus@hagander.net> writes:
>> Exactly what type of changes affec the MSVC build system?

> Anything the system doesn't pick up automatically. That means most new
> build targets (but it will pick up a contrib automatically, and a
> conversion proc, for example). Also if modifications are made to the
> scripts that run (like gen_fmgr.sh).

> Just adding new files to exisitng makefiles, or adding a new subdir that
> adds more files to an existing target, should require no changes.

It might help clarify things if you say why it *didn't* pick up these
new foreign-server libraries.  Is it because they were new build
targets, or ??
        regards, tom lane

Tom Lane wrote:
> Magnus Hagander <magnus@hagander.net> writes:
>>> Exactly what type of changes affec the MSVC build system?
> 
>> Anything the system doesn't pick up automatically. That means most new
>> build targets (but it will pick up a contrib automatically, and a
>> conversion proc, for example). Also if modifications are made to the
>> scripts that run (like gen_fmgr.sh).
> 
>> Just adding new files to exisitng makefiles, or adding a new subdir that
>> adds more files to an existing target, should require no changes.
> 
> It might help clarify things if you say why it *didn't* pick up these
> new foreign-server libraries.  Is it because they were new build
> targets, or ??

Yes, they are new build targets, that's why it didn't catch them.

//Magnus

Magnus Hagander wrote:
>>
>>> Just adding new files to exisitng makefiles, or adding a new subdir that
>>> adds more files to an existing target, should require no changes.
>>>       
>> It might help clarify things if you say why it *didn't* pick up these
>> new foreign-server libraries.  Is it because they were new build
>> targets, or ??
>>     
>
> Yes, they are new build targets, that's why it didn't catch them.
>
>
>   

Also, because one of the Makefiles involved (src/foreign/Makefile) 
doesn't follow one of our standard patterns.

We (i.e. probably Magnus and I in the first instance) should think about 
creating a bit of a cookbook if we're going to persist with this build 
system.

cheers

andrew

Andrew Dunstan wrote:

> We (i.e. probably Magnus and I in the first instance) should think about  
> creating a bit of a cookbook if we're going to persist with this build  
> system.

Well, we were going to try CMake, but we need somebody to do the work.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera wrote:
> Andrew Dunstan wrote:
>
>   
>> We (i.e. probably Magnus and I in the first instance) should think about  
>> creating a bit of a cookbook if we're going to persist with this build  
>> system.
>>     
>
> Well, we were going to try CMake, but we need somebody to do the work.
>
>   

Yes, indeed.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:
> Also, because one of the Makefiles involved (src/foreign/Makefile) 
> doesn't follow one of our standard patterns.

Is there a really good reason why it doesn't?
(eg, why "FDW" and not "SUBDIRS"?)
        regards, tom lane

Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>> Also, because one of the Makefiles involved (src/foreign/Makefile) 
>> doesn't follow one of our standard patterns.
> 
> Is there a really good reason why it doesn't?
> (eg, why "FDW" and not "SUBDIRS"?)

If you put them in SUBDIRS, don't get they get linked into the backend
itself? They are supposed to be different build targets...

It's more like the conversion_procs stuff, which also doesn't use
SUBDIRS - it uses DIRS, but he idea is the same as FDW.

//Magnus

On Sunday 21 December 2008 00:59:27 Alvaro Herrera wrote:
> Andrew Dunstan wrote:
> > We (i.e. probably Magnus and I in the first instance) should think about
> > creating a bit of a cookbook if we're going to persist with this build
> > system.
>
> Well, we were going to try CMake, but we need somebody to do the work.

It did play around with CMake a while back.  It works OK.  I had libpq and 
psql building, for example.  The problem I see is that converting the build 
system will probably take many man-hours, and in the meantime, it would 
essentially create yet another build system to maintain.  Plus, we are not 
sure, of course, whether we will end up adopting CMake.

My preferred approach now is that the existing makefiles need to be refactored 
more aggressively first, using make functions.  We could incidentally design 
those functions similar to the CMake declarations, so a conversion, if we 
decided to do one, would be simple.  But doing that properly would require a 
newer GNU make version, so it needs some consideration first.  (I'm not 
talking about last week's release, but more like 4 years old versus the 10 
years old that we currently require.)

We can revisit this for the next release cycle.

Peter Eisentraut wrote:
> On Sunday 21 December 2008 00:59:27 Alvaro Herrera wrote:
>> Andrew Dunstan wrote:
>>> We (i.e. probably Magnus and I in the first instance) should think about
>>> creating a bit of a cookbook if we're going to persist with this build
>>> system.
>> Well, we were going to try CMake, but we need somebody to do the work.
> 
> It did play around with CMake a while back.  It works OK.  I had libpq and 
> psql building, for example. 

I did a similar thing for pgAdmin, and I found it pretty easy to do.
However, I think Dave spent some time on doing the "plugins" for
detecting wxWidgets and such. The point being that I think creating the
replacement parts for autoconf are a lot harder than creating them for
the make parts of the system.

How much of that did you look at?


> The problem I see is that converting the build 
> system will probably take many man-hours, and in the meantime, it would 
> essentially create yet another build system to maintain.  Plus, we are not 
> sure, of course, whether we will end up adopting CMake.

Yes, that is the problem. It will take some time, and we don't know.
Now, once it's up to working order we *could* probably get rid of (or at
least very drastically reduce) the current MSVC build stuff, which would
get us back down in the number of build systems. But during the actual
work we'll certainly have one extra, yes.



> My preferred approach now is that the existing makefiles need to be refactored 
> more aggressively first, using make functions.  We could incidentally design 
> those functions similar to the CMake declarations, so a conversion, if we 
> decided to do one, would be simple.  But doing that properly would require a 
> newer GNU make version, so it needs some consideration first.  (I'm not 
> talking about last week's release, but more like 4 years old versus the 10 
> years old that we currently require.)

That makes sense. I wonder how much that is going to require to be
rewritten in the MSVC build stuff. We don't actually parse the "guts" of
the Makefiles there, we just look for things like the list of object
files etc. Would those be affected?

I'm also a bit concerned about the mingw platform, given the experiences
I've had with toolchain compatibility there... But if it's 4 years old,
we're *probably* safe...

//Magnus

On Mon, Dec 29, 2008 at 12:49 PM, Magnus Hagander <magnus@hagander.net> wrote:
> I did a similar thing for pgAdmin, and I found it pretty easy to do.
> However, I think Dave spent some time on doing the "plugins" for
> detecting wxWidgets and such. The point being that I think creating the
> replacement parts for autoconf are a lot harder than creating them for
> the make parts of the system.

I did - in the wxWidgets case, the existing module had some behaved
badly in a number of ways and needed rewriting from scratch imho, and
the PostgreSQL module was non-existent (and pretty quick to write). In
fairness though, I expect the majority of the autoconf stuff that
would need replacing would be much easier - simple library/header
checks are very straightforward.


-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

"Dave Page" <dpage@pgadmin.org> writes:
> On Mon, Dec 29, 2008 at 12:49 PM, Magnus Hagander <magnus@hagander.net> wrote:
>> ... The point being that I think creating the
>> replacement parts for autoconf are a lot harder than creating them for
>> the make parts of the system.

> I did - in the wxWidgets case, the existing module had some behaved
> badly in a number of ways and needed rewriting from scratch imho, and
> the PostgreSQL module was non-existent (and pretty quick to write). In
> fairness though, I expect the majority of the autoconf stuff that
> would need replacing would be much easier - simple library/header
> checks are very straightforward.

But of course those are just as straightforward in autoconf.  It's
the not-straightforward stuff that's going to be a PITA to translate.
        regards, tom lane

On Mon, Dec 29, 2008 at 11:47 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> But of course those are just as straightforward in autoconf.  It's
> the not-straightforward stuff that's going to be a PITA to translate.

As much as I dislike autotools, I really despise CMake; it's a nasty
piece of work and I hope we don't switch to it.  Though, as I must've
missed it, what's the main complaint with the current build system?

-- 
Jonah H. Harris, Senior DBA
myYearbook.com

Jonah H. Harris wrote:
> On Mon, Dec 29, 2008 at 11:47 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > But of course those are just as straightforward in autoconf.  It's
> > the not-straightforward stuff that's going to be a PITA to translate.
> 
> As much as I dislike autotools, I really despise CMake; it's a nasty
> piece of work and I hope we don't switch to it.  Though, as I must've
> missed it, what's the main complaint with the current build system?

Impossible to use autoconf on Win32.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Jonah H. Harris escribió:
> On Mon, Dec 29, 2008 at 11:47 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > But of course those are just as straightforward in autoconf.  It's
> > the not-straightforward stuff that's going to be a PITA to translate.
> 
> As much as I dislike autotools, I really despise CMake; it's a nasty
> piece of work and I hope we don't switch to it.  Though, as I must've
> missed it, what's the main complaint with the current build system?

The fact that we have to have a mostly parallel build system for MSVC.
Anything beyond the most trivial changes must be manually patched there
too, which is annoying and breaks the MSVC build frequently.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Bruce Momjian <bruce@momjian.us> writes:

> Jonah H. Harris wrote:
>> On Mon, Dec 29, 2008 at 11:47 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> > But of course those are just as straightforward in autoconf.  It's
>> > the not-straightforward stuff that's going to be a PITA to translate.
>> 
>> As much as I dislike autotools, I really despise CMake; it's a nasty
>> piece of work and I hope we don't switch to it.  Though, as I must've
>> missed it, what's the main complaint with the current build system?
>
> Impossible to use autoconf on Win32.

I don't think that's actually it. We used to use autoconf when we used cygwin
to build, didn't we? And there are other tools that use autoconf on windows.

We could use autoconf on win32 using cygwin utilities for things like sh and
awk. Just because we use those tools doesn't mean we have to use a cygwin
compiler or linker to actually do the build.

And we could always ship the preconfigured pg_config.h from a normal Windows
machine with cygwin installed since they're all the same (in theory).

I think it has more to do with the build-time tools. We have Makefile rules
that use awk or sed in them and those wouldn't work unless you have cygwin
installed when building. And in any case we want to use MSVC project files and
MSVC's make-equivalent to actually drive the build which kind of rules out
using the Makefile rules as-is.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's RemoteDBA services!

Gregory Stark wrote:
>
> We could use autoconf on win32 using cygwin utilities for things like sh and
> awk. Just because we use those tools doesn't mean we have to use a cygwin
> compiler or linker to actually do the build.
>   

Making Cygwin a build time requirement for using MSVC is something we 
seriously wish to avoid.

> And we could always ship the preconfigured pg_config.h from a normal Windows
> machine with cygwin installed since they're all the same (in theory).
>
> I think it has more to do with the build-time tools. We have Makefile rules
> that use awk or sed in them and those wouldn't work unless you have cygwin
> installed when building. And in any case we want to use MSVC project files and
> MSVC's make-equivalent to actually drive the build which kind of rules out
> using the Makefile rules as-is.
>
>   

Quite so. CMake outputs MSVC Project files, as I understand it. If you 
know of another cross-platform build tool that will do that then speak up.

cheers

andrew

Gregory Stark wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> 
>> Jonah H. Harris wrote:
>>> On Mon, Dec 29, 2008 at 11:47 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>> But of course those are just as straightforward in autoconf.  It's
>>>> the not-straightforward stuff that's going to be a PITA to translate.
>>> As much as I dislike autotools, I really despise CMake; it's a nasty
>>> piece of work and I hope we don't switch to it.  Though, as I must've
>>> missed it, what's the main complaint with the current build system?
>> Impossible to use autoconf on Win32.
> 
> I don't think that's actually it. We used to use autoconf when we used cygwin
> to build, didn't we? And there are other tools that use autoconf on windows.

No. We've never used cygwin for the win32 build, only for the cygwin build.

We've used mingw, which is completely different. But it's still a major
PITA to set up, and binaries produced using it aren't compatible with
windows tracing and debugging tools for example.

Heck, mingw is barely compatible with itself some days :-)


> We could use autoconf on win32 using cygwin utilities for things like sh and
> awk. Just because we use those tools doesn't mean we have to use a cygwin
> compiler or linker to actually do the build.

It does. autoconf is not compatible with MSVC. We did investigate that
before we shipped.


> And we could always ship the preconfigured pg_config.h from a normal Windows
> machine with cygwin installed since they're all the same (in theory).

That's basically what we do now (except it's not cygwin, and it's only
"almost mingw" since MSVC and mingw aren't entirely compatible)


> I think it has more to do with the build-time tools. We have Makefile rules
> that use awk or sed in them and those wouldn't work unless you have cygwin
> installed when building. And in any case we want to use MSVC project files and

Even so, "unix style" Makefiles don't work with MSVC. The compiler is
too different from all unix style compilers. We did try this...

But yes, the build time stuff is what's hard. If you look at the msvc
build stuff now it's mostly parsing our Makefiles and generating the
output. The autoconf stuff is a lot easier since we only target a single
platform.


> MSVC's make-equivalent to actually drive the build which kind of rules out
> using the Makefile rules as-is.

The point is that cmake generates both makefiles and project files.

What we *could* do is define our own "metaformat" for the build stuff,
and use that to *generate* our own makefiles. That would then be instead
of using the CMake format, we use our own perlscript or something. Some
projects do this - I think OpenSSL for example.

That way we'd have a single master, and keep using autoconf for what it
does.

//Magnus

Andrew Dunstan wrote:
> Quite so. CMake outputs MSVC Project files, as I understand it. If you 
> know of another cross-platform build tool that will do that then speak 
> up.
>
I think the wxWidgets team have one, and I think scons has some support 
for doing that, though I haven't tried
that part of scons. The first uses Perl, scons uses Python.