Thread: add server include files to default installation?
Dear hackers, I wish to submit a small patch so that server includes and all necessary configuration files could be installed *by default*. The current status is that server includes files are only installed if explicitly required (make install-all-headers). The idea is that external extension modules (new types or whatever) could then *rely* on the fact that these files are available, which would make developping and installing these extensions quite easier. As an illustration, the apache software installs by default all necessary includes and even a special tailored command (apxs) so as to help extension modules to be compiled, installed and even configured easilly. Is there any opposition to this move? Silence will mean consent;-) Thanks in advance, have a nice day, -- Fabien Coelho - coelho@cri.ensmp.fr
Fabien COELHO <coelho@cri.ensmp.fr> writes:
> I wish to submit a small patch so that server includes and
> all necessary configuration files could be installed *by default*.
There is a reason why install-all-headers is not the default. It is
that it shouldn't be the default: most people do not need it, because
they will never build any C extensions for themselves. It would just
be a waste of disk space for them.
If we did do such a thing it would have zero impact on the majority of
users anyway, because the RPM and other packagers will still put these
files into separate postgresql-devel packages, which would still not be
installed by default.
regards, tom lane
Fabien COELHO wrote: > Dear hackers, > > I wish to submit a small patch so that server includes and > all necessary configuration files could be installed *by default*. > > The current status is that server includes files are only installed > if explicitly required (make install-all-headers). > > The idea is that external extension modules (new types or whatever) > could then *rely* on the fact that these files are available, which > would make developping and installing these extensions quite easier. > > As an illustration, the apache software installs by default all > necessary includes and even a special tailored command (apxs) so as > to help extension modules to be compiled, installed and even configured > easilly. > > Is there any opposition to this move? Silence will mean consent;-) > > Thanks in advance, have a nice day, > If the intention is that external extension types should be able to build using the default installation you must also include the src/Makefile.global and the src/Makefile.shlib. Cygwin ports are dependent on the presence of src/utils/dllinit.c also (perhaps true for win32 too). A dllinit.o would be sufficient of course. Kind regards, Thomas Hallgren
Fabien COELHO wrote: > > Dear hackers, > > I wish to submit a small patch so that server includes and > all necessary configuration files could be installed *by default*. > > The current status is that server includes files are only installed > if explicitly required (make install-all-headers). > > The idea is that external extension modules (new types or whatever) > could then *rely* on the fact that these files are available, which > would make developping and installing these extensions quite easier. > > As an illustration, the apache software installs by default all > necessary includes and even a special tailored command (apxs) so as > to help extension modules to be compiled, installed and even configured > easilly. > > Is there any opposition to this move? Silence will mean consent;-) Agreed. I would also like to see Makefile.global installed. pg_config.h has C-level configs, and Makefile.global has the Makefile-level configs. We have identified that we need to make configuration easier for 3rd party apps. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Sat, 15 May 2004, Bruce Momjian wrote: > Fabien COELHO wrote: > > > > Dear hackers, > > > > I wish to submit a small patch so that server includes and > > all necessary configuration files could be installed *by default*. > > > > The current status is that server includes files are only installed > > if explicitly required (make install-all-headers). > > > > The idea is that external extension modules (new types or whatever) > > could then *rely* on the fact that these files are available, which > > would make developping and installing these extensions quite easier. > > > > As an illustration, the apache software installs by default all > > necessary includes and even a special tailored command (apxs) so as > > to help extension modules to be compiled, installed and even configured > > easilly. > > > > Is there any opposition to this move? Silence will mean consent;-) > > Agreed. I would also like to see Makefile.global installed. > pg_config.h has C-level configs, and Makefile.global has the > Makefile-level configs. Don't agree with Makefile.global, cause then you have to also install the physical template files for the various operating system too, no? What else does Makefile.global rely on? ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
Marc G. Fournier wrote: > > > As an illustration, the apache software installs by default all > > > necessary includes and even a special tailored command (apxs) so as > > > to help extension modules to be compiled, installed and even configured > > > easilly. > > > > > > Is there any opposition to this move? Silence will mean consent;-) > > > > Agreed. I would also like to see Makefile.global installed. > > pg_config.h has C-level configs, and Makefile.global has the > > Makefile-level configs. > > Don't agree with Makefile.global, cause then you have to also install the > physical template files for the various operating system too, no? What > else does Makefile.global rely on? I think you would need the src/makefile files. The template files are only used during configure. You might need some more include stuff too, not sure. I see: include $(top_builddir)/src/Makefile.port and this gets confusing because Makefile.port is a symlink to the actual file. I wonder if we could copy the actual file as Makefile.port. I think that would work. Anyway, I can also see packagers greping the Makefile for information. For example, it is hard to know the thread flags except from the file. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Dear Tom, > > I wish to submit a small patch so that server includes and > > all necessary configuration files could be installed *by default*. > > There is a reason why install-all-headers is not the default. Sure. I hope so! I'm questionning the pros and cons. I'm arguing that the cons against the current situation are strong. Thanks for providing the pros, so it can be discussed. > It is that it shouldn't be the default: most people do not need it, How will people know that they will not need it afterwards? As far as I'm concerned, I noticed it afterwards:-( I'm a pretty stupid user, but I'm sure I'm not the only one. For instance several weeks after installing a pg, I wanted to test tsearch2 on one database. What do I need to do? Well, basically I need a full recompile, or at least a full extract of the sources, re-configure and compilation of the relevant contrib part. When reconfiguring, I need to remember what options I used. Now, as someone trying to develop extensions, I need to rely on some sound and extendable default installation. > because they will never build any C extensions for themselves. Hey, how do they know that before hand? They might need extensions. If they need extensions, it is likely that they may have a C part. Most contrib have a C part. External projects are likely to have a C part. Some in user land (e.g. SPI_*), but not necessarilly. Say aclitem accessor functions or new aggregates that may be needed for some project that may be of interest to people. If there are not in, you have to compile them. If the project cannot rely on the fact that everything needed will be provided, it is no good. > It would just be a waste of disk space for them. 2MB... that is less than 0.002 EUR. Moreover, I do not think that it is wasted, especially as postgresql is moving things outside the main project (pgfoundry, contrib, gborg). You must help external projects to live if you want them to leave;-) As a compromise, we can have two makefile targets, say "install" and "light-install". I argue that the default should be the one which enable later extensions. > If we did do such a thing it would have zero impact on the majority of > users anyway, because the RPM and other packagers will still put these > files into separate postgresql-devel packages, which would still not be > installed by default. Thanks for this argument! If packagers do it (I know of BSD ports) it mean that they think it is useful. So it should be the default for people who *bother* to install pg by hand, so that they don't have to do it twice because they did not notice this installation subtlety. IMHO, it is the core business of postgresql to help external projects by providing a usable default infrastructure for them. As another example of this bad behavior, having external projects (pgadmin or phppgadmin) to "parse" aclitem descriptors because you want to keep things opaque and they need it anyway is damaging. Thanks for you answer, have a nice day, -- Fabien Coelho - coelho@cri.ensmp.fr
Hi all, Attached is a patch against HEAD implementing tablespaces. I've done some testing on Linux and BSD. I've also compiled without HAVE_SYMLINK defined -- which determines whether or not tablespaces are available. The reason for this is that symlinks are used extensively to simplify access to relations in hairy places. This is extensive discussion of this in the archives. There are some outstanding TODOs. These include pg_dump documentation regression tests comment on Chris KL has offered to tackle these. Other outstanding things which need to be resolved: XLog entries: Within a tablespace, the files for different databases are isolated into directories. Those directory names are the database oid for those files. Since we create those files, we should log them. The question is, what is the best way: appending it to an xlrec and storing its length in the xlrec or adding another rdata with that in it? The same goes for symlinks, where we need to store the oldpath and newpath. Alternative database location: Should this code be removed now? Drop tablespace: I haven't added an interlock here (as suggested by Tom) to handle situations where one transaction schedules the removal of a tablespace and concurrently another transaction makes use of that tablespace. I'd appreciate some advice on what this would look like: should a command creating an object touch a CREATE a file and a DROP TABLESPACE block until the file goes away, then check if it can drop the tablespace (ie, that the tablespace is empty)? Likewise when DROP TABLESPACE proceeds, should it block creates? The other thing is handling DROP TABLESPACE in a transaction where the tablespace has been emptied (using DROP TABLE, etc) during that transaction. Because we only schedule the unlink() of the relation, the data files will (and must) still be around. Is it reasonable to look inside PendingRelDeletes to see if the tablespace will be empty at COMMIT? Basic usage rundown initdb sets up a new directory pg_tablespaces with some symlinks with in it. To create a tablespace, you must first up create a tablespace directory. Then you can create a tablespace inside postgres using: CREATE TABLESPACE <name> LOCATION '/path/to/tbl/dir' Then, the DDL commands can have a tablespace associatged with it: CREATE DATABASE ... TABLESPACE <name> CREATE SCHEMA ... TABLESPACE <name> CREATE TABLE ... TABLESPACE <name> CREATE INDEX ... TABLESPACE <name> CREATE SEQUENCE ... TABLESPACE <name> Comments, criticisms, etc, all welcome, of course. Thanks, Gavin
Dear hackers, > Agreed. I would also like to see Makefile.global installed. > pg_config.h has C-level configs, and Makefile.global has the > Makefile-level configs. There is also "config.status" which is definitely of interest as it allows to recreate the build tree, and which is not installed by default. Independently of the actual file list to be included, where could these makefiles be installed? - share is for "architecture independent" files. Not really the case. - include is rather for C files... but Makefile.global and others are actually included, so it may make sense? If so, shouldit be directly in the include/postgresql subdir, or some special subdir, say "include/postgresql/config"? - some other directory? I would tend to put everything in "include/postgresql/config":- Makefile.global- a copy of Makefile.port (the actual file,not the link)- config.status- possibly other included files (NLS? win32? cygwin?) Another issue is that Makefile.global seems hardwired to be in the "src" subdirectory of a postgresql source tree, and to find other files there: # Pull in platform-specific magicinclude $(top_builddir)/src/Makefile.port This is an issue for external tools that would like to include Makefile.global so as to be in the same environment as the server compilation. Any idea? The best I have would be to create a "src" subdir in the installation, so that top_builddir could be set to ".../include/postgresql/config" and everything would work from that point of view. -- Fabien COELHO _ http://www.cri.ensmp.fr/~coelho _ Fabien.Coelho@ensmp.fr CRI-ENSMP, 35, rue Saint-Honoré, 77305 Fontainebleaucedex, France phone: (+33|0) 1 64 69 {voice: 48 52, fax: 47 09, standard: 47 08} ________ All opinionsexpressed here are mine _________
Gavin Sherry wrote: > Alternative database location: > > Should this code be removed now? Yes, I believe we agreed on this. One of the committers will take care of that. The only downside to removal is that folks without symlinks (I believe Win32 only) will loose that functionality with nothing to replace it. However, I think the clarity of removing it is worth it. Also, I think someone had a special way to do symlinks on Win32 and we should look into that. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
> I would tend to put everything in "include/postgresql/config": On second thought, I would hesitate with "lib/...", as config file are plateform specific so they cannot be shared with other plateforms, although it could be the case for include files that they could be shared. -- Fabien Coelho - coelho@cri.ensmp.fr
Bruce Momjian wrote: >The only downside to removal is that folks without symlinks (I believe >Win32 only) will loose that functionality with nothing to replace it. >However, I think the clarity of removing it is worth it. Also, I think >someone had a special way to do symlinks on Win32 and we should look >into that. > > Windows 2000 and later support mount points - you can attach a new partition as C:\pgsql\data\xlog instead of D:\. That might be enough for most users. IIRC there was a tool to create arbitrary links, but it was removed just before W2K final. -- Manfred
"Manfred Spraul" <manfred@colorfullife.com> wrote in message news:40A8EEE9.3000405@colorfullife.com... > Bruce Momjian wrote: > > >The only downside to removal is that folks without symlinks (I believe > >Win32 only) will loose that functionality with nothing to replace it. > >However, I think the clarity of removing it is worth it. Also, I think > >someone had a special way to do symlinks on Win32 and we should look > >into that. > > > > > Windows 2000 and later support mount points - you can attach a new > partition as C:\pgsql\data\xlog instead of D:\. That might be enough for > most users. IIRC there was a tool to create arbitrary links, but it was > removed just before W2K final. > If you run NTFS, it's still possible to use arbitrary links. In the Windows world, they are called junctions. Microsoft does not provide a junction tool for some reason (perhaps because it's limited to NTFS). A good tool, free and with source, can be found here http://www.sysinternals.com/ntw2k/source/misc.shtml#junction I use this tool myself. Works like a charm. Kind regards, Thomas Hallgren
Thomas Hallgren wrote: >"Manfred Spraul" <manfred@colorfullife.com> wrote in message >news:40A8EEE9.3000405@colorfullife.com... > > >>Bruce Momjian wrote: >> >> >> >>>The only downside to removal is that folks without symlinks (I believe >>>Win32 only) will loose that functionality with nothing to replace it. >>>However, I think the clarity of removing it is worth it. Also, I think >>>someone had a special way to do symlinks on Win32 and we should look >>>into that. >>> >>> >>> >>> >>Windows 2000 and later support mount points - you can attach a new >>partition as C:\pgsql\data\xlog instead of D:\. That might be enough for >>most users. IIRC there was a tool to create arbitrary links, but it was >>removed just before W2K final. >> >> >> >If you run NTFS, it's still possible to use arbitrary links. In the Windows >world, they are called junctions. Microsoft does not provide a junction tool >for some reason (perhaps because it's limited to NTFS). A good tool, free >and with source, can be found here >http://www.sysinternals.com/ntw2k/source/misc.shtml#junction I use this tool >myself. Works like a charm. > > > We've looked at it before. Apart from anything else I don't think its license is compatible with PostgreSQL's. Also, IIRC NTFS junctions also have some severe limitations. cheers andrew
> Alternative database location: > > Should this code be removed now? I think that this: CREATE DATABASE blah LOCATION 'xyz'; Should now be interpreted to mean: CREATE TABLESPACE blah_tbsp LOCATION 'xyz'; CREATE DATABSE blah TABLESPACE blah_tbsp; Or something like that... Chris
Marc G. Fournier wrote: > On Sat, 15 May 2004, Bruce Momjian wrote: > >> Fabien COELHO wrote: >> > >> > Dear hackers, >> > >> > I wish to submit a small patch so that server includes and >> > all necessary configuration files could be installed *by default*. >> > >> > The current status is that server includes files are only installed >> > if explicitly required (make install-all-headers). >> > >> > The idea is that external extension modules (new types or whatever) >> > could then *rely* on the fact that these files are available, which >> > would make developping and installing these extensions quite easier. >> > >> > As an illustration, the apache software installs by default all >> > necessary includes and even a special tailored command (apxs) so as >> > to help extension modules to be compiled, installed and even configured >> > easilly. >> > >> > Is there any opposition to this move? Silence will mean consent;-) >> >> Agreed. I would also like to see Makefile.global installed. >> pg_config.h has C-level configs, and Makefile.global has the >> Makefile-level configs. > > Don't agree with Makefile.global, cause then you have to also install the > physical template files for the various operating system too, no? What > else does Makefile.global rely on? Depending on the OS you also need funny things like mkldexport shell scripts and the like. I thought we had a consensus of providing a template build environment for external modules. This half-baked attempt is not doing it. The makefiles make assumptions about their location that aren't true any more when they're installed somewhere else. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Dear Jan, > Depending on the OS you also need funny things like mkldexport shell > scripts and the like. Possibly, yes... I don't know. I really need a list, then it is pretty easy to update all makefiles. > I thought we had a consensus of providing a template build environment > for external modules. Good! Where is it? I mean the template, not the consensus. > This half-baked attempt is not doing it. The makefiles make assumptions > about their location that aren't true any more when they're installed > somewhere else. I noticed and took care of this point. In fact, it really depends on top_builddir, so by setting this macro appropriately it should work. Something like: top_builddir := $(shell pg_config --insbuilddir)include $(top_builddir)/src/Makefile.global This mean that relevant files must be installed in subdirectories that are named as the postgresql source tree... I agree that it is not beautiful, but it seems to me that it is the simplest way to have contrib-like makefiles outside of the main tree, and to be able to reuse the whole infrastructure. Obviously, any better idea is welcome, esp. if I do not have to do it myself;-) The current status is that *nothing* is provided, so I'm trying to have something better than nothing with a reasonnable effort. -- Fabien Coelho - coelho@cri.ensmp.fr
Fabien COELHO wrote:
> Any idea? The best I have would be to create a "src" subdir in the
> installation, so that top_builddir could be set to
> ".../include/postgresql/config" and everything would work from that
> point of view.
>
This is from an extension module author's point of view. My module will
reference these two files only:
Makefile.global
Makefile.shlib
I'm not the least interested in how these files are organized internally
or where they are placed in reference to what they contain. I don't care
where the include files, config files, or other files are located, as
long as these two files will set my flags accordingly. I think it's
important to recognize this separation of concern. Extension modules
should not need to worry about anything but where to find these two files.
If I have a "pg_config --makefiledir" that tells me where to find the
files, that should be all I need:
makefiledir := $(shell pg_config --makefiledir) include $(makefiledir)/Makefile.global ...
include$(makefiledir)/Makefile.shlib
If I go in and parse the makefiles to dig out information or do
something equally horrible, I'm either doing something wrong or
something is missing in the "contract" between the backend and the
extension module and should be rectified there. The contract, the way I
see it, is constituted by the two files.
Just trying to give you a bit more freedom to place your files :-)
Kind regards,
Thomas Hallgren
> This is from an extension module author's point of view. My module will > reference these two files only: > > Makefile.global > Makefile.shlib These files possibly include other makefiles. > I'm not the least interested in how these files are organized internally Sure. > If I have a "pg_config --makefiledir" that tells me where to find the > files, that should be all I need: > > makefiledir := $(shell pg_config --makefiledir) > include $(makefiledir)/Makefile.global > ... > Just trying to give you a bit more freedom to place your files :-) You do not understand how bad it is from the inside;-) In the current Makefile.global, you find things like: include $(top_...)/src/Makefile.port That is the makefile knows where it is expected to reside wrt the source tree, and use this information. In order to allow what you describe, I would have to fix this issue, that is rewrite somehow Makefile.global and other makefiles. Maybe I could do that, but I was trying hard to avoid it, as there may be some kind of portability issues. Thanks for you comment, -- Fabien Coelho - coelho@cri.ensmp.fr
I have a fairly good understanding of how the Makefile.global etc. is organized. My point is, in order to maintain a good separation of concern, I should not use this knowledge. IMHO, there's a need for a well documented "module extension build API" accessible from a well known place. This is more important than to include it in the default installation. Once in place it will lessen the dependencies between the server installation and the extension modules. The PostgreSQL core can then change the make structure, directory layout, etc. without affecting the extension modules that uses the build API. If they don't use it, any incompatibility is their responsability. Your suggestion to rewrite the Makefile.global etc. is a good first step. Kind regards, Thomas Hallgren
Dear Thomas, > My point is, in order to maintain a good separation of concern, I > should not use this knowledge. I agree. > IMHO, there's a need for a well documented "module extension build API" > accessible from a well known place. Sure. I'm not really addressing that at the moment, I'm addressing having the necessary files. Given gnu-make constraints about makefile inclusions, that requires to have the same layout as the initial source tree, whatever it is. I'm not sure that dropping all includes would be an easy move, nor even desirable. > This is more important than to include it in the default installation. Well, the point of having an infrastructure if extensions cannot rely it being there when needed is not clear to me. > Your suggestion to rewrite the Makefile.global etc. is a good first step. It is what I'm trying to avoid;-) It is not just this makefile, but the whole include-based makefiles that are at stake. I think that an infrastructure is possible without fixing these files. So as I understand your point as "the layout constraints should not be visible from the extensions API", I suggest that extension makefiles should look like (adapted from contrib): PGXSDIR := $(shell pg_config --pgxs-dir)include $(PGXSDIR)/pgxs_begin.mk[... set required macros as in contrib ...]include$(PGXSDIR)/pgxs_end.mk It is independent from what my first patch addresses, that is tryinbg to put all required files somewhere on install. Thanks for your point, -- Fabien Coelho - coelho@cri.ensmp.fr
Fabien COELHO wrote: > > Dear Thomas, > > > My point is, in order to maintain a good separation of concern, I > > should not use this knowledge. > > I agree. > > > IMHO, there's a need for a well documented "module extension build API" > > accessible from a well known place. > > Sure. > > I'm not really addressing that at the moment, I'm addressing having the > necessary files. Given gnu-make constraints about makefile inclusions, > that requires to have the same layout as the initial source tree, whatever > it is. I'm not sure that dropping all includes would be an easy move, nor > even desirable. > > > > This is more important than to include it in the default installation. > > Well, the point of having an infrastructure if extensions cannot rely it > being there when needed is not clear to me. Agreed. If we are pushing things out, it seems it is our duty to make it easy for outside things to integrate and build properly. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Fabien COELHO wrote:
>> Well, the point of having an infrastructure if extensions cannot rely it
>> being there when needed is not clear to me.
> Agreed. If we are pushing things out, it seems it is our duty to make
> it easy for outside things to integrate and build properly.
It does not thereby follow that we should try to merge devel and base
packages (to express it in RPM terms). Compiling extension packages is
not and probably never will be something that the average user does,
and there's no reason to burden him with the disk footprint to support
something he's not gonna do.
We do need to work out what our story is for compiling extensions
without the original source/build trees. But the needed files should
be an install option, not the default.
regards, tom lane
Dear Tom, > > Agreed. If we are pushing things out, it seems it is our duty to make > > it easy for outside things to integrate and build properly. > > It does not thereby follow that we should try to merge devel and base > packages (to express it in RPM terms). They are not necessarily merged, and it is quite easy to separate them from the rpm maintainer point of view if s?he desire to do so. This is already done for apache for instance, as the default installation includes all build utilities. There is no problem separate build utils, or even to make it easier than it is already if you want. > Compiling extension packages is not and probably never will be something > that the average user does, and there's no reason to burden him with the > disk footprint to support something he's not gonna do. We're arguing about the public which does compile postgresql. Guys or girls that downloaded the source, configure and so. That is not the average user indeed, but occasional sysadmins like me, and I like my life to be easier. I'm ready to pay 0.002 EUR of disk space for that, especially as the 0.002 EUR (about $0.002 for you) are those of my employer;-) We can discuss whether that price is too high for the average sysadmin, but at the cost of my time and yours, I think we spent quite a lot of money arguing about this issue that would pay for all the disk space in the world that is needed;-) > We do need to work out what our story is for compiling extensions > without the original source/build trees. We agree;-) I've made suggestions and send a "proof of concept patch" which is waiting for your comments and better ideas. I'm not sure it is the best ever possible way to allow extensions, but it works and it is reasonnable enough IMHO, so it is at least a base for discussion. > But the needed files should be an install option, not the default. The previous point can be solved independetly of this issue, as you pointed out. Let's begin with that. -- Fabien Coelho - coelho@cri.ensmp.fr