Thread: Faster install-sh in C
Patchers, I wrote an "install" program in C. It's supposed to replace the config/install-sh script, limited to the functionality we need, i.e. what is in Makefiles in the Pg main source tree. The main objective of this exercise is to reduce "make install" execution time; a part of that is being able to install multiple files with one command. Portability testing right now is limited to my machine, which is Linux with glibc 2.3.2 (Debian Sid). With this in place, "make install" in src/include takes 17 seconds on this machine, where the script version takes more than a minute. I think this is a useful improvement. Right now I'm missing a Makefile rule for it. It needs the pg_progname symbol from src/port, and the includes from $(srcdir)/src/include and $(builddir)/src/include. From the config directory, this works: $(CC) $(CFLAGS) -I$(top_srcdir)/src/include -I$(top_builddir)/src/include ../src/port/path.o ../src/port/exec.o $< -o $@ Also, I don't know how to force it to build before executing "install", short of putting it as a dependency in every makefile (which sounds silly to me). I attach the source code, and a src/include/Makefile modification to use the multi-file install feature. -- Alvaro Herrera (<alvherre[a]dcc.uchile.cl>) "Hay quien adquiere la mala costumbre de ser infeliz" (M. A. Evans)
Attachment
This has been saved for the 8.1 release:
http:/momjian.postgresql.org/cgi-bin/pgpatches2
---------------------------------------------------------------------------
Alvaro Herrera wrote:
> Patchers,
>
> I wrote an "install" program in C. It's supposed to replace the
> config/install-sh script, limited to the functionality we need, i.e.
> what is in Makefiles in the Pg main source tree. The main objective of
> this exercise is to reduce "make install" execution time; a part of that
> is being able to install multiple files with one command.
>
> Portability testing right now is limited to my machine, which is Linux
> with glibc 2.3.2 (Debian Sid).
>
> With this in place, "make install" in src/include takes 17 seconds on
> this machine, where the script version takes more than a minute. I
> think this is a useful improvement.
>
>
> Right now I'm missing a Makefile rule for it. It needs the pg_progname
> symbol from src/port, and the includes from $(srcdir)/src/include and
> $(builddir)/src/include. From the config directory, this works:
>
> $(CC) $(CFLAGS) -I$(top_srcdir)/src/include -I$(top_builddir)/src/include ../src/port/path.o ../src/port/exec.o $< -o
$@
>
> Also, I don't know how to force it to build before executing "install",
> short of putting it as a dependency in every makefile (which sounds
> silly to me).
>
> I attach the source code, and a src/include/Makefile modification to
> use the multi-file install feature.
>
> --
> Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
> "Hay quien adquiere la mala costumbre de ser infeliz" (M. A. Evans)
[ Attachment, skipping... ]
[ Attachment, skipping... ]
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
Alvaro Herrera wrote: > I wrote an "install" program in C. It's supposed to replace the > config/install-sh script, limited to the functionality we need, i.e. > what is in Makefiles in the Pg main source tree. The main objective of > this exercise is to reduce "make install" execution time; a part of that > is being able to install multiple files with one command. What's the status of this patch? -Neil
Am Freitag, 4. März 2005 01:11 schrieb Neil Conway: > Alvaro Herrera wrote: > > I wrote an "install" program in C. It's supposed to replace the > > config/install-sh script, limited to the functionality we need, i.e. > > what is in Makefiles in the Pg main source tree. The main objective of > > this exercise is to reduce "make install" execution time; a part of that > > is being able to install multiple files with one command. > > What's the status of this patch? I'm not convinced that this is the best approach. Apparently, PostgreSQL is about the only project that is somehow bothered by this. It would be more productive to find out why we can't use the system install and find fixes for that. Anyway, if you want to speed up your installation today, just override the INSTALL variable by hand. -- Peter Eisentraut http://developer.postgresql.org/~petere/
On Fri, Mar 04, 2005 at 11:20:07AM +0100, Peter Eisentraut wrote: > Am Freitag, 4. März 2005 01:11 schrieb Neil Conway: > > Alvaro Herrera wrote: > > > I wrote an "install" program in C. It's supposed to replace the > > > config/install-sh script, limited to the functionality we need, i.e. > > > what is in Makefiles in the Pg main source tree. The main objective of > > > this exercise is to reduce "make install" execution time; a part of that > > > is being able to install multiple files with one command. > > > > What's the status of this patch? I was playing a little with Make just before leaving, but since I'm hardly an expert I was having difficulty making it all work. Also, because I need to include bits from both the source dir and the build dir (to get get_progname from the port dir and signal handling from libpq), it was getting somewhat ugly. With a little more effort it can be made to work cleanly. > I'm not convinced that this is the best approach. Apparently, PostgreSQL is > about the only project that is somehow bothered by this. It would be more > productive to find out why we can't use the system install and find fixes for > that. Well, apparently everyone says the system install is not portable and the discussion stops there. Probably some projects need a sh install because they don't require a compiler, but we do; we won't require anything beyond what we already do to have a C install. Maybe most projects don't install a lot of files like we do, or maybe people just don't care. (Neither did I care back when installing all headers wasn't the default behaviour.) I do care now. And given that the work is mostly already done, I don't see why we couldn't use it. > Anyway, if you want to speed up your installation today, just override the > INSTALL variable by hand. This doesn't solve the fact that the INSTALL program is called once for each file in the include directories. -- Alvaro Herrera (<alvherre[@]dcc.uchile.cl>) Y una voz del caos me habló y me dijo "Sonríe y sé feliz, podría ser peor". Y sonreí. Y fui feliz. Y fue peor.
Alvaro Herrera wrote: > Well, apparently everyone says the system install is not portable and > the discussion stops there. Well, who actually says that? I know that I was the one who actually coded up the current avoid-system-install-at-all-cost behavior, but only because we were too annoyed at the time to do more research. Now, as long as someone's spending time on it, it may be worth investigating what is wrong and provide an explanation for the benefit of the entire autotools-using community. -- Peter Eisentraut http://developer.postgresql.org/~petere/
On Mon, Mar 07, 2005 at 07:50:23PM +0100, Peter Eisentraut wrote: > Alvaro Herrera wrote: > > Well, apparently everyone says the system install is not portable and > > the discussion stops there. > > Well, who actually says that? I know that I was the one who actually > coded up the current avoid-system-install-at-all-cost behavior, but > only because we were too annoyed at the time to do more research. Now, > as long as someone's spending time on it, it may be worth investigating > what is wrong and provide an explanation for the benefit of the entire > autotools-using community. Well, the problem with this approach is that while I did have time to write the C replacement, I didn't (and still don't) have access to a lot of systems in order to check what the differences between the install programs are. Also, keep in my that this C install program has the extra feature of being able to install multiple files on one invocation, per suggestion from Tom Lane. This allows us to save the nested for-loop in src/include/Makefile. GNU install (available on my system) also has this capability, but would we be able to use the trick if we had to cater for the lowest common denominator found on other, non-GNU-enabled systems? -- Alvaro Herrera (<alvherre[@]dcc.uchile.cl>) "Llegará una época en la que una investigación diligente y prolongada sacará a la luz cosas que hoy están ocultas" (Séneca, siglo I)
Alvaro Herrera <alvherre@dcc.uchile.cl> writes:
> Also, keep in my that this C install program has the extra feature of
> being able to install multiple files on one invocation, per suggestion
> from Tom Lane. This allows us to save the nested for-loop in
> src/include/Makefile. GNU install (available on my system) also has
> this capability, but would we be able to use the trick if we had to
> cater for the lowest common denominator found on other, non-GNU-enabled
> systems?
We would definitely not ... and my suspicion is that that is the largest
single component of the available speedup.
regards, tom lane
Tom Lane wrote: > Alvaro Herrera <alvherre@dcc.uchile.cl> writes: > > Also, keep in my that this C install program has the extra feature of > > being able to install multiple files on one invocation, per suggestion > > from Tom Lane. This allows us to save the nested for-loop in > > src/include/Makefile. GNU install (available on my system) also has > > this capability, but would we be able to use the trick if we had to > > cater for the lowest common denominator found on other, non-GNU-enabled > > systems? > > We would definitely not ... and my suspicion is that that is the largest > single component of the available speedup. OK, what is 'install' doing for us that 'cp' and 'chmod' would not already do? The following patch reduces the 'gmake install' in /src/include from 8 seconds to 0.8 seconds. Oh, and for testing I use a sourceforge shell farm account, and a HP testdrive shell acount that gives me access to almost every operating system. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073 Index: src/include/Makefile =================================================================== RCS file: /cvsroot/pgsql/src/include/Makefile,v retrieving revision 1.19 diff -c -c -r1.19 Makefile *** src/include/Makefile 6 Jan 2005 21:00:24 -0000 1.19 --- src/include/Makefile 23 Mar 2005 20:22:56 -0000 *************** *** 41,49 **** $(INSTALL_DATA) $$file $(DESTDIR)$(includedir_server)/`basename $$file` || exit; \ done for dir in $(SUBDIRS); do \ ! for file in $(srcdir)/$$dir/*.h; do \ ! $(INSTALL_DATA) $$file $(DESTDIR)$(includedir_server)/$$dir/`basename $$file` || exit; \ ! done \ done installdirs: --- 41,48 ---- $(INSTALL_DATA) $$file $(DESTDIR)$(includedir_server)/`basename $$file` || exit; \ done for dir in $(SUBDIRS); do \ ! cp $(srcdir)/$$dir/*.h $(DESTDIR)$(includedir_server)/$$dir/ || exit; \ ! chmod 644 $(DESTDIR)$(includedir_server)/$$dir/*.h || exit; \ done installdirs:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> OK, what is 'install' doing for us that 'cp' and 'chmod' would not
> already do?
Quite a lot of things, such as coping with busy target files --- not too
important for headers, but very important for executables and shlibs.
We might be able to get away with this for just the headers, though,
and that's certainly the bulk of the install work now.
regards, tom lane
Alvaro Herrera wrote: > Also, keep in my that this C install program has the extra feature of > being able to install multiple files on one invocation, per suggestion > from Tom Lane. This allows us to save the nested for-loop in > src/include/Makefile. GNU install (available on my system) also has > this capability, but would we be able to use the trick if we had to > cater for the lowest common denominator found on other, non-GNU-enabled > systems? Why not check for GNU install in configure, and only enable the optimization if it is available? -Neil
Tom Lane wrote: > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > OK, what is 'install' doing for us that 'cp' and 'chmod' would not > > already do? > > Quite a lot of things, such as coping with busy target files --- not too > important for headers, but very important for executables and shlibs. > > We might be able to get away with this for just the headers, though, > and that's certainly the bulk of the install work now. Here is my next version of the patch that uses 'cp' and 'chmod' to install multiple header files rather than 'install'. I moved the file modes into variables so any changes are propogated to src/include/Makefile. This is 20 times faster than what we have now, 8 seconds vs 0.40 seconds. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073 Index: src/Makefile.global.in =================================================================== RCS file: /cvsroot/pgsql/src/Makefile.global.in,v retrieving revision 1.212 diff -c -c -r1.212 Makefile.global.in *** src/Makefile.global.in 25 Mar 2005 18:17:12 -0000 1.212 --- src/Makefile.global.in 25 Mar 2005 22:58:33 -0000 *************** *** 232,240 **** INSTALL = $(SHELL) $(top_srcdir)/config/install-sh -c INSTALL_PROGRAM = $(INSTALL_PROGRAM_ENV) $(INSTALL) $(INSTALL_STRIP_FLAG) ! INSTALL_SCRIPT = $(INSTALL) -m 755 ! INSTALL_DATA = $(INSTALL) -m 644 INSTALL_STLIB = $(INSTALL_STLIB_ENV) $(INSTALL_DATA) $(INSTALL_STRIP_FLAG) INSTALL_SHLIB = $(INSTALL_SHLIB_ENV) $(INSTALL) $(INSTALL_SHLIB_OPTS) $(INSTALL_STRIP_FLAG) # Override in Makefile.port if necessary --- 232,242 ---- INSTALL = $(SHELL) $(top_srcdir)/config/install-sh -c + INSTALL_SCRIPT_MODE = 755 + INSTALL_DATA_MODE = 644 INSTALL_PROGRAM = $(INSTALL_PROGRAM_ENV) $(INSTALL) $(INSTALL_STRIP_FLAG) ! INSTALL_SCRIPT = $(INSTALL) -m $(INSTALL_SCRIPT_MODE) ! INSTALL_DATA = $(INSTALL) -m $(INSTALL_DATA_MODE) INSTALL_STLIB = $(INSTALL_STLIB_ENV) $(INSTALL_DATA) $(INSTALL_STRIP_FLAG) INSTALL_SHLIB = $(INSTALL_SHLIB_ENV) $(INSTALL) $(INSTALL_SHLIB_OPTS) $(INSTALL_STRIP_FLAG) # Override in Makefile.port if necessary Index: src/include/Makefile =================================================================== RCS file: /cvsroot/pgsql/src/include/Makefile,v retrieving revision 1.19 diff -c -c -r1.19 Makefile *** src/include/Makefile 6 Jan 2005 21:00:24 -0000 1.19 --- src/include/Makefile 25 Mar 2005 22:58:37 -0000 *************** *** 37,49 **** # These headers are needed for server-side development $(INSTALL_DATA) pg_config.h $(DESTDIR)$(includedir_server) $(INSTALL_DATA) pg_config_os.h $(DESTDIR)$(includedir_server) ! for file in $(srcdir)/*.h; do \ ! $(INSTALL_DATA) $$file $(DESTDIR)$(includedir_server)/`basename $$file` || exit; \ ! done for dir in $(SUBDIRS); do \ ! for file in $(srcdir)/$$dir/*.h; do \ ! $(INSTALL_DATA) $$file $(DESTDIR)$(includedir_server)/$$dir/`basename $$file` || exit; \ ! done \ done installdirs: --- 37,48 ---- # These headers are needed for server-side development $(INSTALL_DATA) pg_config.h $(DESTDIR)$(includedir_server) $(INSTALL_DATA) pg_config_os.h $(DESTDIR)$(includedir_server) ! # We don't use INSTALL_DATA for performance reasons --- there are a lot of files ! cp $(srcdir)/*.h $(DESTDIR)$(includedir_server)/ || exit; \ ! chmod $(INSTALL_DATA_MODE) $(DESTDIR)$(includedir_server)/*.h || exit; \ for dir in $(SUBDIRS); do \ ! cp $(srcdir)/$$dir/*.h $(DESTDIR)$(includedir_server)/$$dir/ || exit; \ ! chmod $(INSTALL_DATA_MODE) $(DESTDIR)$(includedir_server)/$$dir/*.h || exit; \ done installdirs:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Tom Lane wrote:
>> We might be able to get away with this for just the headers, though,
>> and that's certainly the bulk of the install work now.
> Here is my next version of the patch that uses 'cp' and 'chmod' to
> install multiple header files rather than 'install'.
> I moved the file modes into variables so any changes are propogated to
> src/include/Makefile.
> This is 20 times faster than what we have now, 8 seconds vs 0.40 seconds.
I like it. Given that we are installing into directories we'd have just
created ourselves, which should have no extraneous files in them, a lot
of the objections one might raise go away.
Why don't you go ahead and apply that to HEAD for the time being, even
if we end up replacing it later? Might as well start saving developers'
time immediately.
regards, tom lane
Tom Lane wrote: > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > Tom Lane wrote: > >> We might be able to get away with this for just the headers, though, > >> and that's certainly the bulk of the install work now. > > > Here is my next version of the patch that uses 'cp' and 'chmod' to > > install multiple header files rather than 'install'. > > > I moved the file modes into variables so any changes are propogated to > > src/include/Makefile. > > > This is 20 times faster than what we have now, 8 seconds vs 0.40 seconds. > > I like it. Given that we are installing into directories we'd have just > created ourselves, which should have no extraneous files in them, a lot > of the objections one might raise go away. > > Why don't you go ahead and apply that to HEAD for the time being, even > if we end up replacing it later? Might as well start saving developers' > time immediately. Done. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
I am using 'cp' in current CVS for this. I assume it is as fast as the C implementation, but if not, please let me know. --------------------------------------------------------------------------- Alvaro Herrera wrote: > Patchers, > > I wrote an "install" program in C. It's supposed to replace the > config/install-sh script, limited to the functionality we need, i.e. > what is in Makefiles in the Pg main source tree. The main objective of > this exercise is to reduce "make install" execution time; a part of that > is being able to install multiple files with one command. > > Portability testing right now is limited to my machine, which is Linux > with glibc 2.3.2 (Debian Sid). > > With this in place, "make install" in src/include takes 17 seconds on > this machine, where the script version takes more than a minute. I > think this is a useful improvement. > > > Right now I'm missing a Makefile rule for it. It needs the pg_progname > symbol from src/port, and the includes from $(srcdir)/src/include and > $(builddir)/src/include. From the config directory, this works: > > $(CC) $(CFLAGS) -I$(top_srcdir)/src/include -I$(top_builddir)/src/include ../src/port/path.o ../src/port/exec.o $< -o $@ > > Also, I don't know how to force it to build before executing "install", > short of putting it as a dependency in every makefile (which sounds > silly to me). > > I attach the source code, and a src/include/Makefile modification to > use the multi-file install feature. > > -- > Alvaro Herrera (<alvherre[a]dcc.uchile.cl>) > "Hay quien adquiere la mala costumbre de ser infeliz" (M. A. Evans) [ Attachment, skipping... ] [ Attachment, skipping... ] > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Don't 'kill -9' the postmaster -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
On Sat, Jun 04, 2005 at 08:42:29PM -0400, Bruce Momjian wrote: > > I am using 'cp' in current CVS for this. I assume it is as fast as the > C implementation, but if not, please let me know. Yes, your change did make the installation considerably faster so I abandoned the binary install idea. Thanks. -- Alvaro Herrera (<alvherre[@]dcc.uchile.cl>) "Lo esencial es invisible para los ojos" (A. de Saint Exúpery)