Thread: Development installation fails
For some reason I can never properly install a build from the development sources. I have always avoided it, but now I must. I don't have the very latest sources, but it seems I am doing something fundamentally wrong here because all hell usually breaks loose when I try. So perhaps someone that does this all the time can give me a few hints here. I have a 6.5.* installation in /usr/local/pgsql (from /usr/src/pgsql). /usr/local/pgsql is in the path, /usr/local/pgsql/lib is in ld.config. (Linux system). The development sources I have in /home/peter/pgsql and I want to install into /home/peter/postgres-cur. That's the setup. ~/pgsql/src$ ./configure --prefix=/home/peter/postgres-cur works fine. ~/pgsql/src$ make bombs out somewhere in backend/optimizer with internal compiler error (gcc 2.8.1) and/or weird make file complaints (GNU make 3.76.1). Try again with egcs 2.91.66 works. ~/pgsql/src$ make install ~/postgres-cur/bin$ ./initdb --pglib=/home/peter/postgres-cur/lib \ --pgdata=/home/peter/postgres-cur/data We are initializing the database system with username peter (uid=500). This user will own all the files and must also own the server process. Creating Postgres database system directory /home/peter/postgres-cur/data Creating Postgres database system directory /home/peter/postgres-cur/data/base Creating Postgres database XLOG directory /home/peter/postgres-cur/data/pg_xlog Creating template database in /home/peter/postgres-cur/data/base/template1 -boot: invalid option -- x Usage: postgres -boot [-d] [-C] [-F] [-O] [-Q] [-P portno] [dbName] d: debug mode C: disable version checking F:turn off fsync O: set BootstrapProcessing mode P portno: specify port number initdb: could not create template database Okay, that's the first problem. So I removed that -x flag out of initdb (look for shell variable FIRSTRUN). Next try: ~/postgres-cur/bin$ ./initdb --pglib=/home/peter/postgres-cur/lib \ --pgdata=/home/peter/postgres-cur/data We are initializing the database system with username peter (uid=500). This user will own all the files and must also own the server process. Creating template database in /home/peter/postgres-cur/data/base/template1 syntax error 2370 : parse error Creating global classes in /home/peter/postgres-cur/data/base Adding template1 database to pg_database... Vacuuming template1 Creating public pg_user view Creating view pg_rules Creating view pg_views Creating view pg_tables Creating view pg_indexes Despite the error things seem to have been completed fine. ~/postgres-cur/bin$ ./postmaster -p 6543 -D /home/peter/postgres-cur/data/ Database system in directory /home/peter/postgres-cur/data/ is not compatible with this version of Postgres, or we are unable to read the PG_VERSION file. Explanation from ValidatePgVersion: Version number in file '/home/peter/postgres-cur/data//PG_VERSION' should be 7.0, not 6.5. No data directory -- can't proceed. Okay, so I change all the 6.5's to 7.0's by hand and try again. ~/postgres-cur/bin$ ./postmaster -p 6543 -D /home/peter/postgres-cur/data DEBUG: Data Base System is starting up at Wed Nov 24 17:58:40 1999 FATAL 2: Open(cntlfile) failed: 2 FATAL 2: Open(cntlfile) failed: 2 Startup failed - abort At this point I thought I'd better stop messing around and ask what's going on here. Probably the paths of my "production" installation and the development installation interfere with each other, but there must be a way I can do this in a simple fashion and without affecting my running database. Thanks,Peter -- Peter Eisentraut Sernanders väg 10:115 peter_e@gmx.net 75262 Uppsala http://yi.org/peter-e/ Sweden
Peter Eisentraut <peter_e@gmx.net> writes: > For some reason I can never properly install a build from the development It looks to me like you have a conflict against a production installation on the same machine. It is *not sufficient* to install into an alternate directory; you must also select an alternate port number. See the new 'run_check.sh' regress test driver for a complete example. Jan reported recently that there were places that didn't pay attention to the -D switch, which is a bug that will likely bite you if you are trying to install into a directory other than the configured-in one. However that doesn't seem to be the issue here. Personally I build test versions with ./configure --with-pgport=5440 --prefix=/users/postgres/testversion (adapt port and prefix to taste, of course) and then I don't have to worry about remembering to do the install specially. I dunno about the other problems you are seeing. I run a full build and install from scratch at least a couple times a week, just to be sure that current sources are not broken. They weren't as of ... hmm ... 20:55 EST 22-Nov was my last CVS pull. Anyway: > ~/pgsql/src$ make > bombs out somewhere in backend/optimizer with internal compiler error (gcc > 2.8.1) and/or weird make file complaints (GNU make 3.76.1). Try again with > egcs 2.91.66 works. Hmm. I have been using make 3.76.1 right along with no problems. What were the make complaints again? Also, I'm still running gcc 2.7.2.2, so I'm a bit surprised to hear that 2.8.1 crashes. (Or maybe not ... there's a reason I never updated to 2.8.* ...) > ~/postgres-cur/bin$ ./initdb --pglib=/home/peter/postgres-cur/lib \ > --pgdata=/home/peter/postgres-cur/data You may need to have env variables PGDATA and PGLIB set during initdb (at least, the install docs recommended that last I looked) and you definitely need to have /home/peter/postgres-cur/bin in your PATH. It looks to me like initdb.sh may need to do "export PGDATA PGLIB" to make sure that whatever values it gets from command line switches are passed down to the programs it invokes... the PATH is just something you have to get right... > Creating template database in /home/peter/postgres-cur/data/base/template1 > -boot: invalid option -- x I think you were already invoking the wrong version of postgres here; the current sources do support -x (although bootstrap.c's usage() neglects to list it). If it wasn't listed in your PATH ahead of the 6.5 version, that's your problem right there. regards, tom lane
On 1999-11-24, Tom Lane mentioned: > > ~/postgres-cur/bin$ ./initdb --pglib=/home/peter/postgres-cur/lib \ > > --pgdata=/home/peter/postgres-cur/data > > You may need to have env variables PGDATA and PGLIB set during initdb > (at least, the install docs recommended that last I looked) and you > definitely need to have /home/peter/postgres-cur/bin in your PATH. Okay, thanks for the tip. I thought about this for a while and came to the conclusion, that this is not only the most likely reason for the oft criticized installation (non-)simplicity, it is also cumbersome and not acceptable to have to set environment variables (especially PATH) during installation. This would also mean that I would have to yank the path away from my production installation. So I thunk that initdb could very well find out itself where it is located and then call ${mydir}/postgres explicitly. That solves the path problem. Secondly, it could also make educated guesses where the PGLIB is at (namely ${mydir}/../lib, or ${mydir}/../lib/pgsql as in the RPMs). That solves the other problem. I included a patch that does exactly that, so now I can do: $ ./configure --prefix=/any/path/here $ make $ make install $ /any/path/here/bin/initdb -D /some/other/path #(*) $ /any/path/here/bin/postmaster -D /some/other/path and I'm up and running. (No PG* env var or special PATH is set.) I'm not kidding, this is exactly what I did (well, different path names) and I'm in business. Okay, there are some other glitches with make install and initdb being very reluctant to creating directories themselves, but that could be fixed in another round of changes. So please examine that patch. It was a very quick hack, so it's not very refined. Let me know if this sounds good, then I'll put the finishing touches on it. -Peter (*) - Notice that I changed the option to -D (formerly -r). This goes more nicely with the equivalent postmaster option, and also with "_D_at's where I want my data to be stored" :) -- Peter Eisentraut Sernanders väg 10:115 peter_e@gmx.net 75262 Uppsala http://yi.org/peter-e/ Sweden
Peter Eisentraut <peter_e@gmx.net> writes: > On 1999-11-24, Tom Lane mentioned: >> You may need to have env variables PGDATA and PGLIB set during initdb >> (at least, the install docs recommended that last I looked) and you >> definitely need to have /home/peter/postgres-cur/bin in your PATH. > Okay, thanks for the tip. I thought about this for a while and came to the > conclusion, that this is not only the most likely reason for the oft > criticized installation (non-)simplicity, it is also cumbersome and not > acceptable to have to set environment variables (especially PATH) during > installation. I never much cared for it either. If you can get rid of it, great! > So I thunk that initdb could very well find out itself where it is located > and then call ${mydir}/postgres explicitly. That solves the path problem. > Secondly, it could also make educated guesses where the PGLIB is at > (namely ${mydir}/../lib, or ${mydir}/../lib/pgsql as in the RPMs). Reasonable, though you should provide a command line switch to override the guess about PGLIB. Possibly initdb should emit a message like "Assuming --pglib=/foo/bar/baz" if it's not given a switch. > So please examine that patch. It was a very quick hack, so it's not very > refined. Let me know if this sounds good, then I'll put the finishing > touches on it. I'm not convinced your "which $0" implementation for finding BINDIR is portable/reliable. On my system, man which says that it determines aliases and path by reading ~/.cshrc, which has got obvious problems if I'm not a csh user. My references say that "which" isn't available on all systems anyway. It'd probably be a good idea to verify that $BINDIR/postgres exists after you think you have the value. BTW, seems like doing PATH=$BINDIR:$PATH in the script would be a lot easier than hacking all the references to programs --- and it covers the possibility that one of the invoked programs tries to call another. The other bit of environment state that initdb currently depends on is USER. This is a big risk factor IMHO, since it won't be right if you are su'd to the postgres account. Can you add code to verify that it is set (or set it if not) and that it matches the actual ownership of the process? regards, tom lane
On Sun, 28 Nov 1999, Tom Lane wrote: > I'm not convinced your "which $0" implementation for finding BINDIR is > portable/reliable. On my system, man which says that it determines > aliases and path by reading ~/.cshrc, which has got obvious problems if > I'm not a csh user. My references say that "which" isn't available on > all systems anyway. It'd probably be a good idea to verify that > $BINDIR/postgres exists after you think you have the value. I did a little "which" survey, it seems you're right. On GNU/Linux systems "which" is a binary which does the seemingly right thing. Under bash "which" is also often aliased to "type -path". That led me to believe that the sh built-in "type" might do, but its output format is rather unpredicable. On FreeBSD "which" is a Perl script, which seems to work fine. On Solaris and SGI the problems you pointed out seem to be present, as "which" is actually implemented as a csh script. However, on the particular systems I surveyed, the sysadmins were smart enough to provide working versions (either the GNU or the FreeBSD variant). To make a long story short, using the implementation I suggested with the checks you suggested would probably benefit 90% of our users. > BTW, seems like doing PATH=$BINDIR:$PATH in the script would be a lot > easier than hacking all the references to programs --- and it covers > the possibility that one of the invoked programs tries to call another. I'm really hesitant to changing the path, even if only temporarily. Will ponder. > The other bit of environment state that initdb currently depends on is > USER. This is a big risk factor IMHO, since it won't be right if you > are su'd to the postgres account. Can you add code to verify that it > is set (or set it if not) and that it matches the actual ownership of > the process? Yes, I noticed that too. Again, I really don't think that the script should set USER. If the user chose to unset it, they might have had a reason for it. The same happened in the createdb, etc. scripts and in those cases there wasn't even a point for knowing the user at all, AFAI recall. Seems like cleaning out the logic of initdb in general is a good idea, so I'll work on that. -- Peter Eisentraut Sernanders vaeg 10:115 peter_e@gmx.net 75262 Uppsala http://yi.org/peter-e/ Sweden
Peter Eisentraut <e99re41@csd.uu.se> writes: > On Sun, 28 Nov 1999, Tom Lane wrote: >> I'm not convinced your "which $0" implementation for finding BINDIR is >> portable/reliable. > To make a long story short, using the implementation I suggested with the > checks you suggested would probably benefit 90% of our users. And fail entirely for the other 10%? Not good enough if so :-( ... the idea is to make install easier not harder. How much code would it take to emulate as much of "which" as we need, do you think? What's our fallback position if it doesn't work? >> The other bit of environment state that initdb currently depends on is >> USER. > Yes, I noticed that too. Again, I really don't think that the script > should set USER. After thinking about it for a while, I think that there shouldn't be any dependency on USER, period. initdb (and anything else that cares) ought to get the name of the user they are executing as, and use that. I can't see any good reason why the name inserted into the databases should be potentially different from the ownership of the files. Is 'whoami' a portable way of getting the current user id, or not? The only reference I have about portable shell programming says that the POSIX-approved command for this is 'id -u -n', and that 'whoami' is a BSD-ism. I've got doubts that either one is universal ... we might have to try both. Grumble. regards, tom lane
On 1999-11-29, Tom Lane mentioned: > > To make a long story short, using the implementation I suggested with the > > checks you suggested would probably benefit 90% of our users. > > And fail entirely for the other 10%? Not good enough if so :-( ... the > idea is to make install easier not harder. How much code would it take > to emulate as much of "which" as we need, do you think? What's our > fallback position if it doesn't work? Okay, now you're putting words into my mouth :) The fallback would be not needing "which" because initdb is called with an explicit path (which is fairly likely during installation, unless you adjust your path ahead of time) and also behaviour as it is now (always a good idea). To reproduce "which" one would need to do some sort of split on PATH, which would probably require awk and we're trying to avoid that. (And I don't know awk, but that's just an excuse, no reason.) > After thinking about it for a while, I think that there shouldn't be any > dependency on USER, period. initdb (and anything else that cares) ought > to get the name of the user they are executing as, and use that. I > can't see any good reason why the name inserted into the databases > should be potentially different from the ownership of the files. One potential goal (which I personally share) of simplifying the installation process would be to not have to su as postgres but do everything as root or a user of your choice. Together with Vince's idea of adding -o and -g options to the install command and a similar option to initdb, we can do that and it would not be hard to understand from an end user's point of view. What I don't like is that certain scripts don't find the USER variable set and then set it themselves. The psql wrapper scripts do that, but I'm cleaning that up. > Is 'whoami' a portable way of getting the current user id, or not? > The only reference I have about portable shell programming says that > the POSIX-approved command for this is 'id -u -n', and that 'whoami' > is a BSD-ism. I've got doubts that either one is universal ... we might > have to try both. Grumble. (The psql wrapper scripts use "whoami".) Once you start wondering in this direction you might soon start noticing that chances are that shell scripts cannot reliably use *any* external program. The exceptions might be the handful listed in the GNU makefile standard, because as we are using autoconf we could assume that they are present. We have already taken one step into the direction of providing our own sh utils collection by way of pg_id (which is also a subset of what "id" does). We could extend that and provide pg_which and whatever else we need (of course pg_which will create a chicken and egg problem). But that could get out of hand and won't help developers either. One thing I am missing from this project that could really help is a goal of what sort of operating system it is we are trying to support. In the beginning the answer was clearly "BSD". Nowadays I believe everyone would agree that it should be POSIX. In practice it is probably safer to say POSIX to the extend to which it is supported by the most popular operating systems. That doesn't mean others should be locked out, but then *they* should have to make the adjustments. That means if all reasonable systems have "id" then the others will have to get a substitute. This also applies to C source code: If all reasonable systems have CONSTANT defined, then others will have to define it themselves (possibly assisted via autoconf), but you should not expect your developers to remember every single constant you invented yourself because of some obscure conflict. (Yes, I'm also alluding to PATH_MAX, but it goes further to C functions etc. Recall the postmaster SSL switch.) But most importantly of all you should not reject useful extensions because of strange incompatibilities. We can easily steal the relevant working utilities from some *BSD* code and provide them as an extra tar ball. "To use PostgreSQL, you need certain basic 'shell utilility' programs, which you do not seem to have installed. Please grab the package pgsql-shutils.tar.gz from the PostgreSQL ftp server. You'll be eternally grateful." Or even include them in the main tarball, if they aren't too large. I think the best we could possibly do at this point is to do a survey of operating systems we support and which are mostly used and see what's really on there and what POSIX says and then use that as a frame of reference. Those OS' should probably include *BSD, BSDI, Linux (various distros), Solaris, SGI, HPUX, Cygwin, (shoot me if I left yours off). I think the developer circle does have access to all of those. If not, then it's potentially pointless to *try* to account for them, if we really can't anyway. This whole portability discussion needs to have a reference. We can't just always say "I think it's not portable." We need to have a way to look it up. Having said that, I'll continue to mess around with initdb on the assumption that nothing works and we'll see what comes out of it. Thanks for reading all the way down to here anyway. -Peter -- Peter Eisentraut Sernanders väg 10:115 peter_e@gmx.net 75262 Uppsala http://yi.org/peter-e/ Sweden
Peter Eisentraut <peter_e@gmx.net> writes: > To reproduce "which" one would need to do some sort of split on PATH, > which would probably require awk and we're trying to avoid that. No, that part is easy: for dir in `echo $PATH | sed 's/:/ /g'` ; do if [ -x $dir/initdb ]; then ... I think the harder part is manipulating $0 to see if it contains a path part or not --- not all systems have dirname, so you need to fake it with a sed pattern. See the configure script for the standard way. (BTW, I see the new run_check.sh driver depends on dirname ... boo hiss.) > One potential goal (which I personally share) of simplifying the > installation process would be to not have to su as postgres but do > everything as root or a user of your choice. Together with Vince's idea of > adding -o and -g options to the install command and a similar option to > initdb, we can do that and it would not be hard to understand from an end > user's point of view. Huh? The user of your choice *is* postgres, or whoever you are su'd as. -o and -g are useless unless you are executing the install as root, which really isn't necessary --- in fact I think we ought to discourage it to prevent people from accidentally installing the postgres files with root ownership. (initdb ought to refuse to run at all as root...) > What I don't like is that certain scripts don't find > the USER variable set and then set it themselves. The psql wrapper scripts > do that, but I'm cleaning that up. Well, if you don't like setting USER I don't have a problem with using a different variable name instead. My point is that there is no good reason to allow the value to be different from the actual effective UID, so we should be working from that and not environment variables at all. > One thing I am missing from this project that could really help is a goal > of what sort of operating system it is we are trying to support. In the > beginning the answer was clearly "BSD". Nowadays I believe everyone would > agree that it should be POSIX. POSIX, BSD, and anything else that anyone in the developer + beta tester population is willing to test/support. In practice, as long as we are pretty conservative about what we assume is in POSIX or BSD (eg, use POSIX.1 but not POSIX.2 stuff), I don't think it's a big deal. regards, tom lane
On Tue, 30 Nov 1999, Tom Lane wrote: > No, that part is easy: > > for dir in `echo $PATH | sed 's/:/ /g'` ; do > if [ -x $dir/initdb ]; then > ... Cool. > I think the harder part is manipulating $0 to see if it contains a path > part or not --- not all systems have dirname, so you need to fake it > with a sed pattern. See the configure script for the standard way. > (BTW, I see the new run_check.sh driver depends on dirname ... boo hiss.) You see, it just gets worse and worse as you start digging ... :( No, seriously: *which* system doesn't have dirname? Does it have basename? > Huh? The user of your choice *is* postgres, or whoever you are su'd as. > -o and -g are useless unless you are executing the install as root, > which really isn't necessary --- in fact I think we ought to discourage > it to prevent people from accidentally installing the postgres files > with root ownership. (initdb ought to refuse to run at all as root...) There is a very good reason for running the installation process as any user you choose, because the usual sequence ./configure make make install will fail to create the installation directory (/usr/local/pgsql). That blows. Then you have to su as root and create it and then go back and continue. Then you notice you made the directory but forgot to change the ownership to postgres. Next time you forget that altogether and then all your files are owned by root. This is exactly what makes the INSTALL so long. I have always resorted to installing everything as root and then doing a chown --recursive (talk about non-portable ;) ), that was much more convenient. I'm not sure about the terminally convenient and flexible way yet either, though. But at least we're agreed that all this environment variable stuff needs to go away. -- Peter Eisentraut Sernanders vaeg 10:115 peter_e@gmx.net 75262 Uppsala http://yi.org/peter-e/ Sweden
On Tue, 30 Nov 1999, Tom Lane wrote: > Peter Eisentraut <peter_e@gmx.net> writes: > > One potential goal (which I personally share) of simplifying the > > installation process would be to not have to su as postgres but do > > everything as root or a user of your choice. Together with Vince's idea of > > adding -o and -g options to the install command and a similar option to > > initdb, we can do that and it would not be hard to understand from an end > > user's point of view. > > Huh? The user of your choice *is* postgres, or whoever you are su'd as. > -o and -g are useless unless you are executing the install as root, > which really isn't necessary --- in fact I think we ought to discourage > it to prevent people from accidentally installing the postgres files > with root ownership. (initdb ought to refuse to run at all as root...) Perhaps the user of choice for running PostgreSQL is postgres, but that's not necessarily the same choice for the installing user. If you happen to install it as postgres then the -o and -g will have no effect on you, but if root is installing it then you want the -o and -g in there. No? I do agree, tho, that initdb should only be able to run as the database superuser as stated at configuration time. Vince. -- ========================================================================== Vince Vielhaber -- KA8CSH email: vev@michvhf.com flame-mail: /dev/null # include <std/disclaimers.h> Have you seenhttp://www.pop4.net? Online Campground Directory http://www.camping-usa.com Online Giftshop Superstore http://www.cloudninegifts.com ==========================================================================
Peter Eisentraut <e99re41@DoCS.UU.SE> writes: > There is a very good reason for running the installation process as any > user you choose, Not "as any user you choose", but as *root* --- the can't-make-top-directory problem can only be solved by root, and on many systems only root can chown files to another account anyway. The difficulty with encouraging people to su to root for install is that it's so easy to make the files root-owned and thereby create a security problem. Perhaps the right compromise is to add a --owner switch to "make install", and to have it refuse to install if the (given or defaulted) ownership is "root" ? Actually, something I'm not real clear on right at the moment is whether it is safe to make the dirs/files created by install be root-owned or not. The ownership of the data directory and everything below it must be postgres (s/postgres/your unprivileged user of choice/g) because the running postgres process must be able to write in those dirs. But offhand I can't think of any reason that any postgres-owned processes need to be able to write in the bin, lib, or include hierarchies. Can anyone else think of one? Maybe the proper approach is "OK, do make install as root if you feel like it; we don't care whether that stuff is root-owned". Only initdb would need to have a --owner switch and take care that the files/dirs it makes are not left root-owned. Then the install sequence could look like ./configure make su root make install initdb --owner=postgres --pgdata=whatever exit from su start postmaster BTW, do we have a check in the postmaster to refuse to start if its euid is root? Shouldn't we? regards, tom lane PS: > No, seriously: *which* system doesn't have dirname? Does it have basename? Horton says that dirname was originally SysV and was standardized in POSIX.1. I'd expect it to be present on most systems these days, but probably there are still some old BSD machines that ain't got it. He rates basename as fully portable though. Basically my take on this stuff is to conform to existing GNU portability practices except where we have very good reason to do otherwise. It's not that hard to use the GNU workaround for dirname, so why not do it...
On Tue, 30 Nov 1999, Tom Lane wrote: > ./configure > make > su root > make install > initdb --owner=postgres --pgdata=whatever > exit from su > start postmaster ./configure --with-pguser=postgres make sudo make install is what I've been pushing for. That way when you do the installation it'll happen something like /usr/bin/install -c -d -g postgres -o postgres FILE-TO-BE-INSTALLED the -c copies, the -d creates the missing directories, different install programs do things differently. The -g and -o come from the configure switch --with-pguser and maybe even --with-pggroup. The defaults should be postgres for both. initdb can then have those values built in, yet overridable. Vince. -- ========================================================================== Vince Vielhaber -- KA8CSH email: vev@michvhf.com flame-mail: /dev/null # include <std/disclaimers.h> Have you seenhttp://www.pop4.net? Online Campground Directory http://www.camping-usa.com Online Giftshop Superstore http://www.cloudninegifts.com ==========================================================================
On Tue, 30 Nov 1999, Tom Lane wrote: > The difficulty with encouraging people to su to root for install is that > it's so easy to make the files root-owned and thereby create a security > problem. Perhaps the right compromise is to add a --owner switch to > "make install", and to have it refuse to install if the (given or > defaulted) ownership is "root" ? See Vince's email about the configure switch to be used in install. That is what I was shooting for. I am not sure to what extend initdb should use those settings (recall: autoconf is not for configuring run time stuff) but if you *insist* on running initdb as root (too lazy to su, forgot to, etc.) there should be an option, as there is now. > offhand I can't think of any reason that any postgres-owned processes > need to be able to write in the bin, lib, or include hierarchies. Can > anyone else think of one? They better not write there. That would certainly be a major bug. > BTW, do we have a check in the postmaster to refuse to start if its euid > is root? Shouldn't we? There is a check and it refuses to start. -- Peter Eisentraut Sernanders vaeg 10:115 peter_e@gmx.net 75262 Uppsala http://yi.org/peter-e/ Sweden