Thread: initdb error
I'm running into the following error message when running initdb (FreeBSD host): ygg# /usr/local/etc/rc.d/postgresql initdb -D /zdb/pgsql/data --debug The files belonging to this database system will be owned by user "pgsql". This user must also own the server process. The database cluster will be initialized with locales COLLATE: C CTYPE: en_US.UTF-8 MESSAGES: en_US.UTF-8 MONETARY: en_US.UTF-8 NUMERIC: en_US.UTF-8 TIME: en_US.UTF-8 The default text search configuration will be set to "english". creating directory /zdb/pgsql/data ... ok creating subdirectories ... ok selecting default max_connections ... 100 selecting default shared_buffers ... 32MB creating configuration files ... ok creating template1 database in /zdb/pgsql/data/base/1 ... FATAL: could not open file "pg_xlog/000000010000000000000001" (log file 0, segment 1): No such file or directory child process exited with exit code 1 initdb: removing data directory "/zdb/pgsql/data" My best guess is that it has something to do with permissions, but I really have no idea. Has anyone seen this before and found a way around it? -David
On 12/13/2012 07:38 AM, David Noel wrote: > I'm running into the following error message when running initdb (FreeBSD host): > > ygg# /usr/local/etc/rc.d/postgresql initdb -D /zdb/pgsql/data --debug > The files belonging to this database system will be owned by user "pgsql". > This user must also own the server process. > > The database cluster will be initialized with locales > COLLATE: C > CTYPE: en_US.UTF-8 > MESSAGES: en_US.UTF-8 > MONETARY: en_US.UTF-8 > NUMERIC: en_US.UTF-8 > TIME: en_US.UTF-8 > The default text search configuration will be set to "english". > > creating directory /zdb/pgsql/data ... ok > creating subdirectories ... ok > selecting default max_connections ... 100 > selecting default shared_buffers ... 32MB > creating configuration files ... ok > creating template1 database in /zdb/pgsql/data/base/1 ... FATAL: > could not open file "pg_xlog/000000010000000000000001" (log file 0, > segment 1): No such file or directory > child process exited with exit code 1 > initdb: removing data directory "/zdb/pgsql/data" > > My best guess is that it has something to do with permissions, but I > really have no idea. Has anyone seen this before and found a way > around it? Not quite what is in the the init script in rc.d, have you tried running the initdb command directly? As you say I believe there are permissions problems. So following the instructions found below may solve your problem: http://www.postgresql.org/docs/9.2/interactive/app-initdb.html > > -David > > -- Adrian Klaver adrian.klaver@gmail.com
On 12/13/12, Adrian Klaver <adrian.klaver@gmail.com> wrote: > On 12/13/2012 07:38 AM, David Noel wrote: >> I'm running into the following error message when running initdb (FreeBSD >> host): >> >> ygg# /usr/local/etc/rc.d/postgresql initdb -D /zdb/pgsql/data --debug >> The files belonging to this database system will be owned by user >> "pgsql". >> This user must also own the server process. >> >> The database cluster will be initialized with locales >> COLLATE: C >> CTYPE: en_US.UTF-8 >> MESSAGES: en_US.UTF-8 >> MONETARY: en_US.UTF-8 >> NUMERIC: en_US.UTF-8 >> TIME: en_US.UTF-8 >> The default text search configuration will be set to "english". >> >> creating directory /zdb/pgsql/data ... ok >> creating subdirectories ... ok >> selecting default max_connections ... 100 >> selecting default shared_buffers ... 32MB >> creating configuration files ... ok >> creating template1 database in /zdb/pgsql/data/base/1 ... FATAL: >> could not open file "pg_xlog/000000010000000000000001" (log file 0, >> segment 1): No such file or directory >> child process exited with exit code 1 >> initdb: removing data directory "/zdb/pgsql/data" >> >> My best guess is that it has something to do with permissions, but I >> really have no idea. Has anyone seen this before and found a way >> around it? > > > Not quite what is in the the init script in rc.d, have you tried running > the initdb command directly? As you say I believe there are permissions > problems. So following the instructions found below may solve your problem: > > http://www.postgresql.org/docs/9.2/interactive/app-initdb.html > > > >> >> -David >> >> > > > -- > Adrian Klaver > adrian.klaver@gmail.com > I've tried initdb directly: initdb -D /zdb/pgsql/data ...and also through pg_ctl: pg_ctl initdb -D /zdb/pgsql/data ...and still seem to wind up with the error: creating template1 database in /zdb/pgsql/data/base/1 ... FATAL: could not open file "pg_xlog/000000010000000000000001" (log file 0, segment 1): No such file or directory -David
On 12/13/2012 08:18 AM, David Noel wrote: > On 12/13/12, Adrian Klaver <adrian.klaver@gmail.com> wrote: >> On 12/13/2012 07:38 AM, David Noel wrote: >>> I'm running into the following error message when running initdb (FreeBSD >>> host): >>> >>> ygg# /usr/local/etc/rc.d/postgresql initdb -D /zdb/pgsql/data --debug >>> The files belonging to this database system will be owned by user >>> "pgsql". >>> This user must also own the server process. >>> >>> The database cluster will be initialized with locales >>> COLLATE: C >>> CTYPE: en_US.UTF-8 >>> MESSAGES: en_US.UTF-8 >>> MONETARY: en_US.UTF-8 >>> NUMERIC: en_US.UTF-8 >>> TIME: en_US.UTF-8 >>> The default text search configuration will be set to "english". >>> >>> creating directory /zdb/pgsql/data ... ok >>> creating subdirectories ... ok >>> selecting default max_connections ... 100 >>> selecting default shared_buffers ... 32MB >>> creating configuration files ... ok >>> creating template1 database in /zdb/pgsql/data/base/1 ... FATAL: >>> could not open file "pg_xlog/000000010000000000000001" (log file 0, >>> segment 1): No such file or directory >>> child process exited with exit code 1 >>> initdb: removing data directory "/zdb/pgsql/data" >>> >>> My best guess is that it has something to do with permissions, but I >>> really have no idea. Has anyone seen this before and found a way >>> around it? >> >> >> Not quite what is in the the init script in rc.d, have you tried running >> the initdb command directly? As you say I believe there are permissions >> problems. So following the instructions found below may solve your problem: >> >> http://www.postgresql.org/docs/9.2/interactive/app-initdb.html >> >> >> >>> >>> -David >>> >>> >> >> >> -- >> Adrian Klaver >> adrian.klaver@gmail.com >> > > > I've tried initdb directly: > > initdb -D /zdb/pgsql/data > > ...and also through pg_ctl: > > pg_ctl initdb -D /zdb/pgsql/data > > ...and still seem to wind up with the error: > > creating template1 database in /zdb/pgsql/data/base/1 ... FATAL: > could not open file "pg_xlog/000000010000000000000001" (log file 0, > segment 1): No such file or directory You are doing the above as the database user ex:postgres? The database user has permissions on /zdb/pgsql/data? > > -David > > -- Adrian Klaver adrian.klaver@gmail.com
David Noel <david.i.noel@gmail.com> writes:
> I've tried initdb directly:
> initdb -D /zdb/pgsql/data
> ...and still seem to wind up with the error:
> creating template1 database in /zdb/pgsql/data/base/1 ... FATAL:
> could not open file "pg_xlog/000000010000000000000001" (log file 0,
> segment 1): No such file or directory
Hm, that eliminates my first theory that the rc.d script is using
initdb's --xlogdir switch to try to put pg_xlog someplace that doesn't
exist on your system. But it seems like this must be some variant of
that. A simple permissions problem is not very credible, because at
this point initdb has successfully made the data directory and a bunch
of files within it already.
You could get more information by using the --noclean switch to prevent
removal of the datadir after failure, and then having a look at the
debris. Is there a pg_xlog subdirectory inside /zdb/pgsql/data, and if
so what permissions has it got? Perhaps it is not a directory, but a
symlink to somewhere else?
regards, tom lane
On Thu, Dec 13, 2012 at 10:00 PM, Adrian Klaver <adrian.klaver@gmail.com> wrote:
You are doing the above as the database user ex:postgres?On 12/13/2012 08:18 AM, David Noel wrote:On 12/13/12, Adrian Klaver <adrian.klaver@gmail.com> wrote:On 12/13/2012 07:38 AM, David Noel wrote:I'm running into the following error message when running initdb (FreeBSD
host):
ygg# /usr/local/etc/rc.d/postgresql initdb -D /zdb/pgsql/data --debug
The files belonging to this database system will be owned by user
"pgsql".
This user must also own the server process.
The database cluster will be initialized with locales
COLLATE: C
CTYPE: en_US.UTF-8
MESSAGES: en_US.UTF-8
MONETARY: en_US.UTF-8
NUMERIC: en_US.UTF-8
TIME: en_US.UTF-8
The default text search configuration will be set to "english".
creating directory /zdb/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database in /zdb/pgsql/data/base/1 ... FATAL:
could not open file "pg_xlog/000000010000000000000001" (log file 0,
segment 1): No such file or directory
child process exited with exit code 1
initdb: removing data directory "/zdb/pgsql/data"
My best guess is that it has something to do with permissions, but I
really have no idea. Has anyone seen this before and found a way
around it?
Not quite what is in the the init script in rc.d, have you tried running
the initdb command directly? As you say I believe there are permissions
problems. So following the instructions found below may solve your problem:
http://www.postgresql.org/docs/9.2/interactive/app-initdb.html
-David
--
Adrian Klaver
adrian.klaver@gmail.com
I've tried initdb directly:
initdb -D /zdb/pgsql/data
...and also through pg_ctl:
pg_ctl initdb -D /zdb/pgsql/data
...and still seem to wind up with the error:
creating template1 database in /zdb/pgsql/data/base/1 ... FATAL:
could not open file "pg_xlog/000000010000000000000001" (log file 0,
segment 1): No such file or directory
The database user has permissions on /zdb/pgsql/data?
The rc.d script executes the command as the correct user. This is the line in rc.d script that runs the initdb command:
su -l -c ${postgresql_class} ${postgresql_user} -c "exec /usr/local/bin/initdb ${postgresql_initdb_flags} -D ${postgresql_data}"
Not sure about his permission though
On 12/13/2012 08:35 AM, Tom Lane wrote: > David Noel <david.i.noel@gmail.com> writes: >> I've tried initdb directly: >> initdb -D /zdb/pgsql/data >> ...and still seem to wind up with the error: >> creating template1 database in /zdb/pgsql/data/base/1 ... FATAL: >> could not open file "pg_xlog/000000010000000000000001" (log file 0, >> segment 1): No such file or directory > > > You could get more information by using the --noclean switch to prevent > removal of the datadir after failure, and then having a look at the > debris. Is there a pg_xlog subdirectory inside /zdb/pgsql/data, and if > so what permissions has it got? Perhaps it is not a directory, but a > symlink to somewhere else? Always learning, did not know --noclean existed. That is handy. Thanks. > > regards, tom lane > > -- Adrian Klaver adrian.klaver@gmail.com
On Thu, Dec 13, 2012 at 10:05 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
David Noel <david.i.noel@gmail.com> writes:
> I've tried initdb directly:
> initdb -D /zdb/pgsql/data> ...and still seem to wind up with the error:Hm, that eliminates my first theory that the rc.d script is using
> creating template1 database in /zdb/pgsql/data/base/1 ... FATAL:
> could not open file "pg_xlog/000000010000000000000001" (log file 0,
> segment 1): No such file or directory
initdb's --xlogdir switch to try to put pg_xlog someplace that doesn't
exist on your system. But it seems like this must be some variant of
that. A simple permissions problem is not very credible, because at
this point initdb has successfully made the data directory and a bunch
of files within it already.
You could get more information by using the --noclean switch to prevent
removal of the datadir after failure, and then having a look at the
debris. Is there a pg_xlog subdirectory inside /zdb/pgsql/data, and if
so what permissions has it got? Perhaps it is not a directory, but a
symlink to somewhere else?
regards, tom lane
Tom
Amitabh
> You are doing the above as the database user ex:postgres? confirmed: ygg:/usr/home/ygg> whoami pgsql > The database user has permissions on /zdb/pgsql/data? confirmed: ygg:/usr/home/ygg> ll /zdb/ total 3 drwxrwxrwx 3 cvs wheel 3 Dec 12 15:33 cvsroot drwxrwxrwx 2 pgsql wheel 2 Dec 13 10:17 pgsql
On 12/13/12, Tom Lane <tgl@sss.pgh.pa.us> wrote: > David Noel <david.i.noel@gmail.com> writes: >> I've tried initdb directly: >> initdb -D /zdb/pgsql/data >> ...and still seem to wind up with the error: >> creating template1 database in /zdb/pgsql/data/base/1 ... FATAL: >> could not open file "pg_xlog/000000010000000000000001" (log file 0, >> segment 1): No such file or directory > > Hm, that eliminates my first theory that the rc.d script is using > initdb's --xlogdir switch to try to put pg_xlog someplace that doesn't > exist on your system. But it seems like this must be some variant of > that. A simple permissions problem is not very credible, because at > this point initdb has successfully made the data directory and a bunch > of files within it already. > > You could get more information by using the --noclean switch to prevent > removal of the datadir after failure, and then having a look at the > debris. Is there a pg_xlog subdirectory inside /zdb/pgsql/data, and if > so what permissions has it got? Perhaps it is not a directory, but a > symlink to somewhere else? > > regards, tom lane > ygg:/usr/home/ygg> initdb -D /zdb/pgsql/data --noclean ... creating template1 database in /zdb/pgsql/data/base/1 ... FATAL: could not open file "pg_xlog/000000010000000000000001" (log file 0, segment 1): No such file or directory child process exited with exit code 1 initdb: data directory "/zdb/pgsql/data" not removed at user's request ygg:/usr/home/ygg> ll /zdb/pgsql/ total 2 drwx------ 14 pgsql wheel 18 Dec 13 12:42 data ygg:/usr/home/ygg> ll /zdb/pgsql/data/ total 53 -rw------- 1 pgsql wheel 4 Dec 13 12:42 PG_VERSION drwx------ 3 pgsql wheel 3 Dec 13 12:42 base drwx------ 2 pgsql wheel 2 Dec 13 12:42 global drwx------ 2 pgsql wheel 2 Dec 13 12:42 pg_clog -rw------- 1 pgsql wheel 4467 Dec 13 12:42 pg_hba.conf -rw------- 1 pgsql wheel 1636 Dec 13 12:42 pg_ident.conf drwx------ 4 pgsql wheel 4 Dec 13 12:42 pg_multixact drwx------ 2 pgsql wheel 3 Dec 13 12:42 pg_notify drwx------ 2 pgsql wheel 2 Dec 13 12:42 pg_serial drwx------ 2 pgsql wheel 2 Dec 13 12:42 pg_snapshots drwx------ 2 pgsql wheel 2 Dec 13 12:42 pg_stat_tmp drwx------ 2 pgsql wheel 2 Dec 13 12:42 pg_subtrans drwx------ 2 pgsql wheel 2 Dec 13 12:42 pg_tblspc drwx------ 2 pgsql wheel 2 Dec 13 12:42 pg_twophase drwx------ 3 pgsql wheel 3 Dec 13 12:42 pg_xlog -rw------- 1 pgsql wheel 19713 Dec 13 12:42 postgresql.conf ygg:/usr/home/ygg> ll /zdb/pgsql/data/pg_xlog/ total 2 drwx------ 2 pgsql wheel 2 Dec 13 12:42 archive_status
On 12/13/2012 10:42 AM, David Noel wrote: >> You are doing the above as the database user ex:postgres? > > confirmed: > > ygg:/usr/home/ygg> whoami > pgsql > > >> The database user has permissions on /zdb/pgsql/data? > > confirmed: > > ygg:/usr/home/ygg> ll /zdb/ > total 3 > drwxrwxrwx 3 cvs wheel 3 Dec 12 15:33 cvsroot > drwxrwxrwx 2 pgsql wheel 2 Dec 13 10:17 pgsql > Hmmm. Saw the subsequent post also. More questions:( The whoami above shows running from ygg home directory. How did you become pgsql? Does the pgsql user have a login account you could use? Where is /zdb ? -- Adrian Klaver adrian.klaver@gmail.com
On 12/13/2012 10:47 AM, David Noel wrote: > On 12/13/12, Tom Lane <tgl@sss.pgh.pa.us> wrote: > ygg:/usr/home/ygg> ll /zdb/pgsql/data/pg_xlog/ > total 2 > drwx------ 2 pgsql wheel 2 Dec 13 12:42 archive_status > Different train of thought, away from permissions. How was Postgres installed? Where there any errors during the install process?-- Is there another version of Postgres on this machine? Adrian Klaver adrian.klaver@gmail.com
On 12/13/12, Adrian Klaver <adrian.klaver@gmail.com> wrote: > On 12/13/2012 10:47 AM, David Noel wrote: >> On 12/13/12, Tom Lane <tgl@sss.pgh.pa.us> wrote: > >> ygg:/usr/home/ygg> ll /zdb/pgsql/data/pg_xlog/ >> total 2 >> drwx------ 2 pgsql wheel 2 Dec 13 12:42 archive_status >> > > Different train of thought, away from permissions. > > How was Postgres installed? > > Where there any errors during the install process?-- > > Is there another version of Postgres on this machine? > > Adrian Klaver > adrian.klaver@gmail.com > I'd logged in as ygg via `su ygg`. /zdb is a zfs volume I've created for cvs and postgres. Postgres was installed via `make install` and there were no errors during the install process. I could try deinstalling and installing it again... maybe that will fix something.
Unfortunately no luck there. Still stick with the same error. On 12/13/12, David Noel <david.i.noel@gmail.com> wrote: > On 12/13/12, Adrian Klaver <adrian.klaver@gmail.com> wrote: >> On 12/13/2012 10:47 AM, David Noel wrote: >>> On 12/13/12, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> >>> ygg:/usr/home/ygg> ll /zdb/pgsql/data/pg_xlog/ >>> total 2 >>> drwx------ 2 pgsql wheel 2 Dec 13 12:42 archive_status >>> >> >> Different train of thought, away from permissions. >> >> How was Postgres installed? >> >> Where there any errors during the install process?-- >> >> Is there another version of Postgres on this machine? >> >> Adrian Klaver >> adrian.klaver@gmail.com >> > > I'd logged in as ygg via `su ygg`. > > /zdb is a zfs volume I've created for cvs and postgres. > > Postgres was installed via `make install` and there were no errors > during the install process. > > I could try deinstalling and installing it again... maybe that will > fix something. >
David Noel <david.i.noel@gmail.com> writes:
> On 12/13/12, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> You could get more information by using the --noclean switch to prevent
>> removal of the datadir after failure, and then having a look at the
>> debris. Is there a pg_xlog subdirectory inside /zdb/pgsql/data, and if
>> so what permissions has it got? Perhaps it is not a directory, but a
>> symlink to somewhere else?
[ nothing obviously wrong there... ]
Well, it looks like there are no easy answers here, so we'll have to
start digging. Is this a purely stock version of Postgres? Which
release exactly? Where did you get it from, or if you built it
yourself, what configuration options did you use? What's the platform
exactly?
What I'd suggest as a next step is to use strace or local equivalent on
the initdb run, to capture a log of everything it does with pg_xlog.
(And everything else too --- the log is likely to be bulky, so please
don't post it here. You could send it to me off-list if it makes no
sense to you.)
regards, tom lane
David Noel <david.i.noel@gmail.com> writes:
> /zdb is a zfs volume I've created for cvs and postgres.
zfs eh? What happens if you point initdb at a non-zfs volume?
(I"m wondering if zfs has issues with the O_DIRECT flag that we'll
probably try to use with pg_xlog files.)
regards, tom lane
On 12/13/12, Tom Lane <tgl@sss.pgh.pa.us> wrote: > David Noel <david.i.noel@gmail.com> writes: >> /zdb is a zfs volume I've created for cvs and postgres. > > zfs eh? What happens if you point initdb at a non-zfs volume? > > (I"m wondering if zfs has issues with the O_DIRECT flag that we'll > probably try to use with pg_xlog files.) I /boot off of a UFS volume so I created a directory there, chown and chmod'ed it, then ran initdb again. Same error, unfortunately.
David Noel <david.i.noel@gmail.com> writes:
> On 12/13/12, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> (I"m wondering if zfs has issues with the O_DIRECT flag that we'll
>> probably try to use with pg_xlog files.)
> I /boot off of a UFS volume so I created a directory there, chown and
> chmod'ed it, then ran initdb again. Same error, unfortunately.
Hm. So far as I can find from googling, ZFS doesn't support O_DIRECT,
or if it does the support is very new. However, if that were the
problem then you ought to be seeing EINVAL or similar, not ENOENT;
and what's more the file should have gotten created before we'd try to
open it with O_DIRECT, so it should have been there after --noclean.
So I'm still baffled. Back to wondering if strace will show anything
interesting.
regards, tom lane
On 12/13/2012 01:40 PM, David Noel wrote: > On 12/13/12, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> David Noel <david.i.noel@gmail.com> writes: >>> /zdb is a zfs volume I've created for cvs and postgres. >> >> zfs eh? What happens if you point initdb at a non-zfs volume? >> >> (I"m wondering if zfs has issues with the O_DIRECT flag that we'll >> probably try to use with pg_xlog files.) > > I /boot off of a UFS volume so I created a directory there, chown and > chmod'ed it, then ran initdb again. Same error, unfortunately. Well this has definitely moved up a support level and past anything I know about. For the record what version of FreeBSD are you running in case someone is searching the archives? > > -- Adrian Klaver adrian.klaver@gmail.com
On Fri, Dec 14, 2012 at 3:10 AM, David Noel <david.i.noel@gmail.com> wrote:
On 12/13/12, Tom Lane <tgl@sss.pgh.pa.us> wrote:> David Noel <david.i.noel@gmail.com> writes:I /boot off of a UFS volume so I created a directory there, chown and
>> /zdb is a zfs volume I've created for cvs and postgres.
>
> zfs eh? What happens if you point initdb at a non-zfs volume?
>
> (I"m wondering if zfs has issues with the O_DIRECT flag that we'll
> probably try to use with pg_xlog files.)
chmod'ed it, then ran initdb again. Same error, unfortunately.
David
Did you use ports to install postgresql? What is the version of postgresql and freebsd you are using? I am getting a different output while running the initdb command through the rc script, and it's not using the -D path to initialize the cluster, it falls back to the default location /usr/local/pgsql/data . Setting postgresql_data="/usr/local/pgsql1/data" in /etc/rc.conf makes it initialize at the right location . I did not had to mess with an permissions and the rc commands were all run as root. The above was tested on FreeBSD 9.1 with Postgresql 9.2 on UFS.
On FreeBSD 8.1 with Postgresql 9.1 on UFS installed though ports, it was the same story, except that I had to change ownership to user pgsql for the /usr/local/pgsql1 directory
Amitabh
> Well this has definitely moved up a support level and past anything I > know about. > For the record what version of FreeBSD are you running in case someone > is searching the archives? Any help is always appreciated. FreeBSD 8.3-RELEASE-p5. (I wonder if upgrading to 9.1 would do anything..)
> Did you use ports to install postgresql? Yes > What is the version of postgresql and freebsd you are using? postgresql client and server v. 9.2.2. If all else fails I could try downgrading to a previous version of postgres. FreeBSD 8.3-RELEASE-p5 > I am getting a different output while running > the initdb command through the rc script, and it's not using the -D path to > initialize the cluster, it falls back to the default location > /usr/local/pgsql/data . Setting postgresql_data="/usr/local/pgsql1/data" > in /etc/rc.conf makes it initialize at the right location . I did not had > to mess with an permissions and the rc commands were all run as root. The > above was tested on FreeBSD 9.1 with Postgresql 9.2 on UFS. I didn't have any luck with the rc script but I was able to use it to get a ktrace dump as root (ktrace as user pgsql doesn't seem to work). So hopefully that will show something(!)
On Fri, Dec 14, 2012 at 4:28 PM, David Noel <david.i.noel@gmail.com> wrote:
If it helps, my versions were FreeBSD 8.1 + PG 9.0.5 using UFS on a dedicated server and FreeBSD 9.1 (available on FTP sites but not announced yet) + PG 9.2.2 using UFS on a virtual box instance. Looks like ZFS is the only major variable here.
> Did you use ports to install postgresql?Yespostgresql client and server v. 9.2.2. If all else fails I could try
> What is the version of postgresql and freebsd you are using?
downgrading to a previous version of postgres.
FreeBSD 8.3-RELEASE-p5I didn't have any luck with the rc script but I was able to use it to
> I am getting a different output while running
> the initdb command through the rc script, and it's not using the -D path to
> initialize the cluster, it falls back to the default location
> /usr/local/pgsql/data . Setting postgresql_data="/usr/local/pgsql1/data"
> in /etc/rc.conf makes it initialize at the right location . I did not had
> to mess with an permissions and the rc commands were all run as root. The
> above was tested on FreeBSD 9.1 with Postgresql 9.2 on UFS.
get a ktrace dump as root (ktrace as user pgsql doesn't seem to work).
So hopefully that will show something(!)
David
Amitabh
On 12/13/12, David Noel <david.i.noel@gmail.com> wrote: > I'm running into the following error message when running initdb (FreeBSD > host): > > ygg# /usr/local/etc/rc.d/postgresql initdb -D /zdb/pgsql/data --debug > The files belonging to this database system will be owned by user "pgsql". > This user must also own the server process. > > The database cluster will be initialized with locales > COLLATE: C > CTYPE: en_US.UTF-8 > MESSAGES: en_US.UTF-8 > MONETARY: en_US.UTF-8 > NUMERIC: en_US.UTF-8 > TIME: en_US.UTF-8 > The default text search configuration will be set to "english". > > creating directory /zdb/pgsql/data ... ok > creating subdirectories ... ok > selecting default max_connections ... 100 > selecting default shared_buffers ... 32MB > creating configuration files ... ok > creating template1 database in /zdb/pgsql/data/base/1 ... FATAL: > could not open file "pg_xlog/000000010000000000000001" (log file 0, > segment 1): No such file or directory > child process exited with exit code 1 > initdb: removing data directory "/zdb/pgsql/data" > > My best guess is that it has something to do with permissions, but I > really have no idea. Has anyone seen this before and found a way > around it? > > -David Interestingly, I have a second--virtually identical--server that I just tried initdb on. FreeBSD 8.3-RELEASE-p5, postgresql-server-9.2.2. Exact same "FATAL: could not open file pg_xlog" error. So it is reproducible.
On 12/14/2012 04:08 AM, David Noel wrote: > > Interestingly, I have a second--virtually identical--server that I > just tried initdb on. FreeBSD 8.3-RELEASE-p5, postgresql-server-9.2.2. > Exact same "FATAL: could not open file pg_xlog" error. So it is > reproducible. Does virtually identical extend to architecture, amd64? > > -- Adrian Klaver adrian.klaver@gmail.com
>> Interestingly, I have a second--virtually identical--server that I >> just tried initdb on. FreeBSD 8.3-RELEASE-p5, postgresql-server-9.2.2. >> Exact same "FATAL: could not open file pg_xlog" error. So it is >> reproducible. > > Does virtually identical extend to architecture, amd64? Yes... and hardware. They only vary in the amount of CPU cache--one has twice the other.
David Noel <david.i.noel@gmail.com> writes:
> I didn't have any luck with the rc script but I was able to use it to
> get a ktrace dump as root (ktrace as user pgsql doesn't seem to work).
> So hopefully that will show something(!)
The relevant part of the ktrace output is
71502 postgres CALL unlink(0x7fffffffc130)
71502 postgres NAMI "pg_xlog/xlogtemp.71502"
71502 postgres RET unlink -1 errno 2 No such file or directory
71502 postgres CALL open(0x7fffffffc130,O_RDWR|O_CREAT|O_EXCL,S_IRUSR|S_IWUSR)
71502 postgres NAMI "pg_xlog/xlogtemp.71502"
71502 postgres RET open 3
71502 postgres CALL write(0x3,0x801a56030,0x2000)
71502 postgres GIO fd 3 wrote 4096 bytes
.... a lot of uninteresting write() calls snipped ...
71502 postgres RET write 8192/0x2000
71502 postgres CALL close(0x3)
71502 postgres RET close 0
71502 postgres CALL unlink(0x7fffffffbc60)
71502 postgres NAMI "pg_xlog/000000010000000000000001"
71502 postgres RET unlink -1 errno 2 No such file or directory
71502 postgres CALL link(0x7fffffffc130,0x7fffffffbc60)
71502 postgres NAMI "pg_xlog/xlogtemp.71502"
71502 postgres NAMI "pg_xlog/000000010000000000000001"
71502 postgres RET link -1 errno 1 Operation not permitted
71502 postgres CALL unlink(0x7fffffffc130)
71502 postgres NAMI "pg_xlog/xlogtemp.71502"
71502 postgres RET unlink 0
71502 postgres CALL open(0x7fffffffc530,O_RDWR,<unused>0x180)
71502 postgres NAMI "pg_xlog/000000010000000000000001"
71502 postgres RET open -1 errno 2 No such file or directory
This corresponds to the execution of XLogFileInit(), and what's
evidently happening is that we successfully create and zero-fill
the first xlog segment file under a temporary name, but then
the attempt to rename it into place with link() fails with EPERM.
This is really a WTF kind of failure, I think. The directory is
certainly writable --- it was made under our own UID, and what's
more we just managed to create the file there under its temp name.
So how can we get an EPERM failure from link()?
I think this is a kernel bug.
regards, tom lane
PS: one odd thing here is that the ereport(LOG) in
InstallXLogFileSegment isn't doing anything; otherwise we'd have gotten
a much more helpful error report about "could not link file". I don't
think we run the bootstrap mode with log_min_messages set high enough to
disable LOG messages, so why isn't it printing? Nonetheless, this error
shouldn't have occurred.
On 12/14/12, Tom Lane <tgl@sss.pgh.pa.us> wrote: > David Noel <david.i.noel@gmail.com> writes: >> I didn't have any luck with the rc script but I was able to use it to >> get a ktrace dump as root (ktrace as user pgsql doesn't seem to work). >> So hopefully that will show something(!) > > The relevant part of the ktrace output is > > 71502 postgres CALL unlink(0x7fffffffc130) > 71502 postgres NAMI "pg_xlog/xlogtemp.71502" > 71502 postgres RET unlink -1 errno 2 No such file or directory > 71502 postgres CALL > open(0x7fffffffc130,O_RDWR|O_CREAT|O_EXCL,S_IRUSR|S_IWUSR) > 71502 postgres NAMI "pg_xlog/xlogtemp.71502" > 71502 postgres RET open 3 > 71502 postgres CALL write(0x3,0x801a56030,0x2000) > 71502 postgres GIO fd 3 wrote 4096 bytes > .... a lot of uninteresting write() calls snipped ... > 71502 postgres RET write 8192/0x2000 > 71502 postgres CALL close(0x3) > 71502 postgres RET close 0 > 71502 postgres CALL unlink(0x7fffffffbc60) > 71502 postgres NAMI "pg_xlog/000000010000000000000001" > 71502 postgres RET unlink -1 errno 2 No such file or directory > 71502 postgres CALL link(0x7fffffffc130,0x7fffffffbc60) > 71502 postgres NAMI "pg_xlog/xlogtemp.71502" > 71502 postgres NAMI "pg_xlog/000000010000000000000001" > 71502 postgres RET link -1 errno 1 Operation not permitted > 71502 postgres CALL unlink(0x7fffffffc130) > 71502 postgres NAMI "pg_xlog/xlogtemp.71502" > 71502 postgres RET unlink 0 > 71502 postgres CALL open(0x7fffffffc530,O_RDWR,<unused>0x180) > 71502 postgres NAMI "pg_xlog/000000010000000000000001" > 71502 postgres RET open -1 errno 2 No such file or directory > > This corresponds to the execution of XLogFileInit(), and what's > evidently happening is that we successfully create and zero-fill > the first xlog segment file under a temporary name, but then > the attempt to rename it into place with link() fails with EPERM. > > This is really a WTF kind of failure, I think. The directory is > certainly writable --- it was made under our own UID, and what's > more we just managed to create the file there under its temp name. > So how can we get an EPERM failure from link()? > > I think this is a kernel bug. > > regards, tom lane > > PS: one odd thing here is that the ereport(LOG) in > InstallXLogFileSegment isn't doing anything; otherwise we'd have gotten > a much more helpful error report about "could not link file". I don't > think we run the bootstrap mode with log_min_messages set high enough to > disable LOG messages, so why isn't it printing? Nonetheless, this error > shouldn't have occurred. Thanks so much for the analysis. Where to from here? The freebsd-database@freebsd.org mailing list? The postgresql port maintainer? Who should I be in touch with?
David Noel <david.i.noel@gmail.com> writes:
> On 12/14/12, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> This corresponds to the execution of XLogFileInit(), and what's
>> evidently happening is that we successfully create and zero-fill
>> the first xlog segment file under a temporary name, but then
>> the attempt to rename it into place with link() fails with EPERM.
>>
>> This is really a WTF kind of failure, I think. The directory is
>> certainly writable --- it was made under our own UID, and what's
>> more we just managed to create the file there under its temp name.
>> So how can we get an EPERM failure from link()?
>>
>> I think this is a kernel bug.
> Thanks so much for the analysis. Where to from here? The
> freebsd-database@freebsd.org mailing list? The postgresql port
> maintainer? Who should I be in touch with?
You need to talk to some FreeBSD kernel hackers about why link()
might be failing here. Since you see it on UFS too, we can probably
exonerate the ZFS filesystem-specific code.
I did some googling and found that EPERM can be issued if the filesystem
doesn't support hard links (which shouldn't apply to ZFS I trust).
Also, Linux has a "protected_hardlinks" option that causes certain
attempts at creating hard links to fail --- but our use-case here
doesn't fall foul of any of those restrictions AFAICS, and of course
FreeBSD isn't Linux. Still, I wonder if you're running into some
misdesigned or misimplemented security restriction. You might want
to look at your kernel parameters and see if any of them look like
they might have to do with restricting hard-link operations.
Also, since Amitabh failed to duplicate the failure on both earlier
and later FreeBSD kernels, and we've not heard reports of this from
anybody else either, it seems more than possible that it's a plain
old bug in the specific kernel version you're using.
As a short-term workaround, I'd suggest rebuilding with
HAVE_WORKING_LINK disabled. (Just remove that #define from
src/include/pg_config_manual.h and rebuild.)
regards, tom lane
> You need to talk to some FreeBSD kernel hackers about why link() > might be failing here. Since you see it on UFS too, we can probably > exonerate the ZFS filesystem-specific code. > > I did some googling and found that EPERM can be issued if the filesystem > doesn't support hard links (which shouldn't apply to ZFS I trust). > Also, Linux has a "protected_hardlinks" option that causes certain > attempts at creating hard links to fail --- but our use-case here > doesn't fall foul of any of those restrictions AFAICS, and of course > FreeBSD isn't Linux. Still, I wonder if you're running into some > misdesigned or misimplemented security restriction. You might want > to look at your kernel parameters and see if any of them look like > they might have to do with restricting hard-link operations. > > Also, since Amitabh failed to duplicate the failure on both earlier > and later FreeBSD kernels, and we've not heard reports of this from > anybody else either, it seems more than possible that it's a plain > old bug in the specific kernel version you're using. > > As a short-term workaround, I'd suggest rebuilding with > HAVE_WORKING_LINK disabled. (Just remove that #define from > src/include/pg_config_manual.h and rebuild.) OK, thanks. I've gotten in touch with the freebsd-hackers mailing list. Hopefully we'll be able to get this one figured out. I compiled a GENERIC kernel and tried it again. Still no luck. But at least we know now it wasn't a configuration error.
David
I just tried installing PG 9.2.2 on FreeBSD 8.3 and initdb went smoothly without any problems. the only difference would be that I had it running in a virtualbox instance. My config were as follows;
FreeBSD 8.3-RELEASE-p5 (binary updated through freebsd-update)
PG 9.2.2 (installed through ports)
root filesystem on UFS
ZFS mounted on /example
PG data directory set to /example/pgsql/data
relevant rc.conf entries:
postgresql_enable="YES"
postgresql_data="/example/pgsql/data"
running "/usr/local/etc/rc.d/postgresql initdb" works perfectly. Just needed to set the correct permissions on /example/pgsql directory.
Amitabh
On Sat, Dec 15, 2012 at 10:32 AM, David Noel <david.i.noel@gmail.com> wrote:
> You need to talk to some FreeBSD kernel hackers about why link()OK, thanks. I've gotten in touch with the freebsd-hackers mailing
> might be failing here. Since you see it on UFS too, we can probably
> exonerate the ZFS filesystem-specific code.
>
> I did some googling and found that EPERM can be issued if the filesystem
> doesn't support hard links (which shouldn't apply to ZFS I trust).
> Also, Linux has a "protected_hardlinks" option that causes certain
> attempts at creating hard links to fail --- but our use-case here
> doesn't fall foul of any of those restrictions AFAICS, and of course
> FreeBSD isn't Linux. Still, I wonder if you're running into some
> misdesigned or misimplemented security restriction. You might want
> to look at your kernel parameters and see if any of them look like
> they might have to do with restricting hard-link operations.
>
> Also, since Amitabh failed to duplicate the failure on both earlier
> and later FreeBSD kernels, and we've not heard reports of this from
> anybody else either, it seems more than possible that it's a plain
> old bug in the specific kernel version you're using.
>
> As a short-term workaround, I'd suggest rebuilding with
> HAVE_WORKING_LINK disabled. (Just remove that #define from
> src/include/pg_config_manual.h and rebuild.)
list. Hopefully we'll be able to get this one figured out.
I compiled a GENERIC kernel and tried it again. Still no luck. But at
least we know now it wasn't a configuration error.
Ugh. It turns out I had security.bsd.hardlink_check_gid=1 and security.bsd.hardlink_check_uid=1 in sysctl.conf. Setting them to 0 fixed everything. That's frustrating. Sorry for all the trouble. Many thanks for helping resolve this. On 12/15/12, Amitabh Kant <amitabhkant@gmail.com> wrote: > David > > I just tried installing PG 9.2.2 on FreeBSD 8.3 and initdb went smoothly > without any problems. the only difference would be that I had it running in > a virtualbox instance. My config were as follows; > > FreeBSD 8.3-RELEASE-p5 (binary updated through freebsd-update) > PG 9.2.2 (installed through ports) > root filesystem on UFS > ZFS mounted on /example > PG data directory set to /example/pgsql/data > > relevant rc.conf entries: > postgresql_enable="YES" > postgresql_data="/example/pgsql/data" > > running "/usr/local/etc/rc.d/postgresql initdb" works perfectly. Just > needed to set the correct permissions on /example/pgsql directory. > > Amitabh > > > > > On Sat, Dec 15, 2012 at 10:32 AM, David Noel <david.i.noel@gmail.com> > wrote: > >> > You need to talk to some FreeBSD kernel hackers about why link() >> > might be failing here. Since you see it on UFS too, we can probably >> > exonerate the ZFS filesystem-specific code. >> > >> > I did some googling and found that EPERM can be issued if the >> > filesystem >> > doesn't support hard links (which shouldn't apply to ZFS I trust). >> > Also, Linux has a "protected_hardlinks" option that causes certain >> > attempts at creating hard links to fail --- but our use-case here >> > doesn't fall foul of any of those restrictions AFAICS, and of course >> > FreeBSD isn't Linux. Still, I wonder if you're running into some >> > misdesigned or misimplemented security restriction. You might want >> > to look at your kernel parameters and see if any of them look like >> > they might have to do with restricting hard-link operations. >> > >> > Also, since Amitabh failed to duplicate the failure on both earlier >> > and later FreeBSD kernels, and we've not heard reports of this from >> > anybody else either, it seems more than possible that it's a plain >> > old bug in the specific kernel version you're using. >> > >> > As a short-term workaround, I'd suggest rebuilding with >> > HAVE_WORKING_LINK disabled. (Just remove that #define from >> > src/include/pg_config_manual.h and rebuild.) >> >> OK, thanks. I've gotten in touch with the freebsd-hackers mailing >> list. Hopefully we'll be able to get this one figured out. >> >> I compiled a GENERIC kernel and tried it again. Still no luck. But at >> least we know now it wasn't a configuration error. >> >
David Noel <david.i.noel@gmail.com> writes:
> Ugh. It turns out I had security.bsd.hardlink_check_gid=1 and
> security.bsd.hardlink_check_uid=1 in sysctl.conf. Setting them to 0
> fixed everything. That's frustrating. Sorry for all the trouble. Many
> thanks for helping resolve this.
Hah, I wondered if it might be something like that. Still, it's hard to
see how this link() would fail any security check that didn't amount to
disabling link() altogether. The UID and GID of the process, the old
file, and the directory will all match --- so exactly what rule are we
falling foul of? (Or IOW, this still seems like a kernel bug, though
perhaps of a design rather than implementation kind.)
regards, tom lane
I wrote:
> Hah, I wondered if it might be something like that. Still, it's hard to
> see how this link() would fail any security check that didn't amount to
> disabling link() altogether. The UID and GID of the process, the old
> file, and the directory will all match --- so exactly what rule are we
> falling foul of? (Or IOW, this still seems like a kernel bug, though
> perhaps of a design rather than implementation kind.)
Ah, I found your thread on freebsd-hackers, and it looks like Konstantin
Belousov has the answer:
>> The sysctl indeed has an interesting interaction with the fact that the
>> created inode inherits gid from the parent directory. If you use it, you
>> must ensure that the directory group is in the group set of the process.
IOW, the problem is that you'd created $PGDATA with group "wheel", but
this wasn't a group that the initdb process belonged to. It's a bit
weird that of all the operations initdb had tried so far, only the
link() would fail, but there you have it.
Anyway the takeaway for us is that it's bad that the link() failure per
se didn't get logged; this might have been less mystifying if it was.
I've started a thread on pgsql-hackers about fixing the bootstrap-time
logging rules for that.
regards, tom lane