Thread: postgresql doesn't start
hi there,
I've a big problem with my postgresql installation
I've postgresql 9.3 installed on docker; I start it via supervisord
I've never had issues with postgresql and I don't touched nothing (no update, no changes)
this is my configuration
there's a file called postgresql.conf in /etc/supervisor/conf.d/
cat command:
[program:postgres]
command=/opt/postgresql.sh
autostart=true
autorestart=true
stopsignal=QUIT
lookt at /opt/postgresql.sh
#!/bin/sh
# This script is run by Supervisor to start PostgreSQL 9.3 in foreground mode
if [ -d /var/run/postgresql ]; then
chmod 2775 /var/run/postgresql
else
install -d -m 2775 -o postgres -g postgres /var/run/postgresql
fi
exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf"
until yesterday there were no problem
right now I see this in the /var/log/supervisor's directory:
2015-10-01 21:40:18 UTC HINT: The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again.
2015-10-01 21:40:20 UTC FATAL: could not remove old lock file "postmaster.pid": Permission denied
if I remove this file with sudo permissions when I re-run the process I've the same error and I don't understand why. I tried the same configuration (postgresql installation and configuration) in another new docker installation and works fine.
what's the problem?
someone help me?
I've very important databases and I dont' know how to recover it
please let me know, thanks in advance
Paolo De Michele <paolo@paolodemichele.it> writes:
> 2015-10-01 21:40:20 UTC FATAL: could not remove old lock file
> "postmaster.pid": Permission denied
Looks like something removed the postmaster's write permission on
the data directory itself.
regards, tom lane
On 10/2/2015 2:02 PM, Paolo De Michele wrote: > I've postgresql 9.3 installed on docker docker is a packaging system for linux containers, its not an operating system per se. what OS is your docker container running on? supervisord is a process/daemon manager, its not part of 'postgresql', although apparently your docker container is using it to run postgresql. your questions all seem related to the specific packaging of this particular docker container, you might see who created this container configuration, and ask them these questions as there's a lot going on there abstracting things. -- john r pierce, recycling bits in santa cruz
On 10/02/2015 02:02 PM, Paolo De Michele wrote: > hi there, > > I've a big problem with my postgresql installation > I've postgresql 9.3 installed on docker; I start it via supervisord > I've never had issues with postgresql and I don't touched nothing (no > update, no changes) > > this is my configuration > there's a file called postgresql.conf in /etc/supervisor/conf.d/ > cat command: > > [program:postgres] > command=/opt/postgresql.sh > autostart=true > autorestart=true > stopsignal=QUIT If it where me I would change the above to: [program:postgres] command=/opt/postgresql.sh autostart=true stopsignal=TERM > > lookt at /opt/postgresql.sh > > #!/bin/sh > > # This script is run by Supervisor to start PostgreSQL 9.3 in foreground > mode > > if [ -d /var/run/postgresql ]; then > chmod 2775 /var/run/postgresql > else > install -d -m 2775 -o postgres -g postgres /var/run/postgresql > fi > > exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D > /var/lib/postgresql/9.3/main -c > config_file=/etc/postgresql/9.3/main/postgresql.conf" > > until yesterday there were no problem > right now I see this in the /var/log/supervisor's directory: > > 2015-10-01 21:40:18 UTC HINT: The file seems accidentally left over, > but it could not be removed. Please remove the file by hand and try again. > 2015-10-01 21:40:20 UTC FATAL: could not remove old lock file > "postmaster.pid": Permission denied > > if I remove this file with sudo permissions when I re-run the process > I've the same error and I don't understand why. I tried the same > configuration (postgresql installation and configuration) in another new > docker installation and works fine. > > what's the problem? > someone help me? > I've very important databases and I dont' know how to recover it > > please let me know, thanks in advance > > -- Adrian Klaver adrian.klaver@aklaver.com
On 10/2/2015 2:02 PM, Paolo De Michele wrote:
> exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D
> /var/lib/postgresql/9.3/main -c
> config_file=/etc/postgresql/9.3/main/postgresql.conf"
>
> until yesterday there were no problem
> right now I see this in the /var/log/supervisor's directory:
>
> 2015-10-01 21:40:18 UTC HINT: The file seems accidentally left over,
> but it could not be removed. Please remove the file by hand and try again.
> 2015-10-01 21:40:20 UTC FATAL: could not remove old lock file
> "postmaster.pid": Permission denied
try...
ls -la /var/lib/postgresql/9.3/main
The directory . should be owned by the postgres user, and it should have
700, 750, or 770 permissions. all the files in it should also be owned
by postgres.
also look and see if postgres logged anything in its own system log
files (/var/log/postgresql/9.3 or whatever).
--
john r pierce, recycling bits in santa cruz
Are you using docker on centos ? I had problem with centos/docker/postgresql because container size was (maybe still is) limited to 20GB on that specific OS. Maybe not related, but good to know
2015-10-03 0:03 GMT+02:00 John R Pierce <pierce@hogranch.com>:
On 10/2/2015 2:02 PM, Paolo De Michele wrote:exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf"
until yesterday there were no problem
right now I see this in the /var/log/supervisor's directory:
2015-10-01 21:40:18 UTC HINT: The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again.
2015-10-01 21:40:20 UTC FATAL: could not remove old lock file "postmaster.pid": Permission denied
try...
ls -la /var/lib/postgresql/9.3/main
The directory . should be owned by the postgres user, and it should have 700, 750, or 770 permissions. all the files in it should also be owned by postgres.
also look and see if postgres logged anything in its own system log files (/var/log/postgresql/9.3 or whatever).
--
john r pierce, recycling bits in santa cruz--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
hi there,
thanks a lot
so, I'm using docker on ubuntu 14.04
about permissions:
# ls -la /var/lib/postgresql/9.3/main
total 72
drwx------ 28 postgres postgres 4096 Oct 3 12:41 .
drwxr-xr-x 4 postgres postgres 4096 Oct 1 11:59 ..
-rwx------ 1 postgres postgres 4 Mar 21 2015 PG_VERSION
drwx------ 12 postgres postgres 4096 Oct 1 11:59 base
drwx------ 2 postgres postgres 4096 Oct 1 11:59 global
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_clog
drwx------ 6 postgres postgres 4096 Oct 1 11:59 pg_multixact
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_notify
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_serial
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_snapshots
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_stat
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_stat_tmp
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_subtrans
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_tblspc
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_twophase
drwx------ 4 postgres postgres 4096 Oct 1 12:18 pg_xlog
-rwx------ 1 postgres postgres 133 Oct 1 20:00 postmaster.opts
-rw------- 1 postgres postgres 50 Oct 3 12:41 postmaster.pid
in /var/log/postgresql/ I've a file with 0kb
it is weird
is there a way to recover all dbs without starting the daemon?
because I've another one docker with the same configuration and works fine (same os, same versione of postgresql)
On 3 October 2015 at 07:13, Nicolas Paris <niparisco@gmail.com> wrote:
Are you using docker on centos ? I had problem with centos/docker/postgresql because container size was (maybe still is) limited to 20GB on that specific OS. Maybe not related, but good to know2015-10-03 0:03 GMT+02:00 John R Pierce <pierce@hogranch.com>:On 10/2/2015 2:02 PM, Paolo De Michele wrote:exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf"
until yesterday there were no problem
right now I see this in the /var/log/supervisor's directory:
2015-10-01 21:40:18 UTC HINT: The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again.
2015-10-01 21:40:20 UTC FATAL: could not remove old lock file "postmaster.pid": Permission denied
try...
ls -la /var/lib/postgresql/9.3/main
The directory . should be owned by the postgres user, and it should have 700, 750, or 770 permissions. all the files in it should also be owned by postgres.
also look and see if postgres logged anything in its own system log files (/var/log/postgresql/9.3 or whatever).
--
john r pierce, recycling bits in santa cruz--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On 10/03/2015 05:50 AM, Paolo De Michele wrote: > hi there, > > thanks a lot > so, I'm using docker on ubuntu 14.04 > about permissions: > > # ls -la /var/lib/postgresql/9.3/main > total 72 > drwx------ 28 postgres postgres 4096 Oct 3 12:41 . > drwxr-xr-x 4 postgres postgres 4096 Oct 1 11:59 .. > -rwx------ 1 postgres postgres 4 Mar 21 2015 PG_VERSION > drwx------ 12 postgres postgres 4096 Oct 1 11:59 base > drwx------ 2 postgres postgres 4096 Oct 1 11:59 global > drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_clog > drwx------ 6 postgres postgres 4096 Oct 1 11:59 pg_multixact > drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_notify > drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_serial > drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_snapshots > drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_stat > drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_stat_tmp > drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_subtrans > drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_tblspc > drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_twophase > drwx------ 4 postgres postgres 4096 Oct 1 12:18 pg_xlog > -rwx------ 1 postgres postgres 133 Oct 1 20:00 postmaster.opts > -rw------- 1 postgres postgres 50 Oct 3 12:41 postmaster.pid > > in /var/log/postgresql/ I've a file with 0kb > it is weird > is there a way to recover all dbs without starting the daemon? What OS and version are you running? Assuming a Debian/Ubuntu flavor for now. How was Postgres installed? So there is a pid file present in the data directory, how about in the run directory, /var/run/postgresql? Is Postgres running? so: ps ax|grep post or pg_ctl status -D /var/lib/postgresql/9.3/main If not what happens if you remove the pid and start Postgres manually, not through supervisor? > because I've another one docker with the same configuration and works > fine (same os, same versione of postgresql) > > On 3 October 2015 at 07:13, Nicolas Paris <niparisco@gmail.com > <mailto:niparisco@gmail.com>> wrote: > > Are you using docker on centos ? I had problem with > centos/docker/postgresql because container size was (maybe still is) > limited to 20GB on that specific OS. Maybe not related, but good to > know > > 2015-10-03 0:03 GMT+02:00 John R Pierce <pierce@hogranch.com > <mailto:pierce@hogranch.com>>: > > On 10/2/2015 2:02 PM, Paolo De Michele wrote: > > exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D > /var/lib/postgresql/9.3/main -c > config_file=/etc/postgresql/9.3/main/postgresql.conf" > > until yesterday there were no problem > right now I see this in the /var/log/supervisor's directory: > > 2015-10-01 21:40:18 UTC HINT: The file seems accidentally > left over, but it could not be removed. Please remove the > file by hand and try again. > 2015-10-01 21:40:20 UTC FATAL: could not remove old lock > file "postmaster.pid": Permission denied > > > > try... > ls -la /var/lib/postgresql/9.3/main > > The directory . should be owned by the postgres user, and it > should have 700, 750, or 770 permissions. all the files in it > should also be owned by postgres. > > also look and see if postgres logged anything in its own system > log files (/var/log/postgresql/9.3 or whatever). > > > > -- > john r pierce, recycling bits in santa cruz > > > > -- > Sent via pgsql-general mailing list > (pgsql-general@postgresql.org <mailto:pgsql-general@postgresql.org>) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-general > > > -- Adrian Klaver adrian.klaver@aklaver.com
Adrian,
thanks a lot
about your questions
1. postgresql was installed via apt-get install postgresql -y command
2. in /var/run/postgresql/ there is only one file called 9.3-main.pid
3. postgres is not running, I did not find nothing with ps command
4. I remove the postmaster.pid file and I tried to run postgresql manually. this is the result:
# /etc/init.d/postgresql start
* Starting PostgreSQL 9.3 database server
The PostgreSQL server failed to start. Please check the log output.
...fail!
I verified in /var/log/postgresql/ and I've not found nothing
On 3 October 2015 at 15:19, Adrian Klaver <adrian.klaver@aklaver.com> wrote:
What OS and version are you running? Assuming a Debian/Ubuntu flavor for now.On 10/03/2015 05:50 AM, Paolo De Michele wrote:hi there,
thanks a lot
so, I'm using docker on ubuntu 14.04
about permissions:
# ls -la /var/lib/postgresql/9.3/main
total 72
drwx------ 28 postgres postgres 4096 Oct 3 12:41 .
drwxr-xr-x 4 postgres postgres 4096 Oct 1 11:59 ..
-rwx------ 1 postgres postgres 4 Mar 21 2015 PG_VERSION
drwx------ 12 postgres postgres 4096 Oct 1 11:59 base
drwx------ 2 postgres postgres 4096 Oct 1 11:59 global
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_clog
drwx------ 6 postgres postgres 4096 Oct 1 11:59 pg_multixact
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_notify
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_serial
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_snapshots
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_stat
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_stat_tmp
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_subtrans
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_tblspc
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_twophase
drwx------ 4 postgres postgres 4096 Oct 1 12:18 pg_xlog
-rwx------ 1 postgres postgres 133 Oct 1 20:00 postmaster.opts
-rw------- 1 postgres postgres 50 Oct 3 12:41 postmaster.pid
in /var/log/postgresql/ I've a file with 0kb
it is weird
is there a way to recover all dbs without starting the daemon?
How was Postgres installed?
So there is a pid file present in the data directory, how about in the run directory, /var/run/postgresql?
Is Postgres running? so:
ps ax|grep post
or
pg_ctl status -D /var/lib/postgresql/9.3/main
If not what happens if you remove the pid and start Postgres manually, not through supervisor?because I've another one docker with the same configuration and works
fine (same os, same versione of postgresql)
On 3 October 2015 at 07:13, Nicolas Paris <niparisco@gmail.com
<mailto:niparisco@gmail.com>> wrote:
Are you using docker on centos ? I had problem with
centos/docker/postgresql because container size was (maybe still is)
limited to 20GB on that specific OS. Maybe not related, but good to
know
2015-10-03 0:03 GMT+02:00 John R Pierce <pierce@hogranch.com
<mailto:pierce@hogranch.com>>:
On 10/2/2015 2:02 PM, Paolo De Michele wrote:
exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D
/var/lib/postgresql/9.3/main -c
config_file=/etc/postgresql/9.3/main/postgresql.conf"
until yesterday there were no problem
right now I see this in the /var/log/supervisor's directory:
2015-10-01 21:40:18 UTC HINT: The file seems accidentally
left over, but it could not be removed. Please remove the
file by hand and try again.
2015-10-01 21:40:20 UTC FATAL: could not remove old lock
file "postmaster.pid": Permission denied
try...
ls -la /var/lib/postgresql/9.3/main
The directory . should be owned by the postgres user, and it
should have 700, 750, or 770 permissions. all the files in it
should also be owned by postgres.
also look and see if postgres logged anything in its own system
log files (/var/log/postgresql/9.3 or whatever).
--
john r pierce, recycling bits in santa cruz
--
Sent via pgsql-general mailing list
(pgsql-general@postgresql.org <mailto:pgsql-general@postgresql.org>)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
--
Adrian Klaver
adrian.klaver@aklaver.com
On 10/04/2015 02:45 AM, Paolo De Michele wrote: > Adrian, > > thanks a lot > about your questions > > 1. postgresql was installed via apt-get install postgresql -y command What OS and version? > 2. in /var/run/postgresql/ there is only one file called 9.3-main.pid > 3. postgres is not running, I did not find nothing with ps command > 4. I remove the postmaster.pid file and I tried to run postgresql Just the postmaster.pid or it and the 9.3-main.pid? > manually. this is the result: > > # /etc/init.d/postgresql start > * Starting PostgreSQL 9.3 database server > The PostgreSQL server failed to start. Please check the log output. > ...fail! Try: pg_ctlcluster 9.3 main start > > I verified in /var/log/postgresql/ and I've not found nothing Nothing as no text at all or nothing as no messages that apply? What does the system log show? > > > > On 3 October 2015 at 15:19, Adrian Klaver <adrian.klaver@aklaver.com > <mailto:adrian.klaver@aklaver.com>> wrote: > > On 10/03/2015 05:50 AM, Paolo De Michele wrote: > > hi there, > > thanks a lot > so, I'm using docker on ubuntu 14.04 > about permissions: > > # ls -la /var/lib/postgresql/9.3/main > total 72 > drwx------ 28 postgres postgres 4096 Oct 3 12:41 . > drwxr-xr-x 4 postgres postgres 4096 Oct 1 11:59 .. > -rwx------ 1 postgres postgres 4 Mar 21 2015 PG_VERSION > drwx------ 12 postgres postgres 4096 Oct 1 11:59 base > drwx------ 2 postgres postgres 4096 Oct 1 11:59 global > drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_clog > drwx------ 6 postgres postgres 4096 Oct 1 11:59 pg_multixact > drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_notify > drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_serial > drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_snapshots > drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_stat > drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_stat_tmp > drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_subtrans > drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_tblspc > drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_twophase > drwx------ 4 postgres postgres 4096 Oct 1 12:18 pg_xlog > -rwx------ 1 postgres postgres 133 Oct 1 20:00 postmaster.opts > -rw------- 1 postgres postgres 50 Oct 3 12:41 postmaster.pid > > in /var/log/postgresql/ I've a file with 0kb > it is weird > is there a way to recover all dbs without starting the daemon? > > > > What OS and version are you running? Assuming a Debian/Ubuntu flavor > for now. > > How was Postgres installed? > > So there is a pid file present in the data directory, how about in > the run directory, /var/run/postgresql? > > Is Postgres running? so: > > ps ax|grep post > > or > > pg_ctl status -D /var/lib/postgresql/9.3/main > > If not what happens if you remove the pid and start Postgres > manually, not through supervisor? > > > because I've another one docker with the same configuration and > works > fine (same os, same versione of postgresql) > > On 3 October 2015 at 07:13, Nicolas Paris <niparisco@gmail.com > <mailto:niparisco@gmail.com> > <mailto:niparisco@gmail.com <mailto:niparisco@gmail.com>>> wrote: > > Are you using docker on centos ? I had problem with > centos/docker/postgresql because container size was (maybe > still is) > limited to 20GB on that specific OS. Maybe not related, > but good to > know > > 2015-10-03 0:03 GMT+02:00 John R Pierce > <pierce@hogranch.com <mailto:pierce@hogranch.com> > <mailto:pierce@hogranch.com <mailto:pierce@hogranch.com>>>: > > On 10/2/2015 2:02 PM, Paolo De Michele wrote: > > exec su postgres -c > "/usr/lib/postgresql/9.3/bin/postgres -D > /var/lib/postgresql/9.3/main -c > config_file=/etc/postgresql/9.3/main/postgresql.conf" > > until yesterday there were no problem > right now I see this in the /var/log/supervisor's > directory: > > 2015-10-01 21:40:18 UTC HINT: The file seems > accidentally > left over, but it could not be removed. Please > remove the > file by hand and try again. > 2015-10-01 21:40:20 UTC FATAL: could not remove > old lock > file "postmaster.pid": Permission denied > > > > try... > ls -la /var/lib/postgresql/9.3/main > > The directory . should be owned by the postgres user, > and it > should have 700, 750, or 770 permissions. all the > files in it > should also be owned by postgres. > > also look and see if postgres logged anything in its > own system > log files (/var/log/postgresql/9.3 or whatever). > > > > -- > john r pierce, recycling bits in santa cruz > > > > -- > Sent via pgsql-general mailing list > (pgsql-general@postgresql.org > <mailto:pgsql-general@postgresql.org> > <mailto:pgsql-general@postgresql.org > <mailto:pgsql-general@postgresql.org>>) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-general > > > > > > -- > Adrian Klaver > adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com> > > -- Adrian Klaver adrian.klaver@aklaver.com
On 10/04/2015 02:45 AM, Paolo De Michele wrote: > Adrian, > > thanks a lot > about your questions > > 1. postgresql was installed via apt-get install postgresql -y command > 2. in /var/run/postgresql/ there is only one file called 9.3-main.pid > 3. postgres is not running, I did not find nothing with ps command > 4. I remove the postmaster.pid file and I tried to run postgresql > manually. this is the result: > > # /etc/init.d/postgresql start > * Starting PostgreSQL 9.3 database server > The PostgreSQL server failed to start. Please check the log output. > ...fail! Also meant to ask, do you still have the supervisor task running with?: autostart=true If so I would suggest changing that to: autostart=false while you are troubleshooting this. > > I verified in /var/log/postgresql/ and I've not found nothing > > -- Adrian Klaver adrian.klaver@aklaver.com