Thread: Postgres no longer starts

On 05/30/12 8:25 AM, Bart Lateur wrote:
>
> After an update on the system, and adding mod_ssl in Apache (is this
> related? No idea.), Postgres no longer starts up. It just fails
> silently. “pgstartup.log” contains only one single line:
>
> runuser: cannot set groups: Operation not permitted
>


did you use yum to install everything? do you use any non-standard
CentOS repositories ?

CentOS 5 came with an older version of PG (8.1, I think?), so where did
you get 8.4 from, and how did you install it originally?

the PGDG packaged postgresql versions for CentOS/RHEL appear to use
`runuser` instead of `su` if selinux is enabled ... so this is likely a
selinux related issue.





--
john r pierce                            N 37, W 122
santa cruz ca                         mid-left coast

(Sorry if message threading is a bit off; I'm replying from a different
mail account as the previous post, so the "ref" header won't match.)

Well I tried

    /usr/bin/pg_ctl -D /var/lib/pgsql/data start -l /tmp/pglogfile

and it just says

    server starting

and then... nothing. It seems to quit immediately.

The logfile, /tmp/logfile exists, but is empty. (I noticed the previous
logfile, "pgstartup.log", was still the same and hadn't changed since a
few hours earlier, so the "modified" timestamp confirmed, so it wasn't
being updated any more.)

As where I did get the installer of Postgres for CentOS: I don't quite
remember the details, as it has been running well for several months.
But I got the idea from one of the many blog posts about it, such as
these:
http://blog.lystor.org.ua/2010/05/upgrading-postgresql-81-to-84-centos-55.html
http://www.if-not-true-then-false.com/2010/howto-install-postgresql-8-4-database-server-on-centos-fedora-red-hat/

I think it just comes from one of the unofficial CentOS repositories.

Like I said, it has been running well for months and the latest upgrade
was unrelated to PostGres.

It may have been some manual tweak I did a few weeks ago... PostGres
hadn't been restarted since then.

If  I knew what it was complaining about, I might be able to fix it.

I have already made a backup of the /var/lib/pgsql/data subdirectory, I
might just remove all of Postgres, and then reinstall.

Luckily this is a development machine, but as we don't know what causes
the problem we fear we might one day face the exact same problem where
it does matter: on a production machine. So we'd like to know exactly
what went wrong...

--
    Bart Lateur
    bart.lateur@telenet.be

Bart:

Failing a more definitive diagnostic approach, I suggest that you post
your entire pgstartup.log rather than just the error message.  My guess
is that the position in that log where the error occurs will give folks
who are more familiar with the startup sequence a reasonable idea of
where in the start process things failed and may at least help you to
localize your search for problems.

Good luck,

John

On Wednesday, May 30, 2012 08:22:58 PM Bart Lateur wrote:
> Luckily this is a development machine, but as we don't know what causes
> the problem we fear we might one day face the exact same problem where
> it does matter: on a production machine. So we'd like to know exactly
> what went wrong..

Change selinux to permissive instead of enforcing and see if PostgreSQL then
runs. If it does, you can look at the selinux logging to see what would have
been denied in enforcing mode, and hopefully fix it from there.

Alan Hodgson <ahodgson@simkin.ca> writes:
> On Wednesday, May 30, 2012 08:22:58 PM Bart Lateur wrote:
>> Luckily this is a development machine, but as we don't know what causes
>> the problem we fear we might one day face the exact same problem where
>> it does matter: on a production machine. So we'd like to know exactly
>> what went wrong..

> Change selinux to permissive instead of enforcing and see if PostgreSQL then
> runs. If it does, you can look at the selinux logging to see what would have
> been denied in enforcing mode, and hopefully fix it from there.

Yeah, I concur that this smells like a selinux issue.  Most likely, the
software update you did messed up the selinux "context" settings for
some files.  restorecon should be able to fix it for you, if so.
As Alan says, the kernel log (or separate avc log, depending on how your
system is set up) should show evidence of the problem if this is where
it is.

            regards, tom lane

> Alan Hodgson <ahodgson@simkin.ca> writes:
> > On Wednesday, May 30, 2012 08:22:58 PM Bart Lateur wrote:
> >> Luckily this is a development machine, but as we don't know what
> >> causes the problem we fear we might one day face the exact same
> >> problem where it does matter: on a production machine. So we'd like
> >> to know exactly what went wrong..
>
> > Change selinux to permissive instead of enforcing and see if
> > PostgreSQL then runs. If it does, you can look at the selinux logging
> > to see what would have been denied in enforcing mode, and hopefully fix
it from there.
>
> Yeah, I concur that this smells like a selinux issue.  Most likely, the
software update you did messed up the selinux "context" settings for some
files.  restorecon should be able to fix it for you, if so.
> As Alan says, the kernel log (or separate avc log, depending on how your
system is set up) should show evidence of the problem if this is where it
is.

I thought the same, and I was happy to go in and fix it, but then I found
that SElinux was not even enabled.

Hunting around for more logs I finally found a recently updated log file in
the subdirectory /var/lib/pgsql/data/pg_log/. And there I found the message
that pg_hba.conf could not be loaded due to a syntax error in it. Weird, it
was running before...?

It's suboptimal that starting Postgres fails silently. It's also less than
optimal that the location of the log files is a bit of a secret. It's also
suboptimal that Postgres refuses to run because it doesn't understand 1 line
in pg_hba.conf. After all, it's just a data grid, not prose...

Anyway, hint for Postgres newbies (or at least, people who don't spend whole
days administering Postgres, which is about everybody, I guess): find the
logs. They're in the subdirectory pg_log and they have names like
"postgresql-DDD.log" where DDD is the three letter name of the day. The
currently active log is one of them.

Thanks a lot, all of you who replied.

--
    Bart Lateur.

On 05/31/2012 12:45 AM, Bart Lateur wrote:
>> Alan Hodgson<ahodgson@simkin.ca>  writes:
>>> On Wednesday, May 30, 2012 08:22:58 PM Bart Lateur wrote:
>>>> Luckily this is a development machine, but as we don't know what
>>>> causes the problem we fear we might one day face the exact same
>>>> problem where it does matter: on a production machine. So we'd like
>>>> to know exactly what went wrong..
>>
>>> Change selinux to permissive instead of enforcing and see if
>>> PostgreSQL then runs. If it does, you can look at the selinux logging
>>> to see what would have been denied in enforcing mode, and hopefully fix
> it from there.
>>
>> Yeah, I concur that this smells like a selinux issue.  Most likely, the
> software update you did messed up the selinux "context" settings for some
> files.  restorecon should be able to fix it for you, if so.
>> As Alan says, the kernel log (or separate avc log, depending on how your
> system is set up) should show evidence of the problem if this is where it
> is.
>
> I thought the same, and I was happy to go in and fix it, but then I found
> that SElinux was not even enabled.
>
> Hunting around for more logs I finally found a recently updated log file in
> the subdirectory /var/lib/pgsql/data/pg_log/. And there I found the message
> that pg_hba.conf could not be loaded due to a syntax error in it. Weird, it
> was running before...?
>
> It's suboptimal that starting Postgres fails silently. It's also less than
> optimal that the location of the log files is a bit of a secret. It's also
> suboptimal that Postgres refuses to run because it doesn't understand 1 line
> in pg_hba.conf. After all, it's just a data grid, not prose...

Actually it did not fail silently, it recorded it to the log. The
location of the log file is no great secret, open postgresql.conf(the
server confog file) and look for log_directory. pg_hba.conf is access
control and the first line of defense in security. I would prefer that
it fail closed instead of open.

>
> Anyway, hint for Postgres newbies (or at least, people who don't spend whole
> days administering Postgres, which is about everybody, I guess): find the
> logs. They're in the subdirectory pg_log and they have names like
> "postgresql-DDD.log" where DDD is the three letter name of the day. The
> currently active log is one of them.

Not necessarily. It depends on how Postgres was packaged. See hint above
about looking in postgresql.conf.

>
> Thanks a lot, all of you who replied.
>


--
Adrian Klaver
adrian.klaver@gmail.com

came across this problem myself.

turned out after much playing around that it was a change to the pg_hba.conf

was a syntax error in the all all posgres trust sameuser line.
deleted it and postgres fired up from /etc/init.d or as a service.

just my very late twopenneth....




--
View this message in context: http://postgresql.1045698.n5.nabble.com/Postgres-no-longer-starts-tp5710560p5731011.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.