Thread: Postgres won't start. Nothing in the log.

I just did an upgrade on two of my servers (the main and the
failover). The main went OK but the postgres on the failover won't
start.

Unfortunately there is nothing anywhere telling me what the problem
is. The log file is empty, there is nothing in the /var/log/messages
or /var/log/syslog either. The only output I get is this.

* Starting PostgreSQL 8.3 database server
 * The PostgreSQL server failed to start. Please check the log output.


What can I do to figure out why it won't start?

Hi!

pgsql-general-owner@postgresql.org wrote:
> Unfortunately there is nothing anywhere telling me what the
> problem is. The log file is empty, there is nothing in the
> /var/log/messages or /var/log/syslog either. The only output
> I get is this.
>
> * Starting PostgreSQL 8.3 database server
>  * The PostgreSQL server failed to start. Please check the log output.
> What can I do to figure out why it won't start?

Did you check if the directory PostgreSQL wants to write the logfile to
has appropriate rights to allow the user PostgreSQL runs under to write
to it? Another fairly common issue would be a postgresql.conf that is
not readable by the PostgreSQL-user. In case of such an outright failure
to start, it's usually something fairly simple to fix such as file or
directory permissions.

Kind regards

   Markus



Computec Media AG
Sitz der Gesellschaft und Registergericht: Furth (HRB 8818)
Vorstandsmitglieder: Albrecht Hengstenberg (Vorsitzender) und Rainer Rosenbusch
Vorsitzender des Aufsichtsrates: Jurg Marquard
Umsatzsteuer-Identifikationsnummer: DE 812 575 276

Was it a minor-release upgrade? (i.e. 8.3.1 to 8.3.7 and not 8.2. to 8.3)  If not, you would have needed to do a complete dump and restore of your data.

To force a log file, try pg_ctl start -l /path/to/logfile

You might as well use /tmp/pg.log for the logfile for diagnosis.  This should tell you what the problem is.

Thom

"Markus Wollny" <Markus.Wollny@computec.de> writes:
> pgsql-general-owner@postgresql.org wrote:
>> Unfortunately there is nothing anywhere telling me what the
>> problem is. The log file is empty, there is nothing in the
>> /var/log/messages or /var/log/syslog either. The only output
>> I get is this.
>>
>> * Starting PostgreSQL 8.3 database server
>> * The PostgreSQL server failed to start. Please check the log output.
>> What can I do to figure out why it won't start?

> Did you check if the directory PostgreSQL wants to write the logfile to
> has appropriate rights to allow the user PostgreSQL runs under to write
> to it? Another fairly common issue would be a postgresql.conf that is
> not readable by the PostgreSQL-user. In case of such an outright failure
> to start, it's usually something fairly simple to fix such as file or
> directory permissions.

As Markus suggests, it's highly unlikely that Postgres is failing
without any message whatsoever.  What is more likely is that the message
is being sent somewhere you're not looking, or there is a permissions
problem or something with where it was told to send it.  Take a look at
the init script you're using to start the server, and see where it sends
the postmaster's stderr.  You need to check that in case of failures
that occur before the configured log destination has taken effect.
(I have seen some scripts that redirect stderr to /dev/null :-( ...
if you have one of those, change it, and file a bug report with the
packager.)

            regards, tom lane

On Mon, 2009-10-05 at 17:39 +1300, Tim Uckun wrote:

> Unfortunately there is nothing anywhere telling me what the problem
> is. The log file is empty, there is nothing in the /var/log/messages
> or /var/log/syslog either. The only output I get is this.
>
> * Starting PostgreSQL 8.3 database server
>  * The PostgreSQL server failed to start. Please check the log output.


Is this Debian/Ubuntu? If so, the logs should be
at /var/log/postgresql/postgresql-8.3-main.log (or so).
--
Devrim GÜNDÜZ, RHCE
Command Prompt - http://www.CommandPrompt.com
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz

did you install (i should say register) postgres and pg_ctl binaries as postgres user
are you executing postgres and pg_ctl binaries as postgres user
?
Martin Gainty
______________________________________________
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité

Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.

> Subject: Re: [GENERAL] Postgres won't start. Nothing in the log.
> From: devrim@gunduz.org
> To: timuckun@gmail.com
> CC: pgsql-general@postgresql.org
> Date: Mon, 5 Oct 2009 17:16:00 +0300
>
> On Mon, 2009-10-05 at 17:39 +1300, Tim Uckun wrote:
>
> > Unfortunately there is nothing anywhere telling me what the problem
> > is. The log file is empty, there is nothing in the /var/log/messages
> > or /var/log/syslog either. The only output I get is this.
> >
> > * Starting PostgreSQL 8.3 database server
> > * The PostgreSQL server failed to start. Please check the log output.
>
>
> Is this Debian/Ubuntu? If so, the logs should be
> at /var/log/postgresql/postgresql-8.3-main.log (or so).
> --
> Devrim GÜNDÜZ, RHCE
> Command Prompt - http://www.CommandPrompt.com
> devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
> http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz

---

Hotmail: Free, trusted and rich email service. Get it now.

Tim Uckun <timuckun@gmail.com> wrote:

> Unfortunately there is nothing anywhere telling me what the problem
> is. The log file is empty, there is nothing in the /var/log/messages
> or /var/log/syslog either. The only output I get is this.
>
> * Starting PostgreSQL 8.3 database server
>  * The PostgreSQL server failed to start. Please check the log output.
>
>
> What can I do to figure out why it won't start?

I know, from IRC, the problem has been solved, there was no space on the
disk ...

Unfortunately, i haven't logs.


Andreas
--
Really, I'm not out to destroy Microsoft. That will just be a completely
unintentional side effect.                              (Linus Torvalds)
"If I was god, I would recompile penguin with --enable-fly."   (unknown)
Kaufbach, Saxony, Germany, Europe.              N 51.05082°, E 13.56889°

> I know, from IRC, the problem has been solved, there was no space on the
> disk ...
>
> Unfortunately, i haven't logs.
>

Yes. Thanks to everybody on the IRC who helped me out.

The suggestion that was most helpful was to call the posgres binary
directly. /usr/lib/postgresql/8.3/bin/postgres. Calling the
/etc/init.d postgres starter gave me no information at all but calling
the binary told me what the problem was.

In a nutshell somehow the restoring of the log files stopped and the
disk started filling up with the unprocessed log files. Eventually the
disk filled up.

I have since set up monitoring the disk space and the postgres
replication lag using zabbix. The zabbix agent reports the disk space
and I report the replication lag using a ruby script.

In case anybody is using zabbix here is the script.

#! /usr/local/bin/ruby

require 'date'
require 'yaml'
require 'socket'
require 'timeout'
require "base64"

def send_item_to_zabbix (server, port, host, key, value)
    response = ""
    request = "<req><host>"
    request << Base64.encode64(host)
    request << "</host><key>"
    request << Base64.encode64(key)
    request << "</key><data>"
    request << Base64.encode64(value)
    request << "</data></req>"

      status = Timeout::timeout(30){
        socket = TCPSocket.new(server, port)
        socket.puts request
        response = socket.gets
        if response == "OK" || response.nil?
          response = ""
        else
          response = "Monitor rejected key='#{key}' value='#{value}'
host='#{host}' with response #{response}"
        end
      } #timeout
      return response
    rescue Timeout::Error
      return "can't connect to monitoring server at #{server}:#{port}"
    rescue
      return $!
    end #send item to zabbix

output =`/usr/lib/postgresql/8.3/bin/pg_controldata
/var/lib/postgresql/8.3/main`

data = YAML.load output
last_checkpoint = DateTime.parse( data['Time of latest checkpoint'])
hours, mins, secs, ignore_fractions =
Date::day_fraction_to_time(DateTime.now - last_checkpoint)

lag =  hours * 60 * 60 + mins * 60 + secs
output  = send_item_to_zabbix 'my.zabbix..host.com', '10051',
'postgres_host_name'', 'postgres.replication.lag', lag.to_s



~