Thread: Postgres won't start. Nothing in the log.

Postgres won't start. Nothing in the log.

From
Tim Uckun
Date:
I just did an upgrade on two of my servers (the main and the
failover). The main went OK but the postgres on the failover won't
start.

Unfortunately there is nothing anywhere telling me what the problem
is. The log file is empty, there is nothing in the /var/log/messages
or /var/log/syslog either. The only output I get is this.

* Starting PostgreSQL 8.3 database server
 * The PostgreSQL server failed to start. Please check the log output.


What can I do to figure out why it won't start?

Re: Postgres won't start. Nothing in the log.

From
"Markus Wollny"
Date:
Hi!

pgsql-general-owner@postgresql.org wrote:
> Unfortunately there is nothing anywhere telling me what the
> problem is. The log file is empty, there is nothing in the
> /var/log/messages or /var/log/syslog either. The only output
> I get is this.
>
> * Starting PostgreSQL 8.3 database server
>  * The PostgreSQL server failed to start. Please check the log output.
> What can I do to figure out why it won't start?

Did you check if the directory PostgreSQL wants to write the logfile to
has appropriate rights to allow the user PostgreSQL runs under to write
to it? Another fairly common issue would be a postgresql.conf that is
not readable by the PostgreSQL-user. In case of such an outright failure
to start, it's usually something fairly simple to fix such as file or
directory permissions.

Kind regards

   Markus



Computec Media AG
Sitz der Gesellschaft und Registergericht: Furth (HRB 8818)
Vorstandsmitglieder: Albrecht Hengstenberg (Vorsitzender) und Rainer Rosenbusch
Vorsitzender des Aufsichtsrates: Jurg Marquard
Umsatzsteuer-Identifikationsnummer: DE 812 575 276



Re: Postgres won't start. Nothing in the log.

From
Thom Brown
Date:
2009/10/5 Markus Wollny <Markus.Wollny@computec.de>
Hi!

pgsql-general-owner@postgresql.org wrote:
> Unfortunately there is nothing anywhere telling me what the
> problem is. The log file is empty, there is nothing in the
> /var/log/messages or /var/log/syslog either. The only output
> I get is this.
>
> * Starting PostgreSQL 8.3 database server
>  * The PostgreSQL server failed to start. Please check the log output.
> What can I do to figure out why it won't start?



Was it a minor-release upgrade? (i.e. 8.3.1 to 8.3.7 and not 8.2. to 8.3)  If not, you would have needed to do a complete dump and restore of your data.

To force a log file, try pg_ctl start -l /path/to/logfile

You might as well use /tmp/pg.log for the logfile for diagnosis.  This should tell you what the problem is.

Thom

Re: Postgres won't start. Nothing in the log.

From
Tom Lane
Date:
"Markus Wollny" <Markus.Wollny@computec.de> writes:
> pgsql-general-owner@postgresql.org wrote:
>> Unfortunately there is nothing anywhere telling me what the
>> problem is. The log file is empty, there is nothing in the
>> /var/log/messages or /var/log/syslog either. The only output
>> I get is this.
>>
>> * Starting PostgreSQL 8.3 database server
>> * The PostgreSQL server failed to start. Please check the log output.
>> What can I do to figure out why it won't start?

> Did you check if the directory PostgreSQL wants to write the logfile to
> has appropriate rights to allow the user PostgreSQL runs under to write
> to it? Another fairly common issue would be a postgresql.conf that is
> not readable by the PostgreSQL-user. In case of such an outright failure
> to start, it's usually something fairly simple to fix such as file or
> directory permissions.

As Markus suggests, it's highly unlikely that Postgres is failing
without any message whatsoever.  What is more likely is that the message
is being sent somewhere you're not looking, or there is a permissions
problem or something with where it was told to send it.  Take a look at
the init script you're using to start the server, and see where it sends
the postmaster's stderr.  You need to check that in case of failures
that occur before the configured log destination has taken effect.
(I have seen some scripts that redirect stderr to /dev/null :-( ...
if you have one of those, change it, and file a bug report with the
packager.)

            regards, tom lane

Re: Postgres won't start. Nothing in the log.

From
Devrim GÜNDÜZ
Date:
On Mon, 2009-10-05 at 17:39 +1300, Tim Uckun wrote:

> Unfortunately there is nothing anywhere telling me what the problem
> is. The log file is empty, there is nothing in the /var/log/messages
> or /var/log/syslog either. The only output I get is this.
>
> * Starting PostgreSQL 8.3 database server
>  * The PostgreSQL server failed to start. Please check the log output.


Is this Debian/Ubuntu? If so, the logs should be
at /var/log/postgresql/postgresql-8.3-main.log (or so).
--
Devrim GÜNDÜZ, RHCE
Command Prompt - http://www.CommandPrompt.com
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz

Attachment

Re: Postgres won't start. Nothing in the log.

From
Martin Gainty
Date:
did you install (i should say register) postgres and pg_ctl binaries as postgres user
are you executing postgres and pg_ctl binaries as postgres user
?
Martin Gainty
______________________________________________
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
 
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.





> Subject: Re: [GENERAL] Postgres won't start. Nothing in the log.
> From: devrim@gunduz.org
> To: timuckun@gmail.com
> CC: pgsql-general@postgresql.org
> Date: Mon, 5 Oct 2009 17:16:00 +0300
>
> On Mon, 2009-10-05 at 17:39 +1300, Tim Uckun wrote:
>
> > Unfortunately there is nothing anywhere telling me what the problem
> > is. The log file is empty, there is nothing in the /var/log/messages
> > or /var/log/syslog either. The only output I get is this.
> >
> > * Starting PostgreSQL 8.3 database server
> > * The PostgreSQL server failed to start. Please check the log output.
>
>
> Is this Debian/Ubuntu? If so, the logs should be
> at /var/log/postgresql/postgresql-8.3-main.log (or so).
> --
> Devrim GÜNDÜZ, RHCE
> Command Prompt - http://www.CommandPrompt.com
> devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
> http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz


Hotmail: Free, trusted and rich email service. Get it now.

Re: Postgres won't start. Nothing in the log.

From
Andreas Kretschmer
Date:
Tim Uckun <timuckun@gmail.com> wrote:

> Unfortunately there is nothing anywhere telling me what the problem
> is. The log file is empty, there is nothing in the /var/log/messages
> or /var/log/syslog either. The only output I get is this.
>
> * Starting PostgreSQL 8.3 database server
>  * The PostgreSQL server failed to start. Please check the log output.
>
>
> What can I do to figure out why it won't start?

I know, from IRC, the problem has been solved, there was no space on the
disk ...

Unfortunately, i haven't logs.


Andreas
--
Really, I'm not out to destroy Microsoft. That will just be a completely
unintentional side effect.                              (Linus Torvalds)
"If I was god, I would recompile penguin with --enable-fly."   (unknown)
Kaufbach, Saxony, Germany, Europe.              N 51.05082°, E 13.56889°

Re: Postgres won't start. Nothing in the log.

From
Tim Uckun
Date:
> I know, from IRC, the problem has been solved, there was no space on the
> disk ...
>
> Unfortunately, i haven't logs.
>

Yes. Thanks to everybody on the IRC who helped me out.

The suggestion that was most helpful was to call the posgres binary
directly. /usr/lib/postgresql/8.3/bin/postgres. Calling the
/etc/init.d postgres starter gave me no information at all but calling
the binary told me what the problem was.

In a nutshell somehow the restoring of the log files stopped and the
disk started filling up with the unprocessed log files. Eventually the
disk filled up.

I have since set up monitoring the disk space and the postgres
replication lag using zabbix. The zabbix agent reports the disk space
and I report the replication lag using a ruby script.

In case anybody is using zabbix here is the script.

#! /usr/local/bin/ruby

require 'date'
require 'yaml'
require 'socket'
require 'timeout'
require "base64"

def send_item_to_zabbix (server, port, host, key, value)
    response = ""
    request = "<req><host>"
    request << Base64.encode64(host)
    request << "</host><key>"
    request << Base64.encode64(key)
    request << "</key><data>"
    request << Base64.encode64(value)
    request << "</data></req>"

      status = Timeout::timeout(30){
        socket = TCPSocket.new(server, port)
        socket.puts request
        response = socket.gets
        if response == "OK" || response.nil?
          response = ""
        else
          response = "Monitor rejected key='#{key}' value='#{value}'
host='#{host}' with response #{response}"
        end
      } #timeout
      return response
    rescue Timeout::Error
      return "can't connect to monitoring server at #{server}:#{port}"
    rescue
      return $!
    end #send item to zabbix

output =`/usr/lib/postgresql/8.3/bin/pg_controldata
/var/lib/postgresql/8.3/main`

data = YAML.load output
last_checkpoint = DateTime.parse( data['Time of latest checkpoint'])
hours, mins, secs, ignore_fractions =
Date::day_fraction_to_time(DateTime.now - last_checkpoint)

lag =  hours * 60 * 60 + mins * 60 + secs
output  = send_item_to_zabbix 'my.zabbix..host.com', '10051',
'postgres_host_name'', 'postgres.replication.lag', lag.to_s



~