Thread: postmaster never finishes starting up, silent to boot
Greetings, I've gotten myself in a pickle and had a postgresql (8.2) instance fill its disk completely and shutdown itself down. I've moved the entire data directory to a new, larger slice however postmaster never finishes "starting". Despite configuring postgresql.conf for excessive 'verboseness' nothing gets outputted to syslog or the --log specified file. I have a feeling I'm just not hitting the right search terms, but shouldn't I be able to simply start a fully copied data directory without issue? at the very least I'd expect some kind of output to some kind of log. I have to kill it with a 'pg_ctl stop -D /mnt/data -m i' -- immediate is the only one that actually kills it; and I get this in syslog: Mar 17 22:36:49 rtg postgres[1879]: [8-1] WARNING: terminating connection because of crash of another server process Mar 17 22:36:49 rtg postgres[1879]: [8-2] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server Mar 17 22:36:49 rtg postgres[1879]: [8-3] process exited abnormally and possibly corrupted shared memory. Mar 17 22:36:49 rtg postgres[1879]: [8-4] HINT: In a moment you should be able to reconnect to the database and repeat your command. Mar 17 22:36:49 rtg postgres[1879]: [8-5] CONTEXT: xlog redo zeropage: 16645 there are no other postgres instances running on this machine; actually there is nothing else but the OS running on this machine. Appreciate a cluebat hit. thanks, aaron.glenn
On Tue, 17 Mar 2009, Aaron Glenn wrote: > Despite configuring postgresql.conf for excessive 'verboseness' nothing > gets outputted to syslog or the --log specified file. You shouldn't trust those destinations for getting really unusual errors starting the server. Change your log_destination temporarily back to just "stderr" and start the server with a simple "pg_ctl start". Errors will show up on the terminal. If that works but doesn't tell you anything interesting about why startup isn't going correctly, try changing log_min_messages=debug2 ; that will show you a bunch more information about what's happening. If you didn't delete the pid file after the crash or the old server is still holding onto some shared memory/ports, that might be causing your problem. Another possibility is that you don't have the right permissions on the copy. Once you get the log output on the screen it should narrow the possibilities here. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
On top of what the other poster said, I'm wondering if you're not getting any kind of "postmaster not cleanly shutdown, recovery initiated or something like that when you first start it up. You don't tend to see a lot of messages after that until recovery is completed. What does top and / or vmstat or other system monitoring tools say about activity? I'm guessing you just need more patience while pgsql recovers from the unclean shutdown. But it could be something more. Hard to say with what you've told us so far.
On Tue, Mar 17, 2009 at 11:27 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote: > Hard to say with what you've told us so far. what more should I post/need? I was suspecting that as well as I've never had postgres be silent and not work -- I've also never let a db fill its disk and get f'ed like this. should I just let the pg_ctl start run it's course? for a 35GB+ database how long should I wait? is there no way to log the status of what the postgres daemon is actually doing while I wait? what's the standard course of action for a postgres instance that filled its disk and shut itself down -- if there is one? apologies for the admittedly amateur questions but I haven't been able to find much for the situation I'm in. thanks, aaron
On Wed, Mar 18, 2009 at 2:18 AM, Aaron Glenn <aaron.glenn@gmail.com> wrote: > On Tue, Mar 17, 2009 at 11:27 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote: >> Hard to say with what you've told us so far. > > what more should I post/need? I was suspecting that as well as I've Remember that mentiion of vmstat and top I made in my last post? That's a good starting point. > never had postgres be silent and not work -- I've also never let a db > fill its disk and get f'ed like this. should I just let the pg_ctl > start run it's course? for a 35GB+ database how long should I wait? is > there no way to log the status of what the postgres daemon is actually > doing while I wait? what's the standard course of action for a > postgres instance that filled its disk and shut itself down -- if > there is one? Waiting is pretty much it, assuming it's doing something. > apologies for the admittedly amateur questions but I haven't been able > to find much for the situation I'm in. Yeah, it's not that common. So, is the machine truly sitting idle during this period or is it trying to recover is the real question.
On Wed, Mar 18, 2009 at 01:18:46AM -0700, Aaron Glenn wrote: > On Tue, Mar 17, 2009 at 11:27 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote: > start run it's course? for a 35GB+ database how long should I wait? is > there no way to log the status of what the postgres daemon is actually > doing while I wait? what's the standard course of action for a use strace or dtrace to connect to the pid. It should give you a clue.