Thread: "show all" command crashes server

Grant Maxwell wrote:
> Hi folks
>
> First time poster here so please extend grace if I don't initially
> provide what is needed to help.
>
> I am running postgresql 8.3.7 on debian lenny
> (postgresql-8.3_8.3.7-0lenny1_i386.deb)

Well that's useful.

> I have three of these servers and generally they run well.

As is that.

> On this one server if I use the command "show all" in psql, phpPgAdmin
> or pgAdmin3 the postgresql server spits the dummy as follows:

> postgres=# show all;
> server closed the connection unexpectedly

Hmm - some modules can provide their own config variables. Do you have
the same modules installed in all three servers?

Can you "show" individual variables?

--
  Richard Huxton
  Archonet Ltd

On Thu, Sep 10, 2009 at 8:37 AM, Grant
Maxwell<grant.maxwell@maxan.com.au> wrote:
> Hi folks
> First time poster here so please extend grace if I don't initially provide
> what is needed to help.
> I am running postgresql 8.3.7 on debian lenny
> (postgresql-8.3_8.3.7-0lenny1_i386.deb).
> I have three of these servers and generally they run well.
SNIP
> Sep 10 23:55:36 theconsole postgres[31118]: [4-1]     0: LOG:  00000: server
> process (PID 31145) was terminated by signal 11: Segmentation fault

Sig 11 is a process crash which can be caused by bad hardware or
corrupted / buggy binaries.  I'd try reinstalling pgsql binaries and
see if that helps.

On 11/09/2009, at 1:09 AM, Richard Huxton wrote:

>
>
>> On this one server if I use the command "show all" in psql,
>> phpPgAdmin
>> or pgAdmin3 the postgresql server spits the dummy as follows:
>
>> postgres=# show all;
>> server closed the connection unexpectedly
>
> Hmm - some modules can provide their own config variables. Do you have
> the same modules installed in all three servers?

        How can I determine what modules are installed ?
        I do know that pgmemcache is installed on this server - but it was
there before the problems started and it works ok.

>
> Can you "show" individual variables?
        I did a show all on one of the other servers, created a script to
use each of the resulting outputs in a single show statement and ran
on the problem server.
        It ran without a fault.

        I then took the postgresql.conf file from the problem server,
grabbed all the config lines and submitted them one at a time (again
with a script) and it also
        worked fine.

regards
Grant Maxwell

Grant Maxwell <grant.maxwell@maxan.com.au> writes:
> On 11/09/2009, at 1:09 AM, Richard Huxton wrote:
>> Hmm - some modules can provide their own config variables. Do you have
>> the same modules installed in all three servers?

>         How can I determine what modules are installed ?

The contents of the local_preload_libraries and shared_preload_libraries
parameters would probably be enough ...

            regards, tom lane

On 11/09/2009, at 8:17 AM, Tom Lane wrote:

> Grant Maxwell <grant.maxwell@maxan.com.au> writes:
>> On 11/09/2009, at 1:09 AM, Richard Huxton wrote:
>>> Hmm - some modules can provide their own config variables. Do you
>>> have
>>> the same modules installed in all three servers?
>
>>         How can I determine what modules are installed ?
>
> The contents of the local_preload_libraries and
> shared_preload_libraries
> parameters would probably be enough ...
>
>             regards, tom lane
>
On the problem server:
    shared_preload_libraries = 'pgmemcache'
    #local_preload_libraries = ''

on the others both are emply.

For good measure I removed pgmemcache but the problem persists.
I have now put it back.

regards
Grant

Grant Maxwell <grant.maxwell@maxan.com.au> writes:
> On the problem server:
>     shared_preload_libraries = 'pgmemcache'
>     #local_preload_libraries = ''

> on the others both are emply.

Sounds like a smoking gun to me.

> For good measure I removed pgmemcache but the problem persists.

Did you restart the postmaster afterwards?  shared_preload_libraries
is only considered at postmaster start.

            regards, tom lane

On 11/09/2009, at 8:36 AM, Tom Lane wrote:

> Grant Maxwell <grant.maxwell@maxan.com.au> writes:
>> On the problem server:
>>     shared_preload_libraries = 'pgmemcache'
>>     #local_preload_libraries = ''
>
>> on the others both are emply.
>
> Sounds like a smoking gun to me.
>
>> For good measure I removed pgmemcache but the problem persists.
>
> Did you restart the postmaster afterwards?  shared_preload_libraries
> is only considered at postmaster start.

    yep - full restart.
>
>             regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

>> shared_preload_libraries = 'pgmemcache'
...
> Sounds like a smoking gun to me.

Yep, known problem with pgmemcache. Bruce and I poked around
with this about a year ago. Bruce, I think you were going
to throw the problem at some EDB people - did that ever happen?

I seem to recall we fixed that particular problem as well
during the codeathon at OpenSQL Camp.

- --
Greg Sabino Mullane greg@turnstep.com
End Point Corporation
PGP Key: 0x14964AC8 200909102039
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAkqpnIEACgkQvJuQZxSWSsgIPQCgnvLBNKLqeAVcx8r2ufEcNPyF
bZ4An2Ed60lQ1kyokrAoGJFPQm1fwpOQ
=3i3z
-----END PGP SIGNATURE-----

Grant Maxwell <grant.maxwell@maxan.com.au> writes:
> On 11/09/2009, at 8:36 AM, Tom Lane wrote:
>> Did you restart the postmaster afterwards?

>     yep - full restart.

okay, next step is to collect a stack trace ...

            regards, tom lane

Grant Maxwell <grant.maxwell@maxan.com.au> writes:
>     The error was that if "show all" was run on this postgresql (version
> 8.3) server, postgres would crash and then recover.

>     The postgres log showed:
> Sep 10 23:55:36 theconsole postgres[31118]: [4-1]     0: LOG:  00000:
> server process (PID 31145) was terminated by signal 11: Segmentation
> fault

>     We had recently ( a month before) had installed splunk on the server.
> It was running ok
>     The combination of splunk and other tasks running had pushed the
> memory too close.
>     What we did not notice was that swap had been almost completely
> consumed - nasty
>     We shut it all down, increased the memory (double) and voila -
> problem gone.

Hmm.  A segfault in that case seems to indicate that something somewhere
is failing to check for a null result from malloc().  Which is a bug we
ought to fix.  Is there any chance of getting a core dump stack trace
from one of those crashes?

            regards, tom lane