Re: emergency outage requiring database restart - Mailing list pgsql-hackers

From Oskari Saarenmaa
Subject Re: emergency outage requiring database restart
Date
Msg-id 5b96f1e3-c958-6bc3-6a1a-a7533a28ee6d@ohmu.fi
Whole thread Raw
In response to Re: emergency outage requiring database restart  (Merlin Moncure <mmoncure@gmail.com>)
Responses Re: emergency outage requiring database restart  (Merlin Moncure <mmoncure@gmail.com>)
List pgsql-hackers
27.10.2016, 21:53, Merlin Moncure kirjoitti:
> As noted earlier, I was not able to reproduce the issue with
> crashme.sh, which was:
>
> NUM_FORKS=16
> do_parallel psql -p 5432  -c"select PushMarketSample('1740')" castaging_test
> do_parallel psql -p 5432  -c"select PushMarketSample('4400')" castaging_test
> do_parallel psql -p 5432  -c"select PushMarketSample('2160')" castaging_test
> do_parallel psql -p 5432  -c"select PushMarketSample('6680')" castaging_test
> <snip>
>
> (do_parallel is simple wrapper to executing the command in parallel up
> to NUM_FORKS).   This is on the same server and cluster as above.
> This kind of suggests that either
> A) there is some concurrent activity from another process that is
> tripping the issue
> or
> B) there is something particular to the session invoking the function
> that is participating in the problem.  As the application is
> structured, a single threaded node.js app is issuing the query that is
> high traffic and long lived.  It's still running in fact and I'm kind
> of tempted to find some downtime to see if I can still reproduce via
> the UI.

Your production system's postgres backends probably have a lot more open 
files associated with them than the simple test case does.  Since 
Postgres likes to keep files open as long as possible and only closes 
them when you need to free up fds to open new files, it's possible that 
your production backends have almost all allowed fds used when you 
execute your pl/sh function.

If that's the case, the sqsh process that's executed may not have enough 
fds to do what it wanted to do and because of busted error handling 
could end up writing to fds that were opened by Postgres and point to 
$PGDATA files.

/ Oskari



pgsql-hackers by date:

Previous
From: Kuntal Ghosh
Date:
Subject: Re: WAL consistency check facility
Next
From: Andres Freund
Date:
Subject: Re: Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)