Hi Adrian,
thank you very much for your patience. I apologise for the missing information.
On 9 March 2016 16:13:00 +01:00, Adrian Klaver <adrian.klaver@aklaver.com> wrote:
Hi Adrian,
thank you very much for your response.
I ran the "VACUUM ANALYZE" command on the master node.
Regarding log messages.
Here is the contents of the log (excluding connections/disconnections):
Assuming the below is from the replica database.
the "LOG: recovery was paused" message was indeed from the replica.
2016-02-22 02:30:08 GMT 24616 LOG: recovery has paused
So what happened to cause the above?
we automatically pause recovery on the replica before running pg_dump. This is in order to make certain that we get a consistent dump of the database.
I am not seeing anything below that indicates the recovery started again.
the reason why we do not see a matching "resume" is that the pg_dump failed and our error handling was insufficient.
2016-02-22 02:30:08 GMT 24616 HINT: Execute pg_xlog_replay_resume() to
continue.
2016-02-22 02:37:19 GMT 23859 DBNAME ERROR: missing chunk number 0 for
toast value 2747579 in pg_toast_22066
2016-02-22 02:37:19 GMT 23859 DBNAME STATEMENT: COPY public.room_shape
(room_uuid, data) TO stdout;
2016-02-22 02:37:41 GMT 2648 DBNAME LOG: could not receive data from
client: Connection reset by peer
2016-02-22 02:37:41 GMT 2648 DBNAME LOG: unexpected EOF on client
connection
What does the log from the master show?
It doesnt seem to show much. It does have these repeated messages, however:
2016-02-22 02:12:18 GMT 30908 LOG: using stale statistics instead of current ones because stats collector is not responding
2016-02-22 02:13:01 GMT 30908 LOG: using stale statistics instead of current ones because stats collector is not responding
2016-02-22 02:13:52 GMT 30908 LOG: using stale statistics instead of current ones because stats collector is not responding
There are lots of these mesages within the timeframe. There seems to be a couple of them every 2-4 hours.
Best regards,
Fredrik Huitfeldt
On 7 March 2016 16:35:29 +01:00, Adrian Klaver
HI All,
i would really appreciate any help I can get on this issue.
basically, a pg_basebackup + streaming attach, led to a database
that we
could not read from afterwards.
From original post:
"The issue remained until we ran a full vacuum analyze on the cluster."
Which cluster was that, the master or the slave?
"I have logfiles from the incident, but I cannot see anything out of
the ordinary (despite having a fair amount of experience investigating
postgresql logs)."
Can we see the section before and after ERROR?
Beset regards,
Fredrik
PS please advise if this is better posted on another list.
--
Adrian Klaver
--
Adrian Klaver
Best regards,
Fredrik
