Thread: pg killed by oom-killer, "invalid contrecord length 2190 at A6C/331AAA90" on slaves

Hello,

months ago I have a similar problem with the OOM-Killer.
Have a look at
http://www.credativ.co.uk/credativ-blog/2010/03/postgresql-and-linux-memory-management

I hope that's helpful.

Regards,
basti

On Sat 25.10.2014 22:55 +0200, Joe Van Dyk <joe@tanga.com> wrote:
> One of my postgres backends was killed by the oom-killer. Now, one of my
> streaming replication slaves is reporting "invalid contrecord length
> 2190 at A6C/331AAA90" in the logs and replication has paused. I have
> other streaming replication slaves that are fine.
>
> Is that expected? It's happened twice in two days.
>
> I'm running 9.3.5 on the master. I have 9.3.4 on the slave that has the
> problem, and 9.3.5 on the slave that doesn't have the problem. Is this
> something that was fixed in 9.3.5?
>
> The slave that has the problem is also located across the country, while
> the slave that works is in the same data center as the master -- not
> sure if that's related at all.
>
> Joe

On Mon, Oct 27, 2014 at 8:16 AM, basti <mailinglist@unix-solution.de> wrote:

Hello,

months ago I have a similar problem with the OOM-Killer.
Have a look at
http://www.credativ.co.uk/credativ-blog/2010/03/postgresql-and-linux-memory-management

Thanks -- my question is not so much about the oom killer, but rather about why just one of the slaves is reporting the "invalid contrecord length" error.

 

I hope that's helpful.

Regards,
basti

On Sat 25.10.2014 22:55 +0200, Joe Van Dyk <joe@tanga.com> wrote:
> One of my postgres backends was killed by the oom-killer. Now, one of my
> streaming replication slaves is reporting "invalid contrecord length
> 2190 at A6C/331AAA90" in the logs and replication has paused. I have
> other streaming replication slaves that are fine.
>
> Is that expected? It's happened twice in two days.
>
> I'm running 9.3.5 on the master. I have 9.3.4 on the slave that has the
> problem, and 9.3.5 on the slave that doesn't have the problem. Is this
> something that was fixed in 9.3.5?
>
> The slave that has the problem is also located across the country, while
> the slave that works is in the same data center as the master -- not
> sure if that's related at all.
>
> Joe

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

El 25/10/14 a las 17:55, Joe Van Dyk escibió:
> One of my postgres backends was killed by the oom-killer. Now, one of
> my streaming replication slaves is reporting "invalid contrecord
> length 2190 at A6C/331AAA90" in the logs and replication has paused. I
> have other streaming replication slaves that are fine.
>
> Is that expected? It's happened twice in two days.
>
> I'm running 9.3.5 on the master. I have 9.3.4 on the slave that has
> the problem, and 9.3.5 on the slave that doesn't have the problem. Is
> this something that was fixed in 9.3.5?
>
> The slave that has the problem is also located across the country,
> while the slave that works is in the same data center as the master --
> not sure if that's related at all.
>
> Joe

It's a corrupted slave. You'll need to regenerate it to ensure that your
data is safe.

The OOM killer doesn't respect what it kills, so this certain things can
happen without
a proper kernel configuration.


--
--
Emanuel Calvo                 http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 2014-10-25 13:55:57 -0700, Joe Van Dyk wrote:
> One of my postgres backends was killed by the oom-killer. Now, one of my
> streaming replication slaves is reporting "invalid contrecord length 2190
> at A6C/331AAA90" in the logs and replication has paused. I have other
> streaming replication slaves that are fine.

Is it a LOG or a PANIC message? Because it's not unexpected to see such
messages when reaching the end of the local and/or restore_command
provided WAL.

> I'm running 9.3.5 on the master. I have 9.3.4 on the slave that has the
> problem, and 9.3.5 on the slave that doesn't have the problem. Is this
> something that was fixed in 9.3.5?

We have really no information to answer that question accurately.

So you really need to provide logs and such.

Greetings,

Andres Freund

--
 Andres Freund                       http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Hello,

months ago I have a similar problem with the OOM-Killer.
Have a look at
http://www.credativ.co.uk/credativ-blog/2010/03/postgresql-and-linux-memory-management

I hope that's helpful.

Regards,
basti

On Sat 25.10.2014 22:55 +0200, Joe Van Dyk <joe@tanga.com> wrote:
> One of my postgres backends was killed by the oom-killer. Now, one of my
> streaming replication slaves is reporting "invalid contrecord length
> 2190 at A6C/331AAA90" in the logs and replication has paused. I have
> other streaming replication slaves that are fine.
>
> Is that expected? It's happened twice in two days.
>
> I'm running 9.3.5 on the master. I have 9.3.4 on the slave that has the
> problem, and 9.3.5 on the slave that doesn't have the problem. Is this
> something that was fixed in 9.3.5?
>
> The slave that has the problem is also located across the country, while
> the slave that works is in the same data center as the master -- not
> sure if that's related at all.
>
> Joe