Thread: [GENERAL] openvz and shared memory trouble

Zitat von Willy-Bas Loos <willybas@gmail.com>:

> Hi,
>
> I have a problem with postgres in combination with openvz.
> My hot standby crashes on me when i give it a normal value for
> shared_buffers (4GB, which 25% of the memory in the container).
> When i tune down the shared_buffers, it works again. But of course this is
> far from desireable.

Have a look at the bean counters from OpenVZ. Some of them have
dependencies to each other and can lead to unexpected out-of-memory
condition inside the container. Check the logs for the reason
PostgreSQL crashed, maybe it was the OOM killer.

> Now, i've seen a couple more messages around the net about openVZ and
> postgres that weren't very encouraging:
>
> http://www.postgresql.org/message-id/CAHyXU0xa5EgvjeH=4vp-eZDJdS5kMQuiDivvTRLjY-uZ62Y44w@mail.gmail.com

This is about Postgres not able to read a block from a file, so it
looks like a filesystem issue to me.

> http://www.postgresql.org/message-id/23206.1254683408@sss.pgh.pa.us

This is about "hot" migration which is in general a dangerous task for
databases. Obviously it doesn't work in this case. We also had no
success some time ago trying "hot" migration with Oracle Databases on
OpenVZ.

>
http://postgresql.1045698.n5.nabble.com/BUG-9721-Fatal-error-on-startup-no-free-slots-in-PMChildFlags-array-td5797390.html

This is also a related problem to the above. Freezing a running
database (which is also done for migration) must be atomic, else the
internal database states might get confused.

> But i am in a position where we already have a pretty strong binding with
> openVZ.
> I also have a colleague that has been using openVZ and postgres on
> production for quite a while now, without any problems. And already 1 of
> our servers is running a production database in openVZ that is well used,
> without any trouble of this kind.
>
> Now, what i would like to know is:
>
>    - is there clarity about what the issues are at all
>    - is it just 1 problem (in openVZ) that make openVZ and postgres so
>    poorly compatible?
>    - how can i stay out of trouble (when using postgres inside openVZ)
>    - who else has trouble with openVZ in combination with postgres, and
>    what constitutes the trouble?

 From my point of view your problem is not related to the URLs shown.
You have to carefully check the Container settings and errors and
avoid freezing/migrating the database container if possible.

In general there is no Problem with databases inside Containers but
you have to carefully check the settings for the container in question
and test the advanced features like hot migration if you really have
to use them.

Regards

Andreas

On 03/28/2014 01:51 AM, Willy-Bas Loos wrote:
> Hi,
>
> I have a problem with postgres in combination with openvz.
> My hot standby crashes on me when i give it a normal value for
> shared_buffers (4GB, which 25% of the memory in the container).
> When i tune down the shared_buffers, it works again. But of course this
> is far from desireable.

What is the error that shows up in the logs when the 'machine' crashes?

25% is a suggested value, not an absolute value. Desirable would seem to
be a value that works for your situation and maintains performance. Is
there any indication that running with a lower value adversely affects
performance?



>
> Now, i've seen a couple more messages around the net about openVZ and
> postgres that weren't very encouraging:
>
> http://www.postgresql.org/message-id/CAHyXU0xa5EgvjeH=4vp-eZDJdS5kMQuiDivvTRLjY-uZ62Y44w@mail.gmail.com
> http://www.postgresql.org/message-id/23206.1254683408@sss.pgh.pa.us
>
http://postgresql.1045698.n5.nabble.com/BUG-9721-Fatal-error-on-startup-no-free-slots-in-PMChildFlags-array-td5797390.html
>
> But i am in a position where we already have a pretty strong binding
> with openVZ.
> I also have a colleague that has been using openVZ and postgres on
> production for quite a while now, without any problems. And already 1 of
> our servers is running a production database in openVZ that is well
> used, without any trouble of this kind.
>
> Now, what i would like to know is:
>
>   * is there clarity about what the issues are at all
>   * is it just 1 problem (in openVZ) that make openVZ and postgres so
>     poorly compatible?
>   * how can i stay out of trouble (when using postgres inside openVZ)
>   * who else has trouble with openVZ in combination with postgres, and
>     what constitutes the trouble?

A quick search found this:

http://forum.openvz.org/index.php?t=msg&goto=12061&&srch=postgresql+shared#msg_12061

and

http://openvz.org/FAQ#User_Beancounters_.28UBC.29


>
> regards,
>
> WBL
>
> --
> "Quality comes from focus and clarity of purpose" -- Mark Shuttleworth


--
Adrian Klaver
adrian.klaver@aklaver.com

On 03/28/2014 02:21 PM, Willy-Bas Loos wrote:
> Hi,
>
> I have a problem with postgres in combination with openvz.
> My hot standby crashes on me when i give it a normal value for
> shared_buffers (4GB, which 25% of the memory in the container).
> When i tune down the shared_buffers, it works again. But of course this is
> far from desireable.
>
> Now, i've seen a couple more messages around the net about openVZ and
> postgres that weren't very encouraging:
>
> http://www.postgresql.org/message-id/CAHyXU0xa5EgvjeH=4vp-eZDJdS5kMQuiDivvTRLjY-uZ62Y44w@mail.gmail.com
> http://www.postgresql.org/message-id/23206.1254683408@sss.pgh.pa.us
>
http://postgresql.1045698.n5.nabble.com/BUG-9721-Fatal-error-on-startup-no-free-slots-in-PMChildFlags-array-td5797390.html
>
> But i am in a position where we already have a pretty strong binding with
> openVZ.
> I also have a colleague that has been using openVZ and postgres on
> production for quite a while now, without any problems. And already 1 of
> our servers is running a production database in openVZ that is well used,
> without any trouble of this kind.
>
> Now, what i would like to know is:
>
>    - is there clarity about what the issues are at all
>    - is it just 1 problem (in openVZ) that make openVZ and postgres so
>    poorly compatible?
>    - how can i stay out of trouble (when using postgres inside openVZ)

Try increasing the share memory "shmpages" of container from host.


>    - who else has trouble with openVZ in combination with postgres, and
>    what constitutes the trouble?
>
> regards,
>
> WBL
>
>

On 03/29/2014 08:19 AM, Willy-Bas Loos wrote:
> The error that shows up is a Bus error.
> That's on the replication slave.
> Here's the log about it:
> 2014-03-29 12:41:33 CET db: ip: us: FATAL:  could not receive data from
> WAL stream: server closed the connection unexpectedly
>          This probably means the server terminated abnormally
>          before or while processing the request.
>
> cp: cannot stat
> `/data/postgresql/9.1/main/wal_archive/00000001000000720000000A': No
> such file or directory
> 2014-03-29 12:41:33 CET db: ip: us: LOG:  unexpected pageaddr
> 71/E9DA0000 in log file 114, segment 10, offset 14286848
> cp: cannot stat
> `/data/postgresql/9.1/main/wal_archive/00000001000000720000000A': No
> such file or directory
> 2014-03-29 12:41:33 CET db: ip: us: LOG:  streaming replication
> successfully connected to primary
> 2014-03-29 12:41:48 CET db: ip: us: LOG:  startup process (PID 17452)
> was terminated by signal 7: Bus error
> 2014-03-29 12:41:48 CET db: ip: us: LOG:  terminating any other active
> server processes
> 2014-03-29 12:41:48 CET db:wbloos ip:[local] us:wbloos WARNING:
> terminating connection because of crash of another server process
> 2014-03-29 12:41:48 CET db:wbloos ip:[local] us:wbloos DETAIL:  The
> postmaster has commanded this server process to roll back the current
> transaction and exit, because another server process exited abnormally
> and possibly corrupted shared memory.
> 2014-03-29 12:41:48 CET db:wbloos ip:[local] us:wbloos HINT:  In a
> moment you should be able to reconnect to the database and repeat your
> command.
>

Well what I am seeing are WAL log errors. One saying no file is
present, the other pointing at a possible file corruption. Shared memory
problems are offered as a possible cause only. Right now I would say we
are seeing only half the picture. The Postgres logs from the same time
period for the primary server, as well as the system logs for the openvz
container would help fill in the other half of the picture.



>
>
> On Fri, Mar 28, 2014 at 2:52 PM, Adrian Klaver
> <adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>> wrote:
>
>
>     25% is a suggested value, not an absolute value. Desirable would
>     seem to be a value that works for your situation and maintains
>     performance. Is there any indication that running with a lower value
>     adversely affects performance?
>
>

>
> Thanks, i'm looking into the bean counters more.
> Maybe there is something to won by guaranteeing the shared memory
> resources, instead of just refraining from constraining them.
>

>
> Ok, that sounds reassuring, and i agree that the other posts are not
> really related. It's just that i found several posts that boiled down to
> memory related issues with openVZ, and experts complaining. That made me
> worry.

A cursory look at memory management in openvz shows it is different from
other virtualization software and physical machines. Whether that is a
problem would seem to be dependent on where you are on the learning
curve:) As to 'experts' complaining, I have heard a lot of that in many
different fields and every once in while they are right, so it tends to
be low on my check list when it comes to doing things.


--
Adrian Klaver
adrian.klaver@aklaver.com

Zitat von Willy-Bas Loos <willybas@gmail.com>:

> On Sat, Mar 29, 2014 at 6:17 PM, Adrian Klaver
> <adrian.klaver@aklaver.com>wrote:
>
>> On 03/29/2014 08:19 AM, Willy-Bas Loos wrote:
>>
>>> The error that shows up is a Bus error.
>>> That's on the replication slave.
>>> Here's the log about it:
>>> 2014-03-29 12:41:33 CET db: ip: us: FATAL:  could not receive data from
>>> WAL stream: server closed the connection unexpectedly
>>>          This probably means the server terminated abnormally
>>>          before or while processing the request.
>>>
>>> cp: cannot stat
>>> `/data/postgresql/9.1/main/wal_archive/00000001000000720000000A': No
>>> such file or directory
>>> 2014-03-29 12:41:33 CET db: ip: us: LOG:  unexpected pageaddr
>>> 71/E9DA0000 in log file 114, segment 10, offset 14286848
>>> cp: cannot stat
>>> `/data/postgresql/9.1/main/wal_archive/00000001000000720000000A': No
>>> such file or directory
>>> 2014-03-29 12:41:33 CET db: ip: us: LOG:  streaming replication
>>> successfully connected to primary
>>> 2014-03-29 12:41:48 CET db: ip: us: LOG:  startup process (PID 17452)
>>> was terminated by signal 7: Bus error
>>> 2014-03-29 12:41:48 CET db: ip: us: LOG:  terminating any other active
>>> server processes
>>> 2014-03-29 12:41:48 CET db:wbloos ip:[local] us:wbloos WARNING:
>>> terminating connection because of crash of another server process
>>> 2014-03-29 12:41:48 CET db:wbloos ip:[local] us:wbloos DETAIL:  The
>>> postmaster has commanded this server process to roll back the current
>>> transaction and exit, because another server process exited abnormally
>>> and possibly corrupted shared memory.
>>> 2014-03-29 12:41:48 CET db:wbloos ip:[local] us:wbloos HINT:  In a
>>> moment you should be able to reconnect to the database and repeat your
>>> command.
>>>
>>>
>> Well what I am seeing are WAL log errors. One saying no file is
>> present, the other pointing at a possible file corruption.
>
> Those are normal notices, nothing to worry about.
>
>
>> Shared memory problems are offered as a possible cause only. Right now I
>> would say we are seeing only half the picture. The Postgres logs from the
>> same time period for the primary server, as well as the system logs for the
>> openvz container would help fill in the other half of the picture.
>>
>
> Here's the log from the primary postgres server:
> 2014-03-29 12:41:29 CET db:wbloos ip:[local] us:wbloos NOTICE:  ALTER TABLE
> will create implicit sequence "test_x_seq" for serial column "test.x"
> 2014-03-29 12:41:33 CET db:[unknown] ip:xxx.xxx.xxx.xxx us:replication
> LOG:  SSL renegotiation failure
> 2014-03-29 12:41:33 CET db:[unknown] ip:xxx.xxx.xxx.xxx us:replication
> LOG:  SSL error: unexpected record
> 2014-03-29 12:41:33 CET db:[unknown] ip:xxx.xxx.xxx.xxx us:replication
> LOG:  could not send data to client: Connection reset by peer
> 2014-03-29 12:41:48 CET db:[unknown] ip:xxx.xxx.xxx.xxx us:replication
> LOG:  could not receive data from client: Connection reset by peer
> 2014-03-29 12:41:48 CET db:[unknown] ip:xxx.xxx.xxx.xxx us:replication
> LOG:  unexpected EOF on standby connection
>
> (the SSL renegotiation failure happens all the time, without the crash)
>
> And here's the syslog form the container:
> Mar 29 12:41:01 mycontainer snmpd[8819]: Connection from UDP:
> [xxx.xxx.xxx.xxx]:59090->[xxx.xxx.xxx.xxx]
> Mar 29 12:42:30 mycontainer snmpd[8819]: Connection from UDP:
> [xxx.xxx.xxx.xxx]:35949->[xxx.xxx.xxx.xxx]
>
> The log on the host doesn't say anything interesting either.
>
> A cursory look at memory management in openvz shows it is different from
>> other virtualization software and physical machines. Whether that is a
>> problem would seem to be dependent on where you are on the learning curve:)
>>
> That sounds like "there is a solution to the problem, all you have to do is
> find out what it is". There doesn't seem to be a variable in the
> beancounters or anywhere else that can prevent the bus error from happening.
> There's seems to be no separate way of guaranteeing shared memory. There's
> no OOM killer active either, nor is host or server running short of memory.
>
> I'm still worried that it's like Tom Lane said in another discussion:"So
> basically, you've got a broken kernel here: it claimed to give PG circa
> (135MB) of memory, but what's actually there is only about (128MB). I don't
> see any connection between those numbers and the shmmax/shmall settings,
> either --- so I think this must be some busted implementation of a VM-level
> limitation."
> (here:
> http://www.postgresql.org/message-id/CAK3UJREBcyVBtr8D7vMfU=uDdkjXkrPnGcuy8erYB0tMfKe1LA@mail.gmail.com
> )
>
> And it makes me wonder what else may be issues that arise from that. But
> especially, what i can do about it.
>

So what does your "shmpages" and your sysctl inside the container
claim to be the max shared memory available. If you set the shmmax,
but the shmpages in your container config is to short Postgres maybe
has no way to find out beside crashing
(http://openvz.org/Postgresql_and_shared_memory).

Regards

Andreas

On 03/31/2014 04:12 AM, Willy-Bas Loos wrote:
>
> On Sat, Mar 29, 2014 at 6:17 PM, Adrian Klaver
> <adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>> wrote:
>
>     On 03/29/2014 08:19 AM, Willy-Bas Loos wrote:
>
>         The error that shows up is a Bus error.
>         That's on the replication slave.
>         Here's the log about it:
>         2014-03-29 12:41:33 CET db: ip: us: FATAL:  could not receive
>         data from
>         WAL stream: server closed the connection unexpectedly
>                   This probably means the server terminated abnormally
>                   before or while processing the request.
>
>         cp: cannot stat
>         `/data/postgresql/9.1/main/__wal_archive/__00000001000000720000000A':
>         No
>         such file or directory
>         2014-03-29 12:41:33 CET db: ip: us: LOG:  unexpected pageaddr
>         71/E9DA0000 in log file 114, segment 10, offset 14286848
>         cp: cannot stat
>         `/data/postgresql/9.1/main/__wal_archive/__00000001000000720000000A':
>         No
>         such file or directory
>         2014-03-29 12:41:33 CET db: ip: us: LOG:  streaming replication
>         successfully connected to primary
>         2014-03-29 12:41:48 CET db: ip: us: LOG:  startup process (PID
>         17452)
>         was terminated by signal 7: Bus error
>         2014-03-29 12:41:48 CET db: ip: us: LOG:  terminating any other
>         active
>         server processes
>         2014-03-29 12:41:48 CET db:wbloos ip:[local] us:wbloos WARNING:
>         terminating connection because of crash of another server process
>         2014-03-29 12:41:48 CET db:wbloos ip:[local] us:wbloos DETAIL:  The
>         postmaster has commanded this server process to roll back the
>         current
>         transaction and exit, because another server process exited
>         abnormally
>         and possibly corrupted shared memory.
>         2014-03-29 12:41:48 CET db:wbloos ip:[local] us:wbloos HINT:  In a
>         moment you should be able to reconnect to the database and
>         repeat your
>         command.
>
>
>     Well what I am seeing are WAL log errors. One saying no file is
>     present, the other pointing at a possible file corruption.
>
> Those are normal notices, nothing to worry about.

Well other then they cause the standby to reconnect to the primary,
during which a crash occurs.

>
>     Shared memory problems are offered as a possible cause only. Right
>     now I would say we are seeing only half the picture. The Postgres
>     logs from the same time period for the primary server, as well as
>     the system logs for the openvz container would help fill in the
>     other half of the picture.
>
>
> Here's the log from the primary postgres server:
> 2014-03-29 12:41:29 CET db:wbloos ip:[local] us:wbloos NOTICE:  ALTER
> TABLE will create implicit sequence "test_x_seq" for serial column "test.x"
> 2014-03-29 12:41:33 CET db:[unknown] ip:xxx.xxx.xxx.xxx us:replication
> LOG:  SSL renegotiation failure
> 2014-03-29 12:41:33 CET db:[unknown] ip:xxx.xxx.xxx.xxx us:replication
> LOG:  SSL error: unexpected record
> 2014-03-29 12:41:33 CET db:[unknown] ip:xxx.xxx.xxx.xxx us:replication
> LOG:  could not send data to client: Connection reset by peer
> 2014-03-29 12:41:48 CET db:[unknown] ip:xxx.xxx.xxx.xxx us:replication
> LOG:  could not receive data from client: Connection reset by peer
> 2014-03-29 12:41:48 CET db:[unknown] ip:xxx.xxx.xxx.xxx us:replication
> LOG:  unexpected EOF on standby connection
>
> (the SSL renegotiation failure happens all the time, without the crash)
>
> And here's the syslog form the container:
> Mar 29 12:41:01 mycontainer snmpd[8819]: Connection from UDP:
> [xxx.xxx.xxx.xxx]:59090->[xxx.xxx.xxx.xxx]
> Mar 29 12:42:30 mycontainer snmpd[8819]: Connection from UDP:
> [xxx.xxx.xxx.xxx]:35949->[xxx.xxx.xxx.xxx]
>
> The log on the host doesn't say anything interesting either.
>
>     A cursory look at memory management in openvz shows it is different
>     from other virtualization software and physical machines. Whether
>     that is a problem would seem to be dependent on where you are on the
>     learning curve:)
>
> That sounds like "there is a solution to the problem, all you have to do
> is find out what it is". There doesn't seem to be a variable in the
> beancounters or anywhere else that can prevent the bus error from happening.
> There's seems to be no separate way of guaranteeing shared memory.
> There's no OOM killer active either, nor is host or server running short
> of memory.

At this point I am not sure it is even obvious what is causing the
error, so finding a solution would be a hit or miss affair at best.

>
> I'm still worried that it's like Tom Lane said in another discussion:"So
> basically, you've got a broken kernel here: it claimed to give PG circa
> (135MB) of memory, but what's actually there is only about (128MB). I
> don't see any connection between those numbers and the shmmax/shmall
> settings, either --- so I think this must be some busted implementation
> of a VM-level limitation."
> (here:
> http://www.postgresql.org/message-id/CAK3UJREBcyVBtr8D7vMfU=uDdkjXkrPnGcuy8erYB0tMfKe1LA@mail.gmail.com)
>
> And it makes me wonder what else may be issues that arise from that. But
> especially, what i can do about it.

I do not use openvz so I do not have a test bed to try out, but this
page seems to be related to your problem:

http://openvz.org/Resource_shortage

or if you want more detail and a link to what looks to a replacement for
beancounters:

http://openvz.org/Setting_UBC_parameters

>
> Cheers,
>
> WBL
>
> --
> "Quality comes from focus and clarity of purpose" -- Mark Shuttleworth


--
Adrian Klaver
adrian.klaver@aklaver.com

On 03/31/2014 04:12 AM, Willy-Bas Loos wrote:
>
> On Sat, Mar 29, 2014 at 6:17 PM, Adrian Klaver
> <adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>> wrote:

>
> And it makes me wonder what else may be issues that arise from that. But
> especially, what i can do about it.

Had to go through it a couple of times, and look up the acronyms, but
the thread below seems to deal with the whole shared memory allocation
and setting process in openvz quite well:

http://forum.openvz.org/index.php?t=msg&goto=12061&&srch=postgresql+shared#msg_12061

In particular, where UBC is User Beancounters and the sysctls is the one
in the container.:

"UBC shmpages parameter controls all the shared memory which possible to
allocate
either via IPC or shmem (e.g. tmpfs).

sysctls above are only for IPC SYSv5 shared memory."

combine that with, where VE is Vitual Environment(container):

"check the same in VE please.
AFAICS, by default kernel sets:
#define SHMMAX 0x2000000
which is 32Mb.
So you have to increase it in VE /etc/sysctl.conf file or in /proc.
"
and there may be a path to a solution.

>
> Cheers,
>
> WBL
>
> --
> "Quality comes from focus and clarity of purpose" -- Mark Shuttleworth


--
Adrian Klaver
adrian.klaver@aklaver.com

Adrian Klaver <adrian.klaver@aklaver.com> writes:
> On 03/31/2014 04:12 AM, Willy-Bas Loos wrote:
>> I'm still worried that it's like Tom Lane said in another discussion:"So
>> basically, you've got a broken kernel here: it claimed to give PG circa
>> (135MB) of memory, but what's actually there is only about (128MB). I
>> don't see any connection between those numbers and the shmmax/shmall
>> settings, either --- so I think this must be some busted implementation
>> of a VM-level limitation."
>> (here:
>> http://www.postgresql.org/message-id/CAK3UJREBcyVBtr8D7vMfU=uDdkjXkrPnGcuy8erYB0tMfKe1LA@mail.gmail.com)
>>
>> And it makes me wonder what else may be issues that arise from that. But
>> especially, what i can do about it.

FWIW, I went back and re-read that message while perusing this thread,
and this time it struck me that there was a significant bit of evidence
I'd overlooked: namely, that the buffer block array is by no means the
last thing in Postgres' shared memory segment.  There are a bunch of
other shared data structures allocated after it, some of which almost
certainly had to have been touched by the startup subprocess.  The gdb
output makes it clear that the kernel stopped providing memory at
0xb6c4b000; but either it resumed doing so further on, or the whole shared
memory segment *had* been provisioned originally, and then part of it
got unmapped again while the startup process was running.

So it's still clearly a kernel bug, but it seems less likely that it is
triggered by some static limit on shared memory size.  Perhaps instead,
the kernel had been filling in pages for the shared segment on-demand,
and then when it got to some limit it refused to do so anymore and allowed
a SIGBUS to happen instead.

> I do not use openvz so I do not have a test bed to try out, but this
> page seems to be related to your problem:
> http://openvz.org/Resource_shortage
> or if you want more detail and a link to what looks to a replacement for
> beancounters:
> http://openvz.org/Setting_UBC_parameters

If this software's idea of resource management is to allow SIGBUS to
happen upon attempting to use memory that had been successfully granted,
then it's a piece of junk that you should get rid of ASAP.  (No, I
don't like Linux's OOM-kill solution to resource overcommit either.)

            regards, tom lane

On 03/31/2014 08:01 AM, Tom Lane wrote:
> Adrian Klaver <adrian.klaver@aklaver.com> writes:
>> On 03/31/2014 04:12 AM, Willy-Bas Loos wrote:
>>> I'm still worried that it's like Tom Lane said in another discussion:"So
>>> basically, you've got a broken kernel here: it claimed to give PG circa
>>> (135MB) of memory, but what's actually there is only about (128MB). I
>>> don't see any connection between those numbers and the shmmax/shmall
>>> settings, either --- so I think this must be some busted implementation
>>> of a VM-level limitation."
>>> (here:
>>> http://www.postgresql.org/message-id/CAK3UJREBcyVBtr8D7vMfU=uDdkjXkrPnGcuy8erYB0tMfKe1LA@mail.gmail.com)
>>>
>>> And it makes me wonder what else may be issues that arise from that. But
>>> especially, what i can do about it.
>
> FWIW, I went back and re-read that message while perusing this thread,
> and this time it struck me that there was a significant bit of evidence
> I'd overlooked: namely, that the buffer block array is by no means the
> last thing in Postgres' shared memory segment.  There are a bunch of
> other shared data structures allocated after it, some of which almost
> certainly had to have been touched by the startup subprocess.  The gdb
> output makes it clear that the kernel stopped providing memory at
> 0xb6c4b000; but either it resumed doing so further on, or the whole shared
> memory segment *had* been provisioned originally, and then part of it
> got unmapped again while the startup process was running.
>
> So it's still clearly a kernel bug, but it seems less likely that it is
> triggered by some static limit on shared memory size.  Perhaps instead,
> the kernel had been filling in pages for the shared segment on-demand,
> and then when it got to some limit it refused to do so anymore and allowed
> a SIGBUS to happen instead.
>
>> I do not use openvz so I do not have a test bed to try out, but this
>> page seems to be related to your problem:
>> http://openvz.org/Resource_shortage
>> or if you want more detail and a link to what looks to a replacement for
>> beancounters:
>> http://openvz.org/Setting_UBC_parameters
>
> If this software's idea of resource management is to allow SIGBUS to
> happen upon attempting to use memory that had been successfully granted,
> then it's a piece of junk that you should get rid of ASAP.  (No, I
> don't like Linux's OOM-kill solution to resource overcommit either.)

At this point the memory allocation as a problem is as much conjecture
as anything else, at least to me. So what is causing SIGBUS is an open
question in my mind.


>
>             regards, tom lane
>
>


--
Adrian Klaver
adrian.klaver@aklaver.com

Adrian Klaver <adrian.klaver@aklaver.com> writes:
> At this point the memory allocation as a problem is as much conjecture
> as anything else, at least to me. So what is causing SIGBUS is an open
> question in my mind.

Agreed, it's unproven what's causing the SIGBUS in the case at hand.
However, in the case I investigated for Josh Kupershmidt, it was provably
the fault of the kernel for failing to supply memory to back the shared
memory segment that it had previously agreed to allocate.  The fact that
both users are trying to run their standby server under OpenVZ is, shall
we say, merely suggestive.

            regards, tom lane

On 03/31/2014 08:28 AM, Tom Lane wrote:
> Adrian Klaver <adrian.klaver@aklaver.com> writes:
>> At this point the memory allocation as a problem is as much conjecture
>> as anything else, at least to me. So what is causing SIGBUS is an open
>> question in my mind.
>
> Agreed, it's unproven what's causing the SIGBUS in the case at hand.
> However, in the case I investigated for Josh Kupershmidt, it was provably
> the fault of the kernel for failing to supply memory to back the shared
> memory segment that it had previously agreed to allocate.  The fact that
> both users are trying to run their standby server under OpenVZ is, shall
> we say, merely suggestive.

Yes, the fact that the problem seems to go away with a change in
shared_buffers seems to point in that direction also. To get through the
smoke to the fire, there are more questions to be answered:

1) The OP says the shared_buffered setting that failed was 4GB and that
'tuning' down the setting solved the problem. So it would be nice to
know what setting worked?

2) In hand with 1), the memory settings for the virtualization host and
the container(s) involved are?

3) What is the host OS?

4) What version of OpenVZ?

5) What is the layout?
Are the primary and standby on the same container, different containers
or some other arrangement?
In other words could there be some other resource conflict going on, say
network?

6) Are the crashes random or do they follow some pattern?

>
>             regards, tom lane
>


--
Adrian Klaver
adrian.klaver@aklaver.com