Thread: New feature proposal

New feature proposal

From
Marc Munro
Date:
Veil http://pgfoundry.org/projects/veil is currently not a very good
Postgres citizen.  It steals what little shared memory it needs from
postgres' shared memory using ShmemAlloc().

For Postgres 8.2 I would like Veil to be a better citizen and use only
what shared memory has been reserved for postgres add-ins.

So, I would like to propose:
1) that a new shared memory area for add-ins be created
2) that it should be accessible only to add-ins specifically requesting
its use
3) that the size of this add-in memory be controlled through a new GUC
setting

If this proposal is acceptable, I'd like some suggestions for naming the
GUC.  If not we will be stuck with add_in_shmem, specified in Kbytes.

My expectation is that this would be a very small change and one that I
am probably competent to make myself, with a little review from my
friends.

__
Marc



Re: New feature proposal

From
Simon Riggs
Date:
On Thu, 2006-05-18 at 17:39 -0700, Marc Munro wrote:

> For Postgres 8.2 I would like Veil to be a better citizen and use only
> what shared memory has been reserved for postgres add-ins.

How would Postgres ask the add-in how much memory it needs? How would
the add-in know how much has been reserved for it? How would an add-in
know whether it was the only add-in and whether it could take all of the
allocation?

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com



Re: New feature proposal

From
Andreas Pflug
Date:
Marc Munro wrote:
> Veil http://pgfoundry.org/projects/veil is currently not a very good
> Postgres citizen.  It steals what little shared memory it needs from
> postgres' shared memory using ShmemAlloc().
> 
> For Postgres 8.2 I would like Veil to be a better citizen and use only
> what shared memory has been reserved for postgres add-ins.

Why should this be individually restricted? AFAICS Veil's functionality 
would be essential to access row level ACL controlled tables, so if it 
fails for low mem conditions it's much like a backend failure.

Regards,
Andreas


Re: New feature proposal

From
Marc Munro
Date:
On Fri, 2006-05-19 at 13:41 -0300, pgsql-hackers-owner@postgresql.org
wrote:
> On Thu, 2006-05-18 at 17:39 -0700, Marc Munro wrote:
>
> > For Postgres 8.2 I would like Veil to be a better citizen and use
> only
> > what shared memory has been reserved for postgres add-ins.
>
> How would Postgres ask the add-in how much memory it needs? How would
> the add-in know how much has been reserved for it? How would an add-in
> know whether it was the only add-in and whether it could take all of
> the
> allocation?

Postgres would not ask any add-ins how much they need, it would simply
allocate the extra amount defined in a GUC and not make that available
through the normal shared memory allocation mechanism.

The add-in would not "know" how much had been allocated to it, but could
be told through it's own config file.  I envisage something like:

in postgresql.conf

# add_in_shmem = 0    # Amount of shared mem to set aside for add-ins                     # in KBytes
add_in_shem = 64


in veil.conf

veil_shmem = 32          # Amount of shared memory we can use from          # the postgres add-ins shared memory pool

I think this is better than add-ins simply stealing from, and contending
for, postgres shared memory which is the only real alternative right
now.

__
Marc

Re: New feature proposal

From
Josh Berkus
Date:
Marc,

> The add-in would not "know" how much had been allocated to it, but could
> be told through it's own config file.  I envisage something like:
>
> in postgresql.conf
>
> # add_in_shmem = 0    # Amount of shared mem to set aside for add-ins
>                       # in KBytes
> add_in_shem = 64
>
>
> in veil.conf
>
> veil_shmem = 32       # Amount of shared memory we can use from
>                       # the postgres add-ins shared memory pool
>
> I think this is better than add-ins simply stealing from, and contending
> for, postgres shared memory which is the only real alternative right
> now.

Hmmmm ... what would happen if I did:

add_in_shmem = 64
veil_shmem = 128

or even:

add_in_shmem = 128
veil_shmem = 64
plperl_shmem = 64
pljava_shmem = 64

... seems like we'll need to check for overallocation, no?

--
Josh Berkus
Aglio Database Solutions
San Francisco


Re: New feature proposal

From
Marc Munro
Date:
On Fri, 2006-05-19 at 10:05 -0700, Josh Berkus wrote:
> Marc,
>
> > The add-in would not "know" how much had been allocated to it, but could
> > be told through it's own config file.  I envisage something like:
> >
> > in postgresql.conf
> >
> > # add_in_shmem = 0    # Amount of shared mem to set aside for add-ins
> >                       # in KBytes
> > add_in_shem = 64
> >
> >
> > in veil.conf
> >
> > veil_shmem = 32       # Amount of shared memory we can use from
> >                       # the postgres add-ins shared memory pool
> >
> > I think this is better than add-ins simply stealing from, and contending
> > for, postgres shared memory which is the only real alternative right
> > now.
>
> Hmmmm ... what would happen if I did:
>
> add_in_shmem = 64
> veil_shmem = 128
>
> or even:
>
> add_in_shmem = 128
> veil_shmem = 64
> plperl_shmem = 64
> pljava_shmem = 64
>

If that happens, one of the add-ins will be sadly disappointed when it
tries to use its allocation.  The same as would happen now, if Veil
attempted to allocate too large a chunk of shared memory.

My proposal makes it possible for properly configured add-ins to have a
guaranteed amount of shared memory available.  It allows add-ins to be
well-behaved in their use of shared memory, and it prevents them from
being able to exhaust postgres' own shared memory.

It doesn't prevent add-ins from over-allocating from the add-in memory
context, nor do I think it can or should do this.

__
Marc

Re: New feature proposal

From
Tom Lane
Date:
Marc Munro <marc@bloodnok.com> writes:
> My proposal makes it possible for properly configured add-ins to have a
> guaranteed amount of shared memory available.

This could all be solved in a cleaner, more bulletproof way if you
simply require such add-ins to be preloaded into the postmaster process
using the existing preload_libraries hook.  Then, such an add-in would
allocate its own shmem segment independent of the main Postgres one.
This totally eliminates worries about one chunk of code eating the other
one's memory, which otherwise we'd have to have additional mechanism to
deal with.

In a Unix environment, such a thing would Just Work because pointers to
the new segment would be inherited through fork().  In the Windows port
you'd need to do more pushups --- perhaps allocate a small amount of
memory in the main Postgres shmem segment containing the ID of the other
shmem segment, which a backend would use to reattach.
        regards, tom lane


Re: New feature proposal

From
Marc Munro
Date:
On Fri, 2006-05-19 at 14:44 -0400, Tom Lane wrote:
> Marc Munro <marc@bloodnok.com> writes:
> > My proposal makes it possible for properly configured add-ins to have a
> > guaranteed amount of shared memory available.
>
> This could all be solved in a cleaner, more bulletproof way if you
> simply require such add-ins to be preloaded into the postmaster process
> using the existing preload_libraries hook.  Then, such an add-in would
> allocate its own shmem segment independent of the main Postgres one.
> This totally eliminates worries about one chunk of code eating the other
> one's memory, which otherwise we'd have to have additional mechanism to
> deal with.

This is an interesting idea that I had not previously considered.  I
will give it some thought.

I'm not convinced that we actually do need to prevent add-ins from
eating each other's memory.  Just as existing add-ins that use palloc
are expected to use the appropriate memory context and behave
themselves, I would expect the same to be true for add-ins that require
shared memory.

> In a Unix environment, such a thing would Just Work because pointers to
> the new segment would be inherited through fork().  In the Windows port
> you'd need to do more pushups --- perhaps allocate a small amount of
> memory in the main Postgres shmem segment containing the ID of the other
> shmem segment, which a backend would use to reattach.
>

For me, adding windows-specific code to Veil is highly unappealling - I
have no easy way to build or test for windows, and no experience of
doing so, so the more I can leverage the existing functionality, the
better.

I had hoped to simply piggyback on Postgres' existing memory management
with a very small change to effectively add an add-in shared memory
context.

On the other hand, if this is the way we have to go, then perhaps it
could be added to Postgres as part of its api, rather than having Veil,
and perhaps other add-ins, implement it for themselves.

Thoughts?

__
Marc

Re: New feature proposal

From
Marc Munro
Date:
On Fri, 2006-05-19 at 12:35 -0700, Marc Munro wrote:
> On Fri, 2006-05-19 at 14:44 -0400, Tom Lane wrote:
> > This could all be solved in a cleaner, more bulletproof way if you
> > simply require such add-ins to be preloaded into the postmaster process
> > using the existing preload_libraries hook.  Then, such an add-in would
> > allocate its own shmem segment independent of the main Postgres one.
> > This totally eliminates worries about one chunk of code eating the other
> > one's memory, which otherwise we'd have to have additional mechanism to
> > deal with.
>
> This is an interesting idea that I had not previously considered.  I
> will give it some thought.

I have give this idea some further thought and I agree; Tom's solution
is more bulletproof and is the right way to go.  My original proposal is
withdrawn.

I am going to look into the best way to implement this but my gut
feeling is that I would like the support infrastructure for this to be
in Postgres rather than in Veil.

By support infrastructure, I mean APIs to create and access new shared
memory segments, and allocate chunks of memory from those shared
segments.

I think this code is better placed in Postgres rather than in specific
add-ins because: it is functionality that could benefit many add-ins; it
can make use of existing postgres code; and it can be easily tested in
the regression suite using the buildfarm.

I don't want to start working on this without knowing there is a chance
of the patch being acceptable, so feedback is invited.

Thanks.
__
Marc