Thread: Re: [COMMITTERS] pgsql: Automatic view update rules Bernd Helmle

--On 23. Januar 2009 21:18:19 -0500 Robert Haas <robertmhaas@gmail.com> 
wrote:

> In the future, I think we should have an expectation that resubmits
> within the same CommitFest should happen within a week, and that if no
> revision is forthcoming within two weeks the patch is declared dead
> (and the submitter can add it to the next CommitFest when they
> resubmit).  Don't think I'm picking on you, either: there was quite a
> bit of it this CommitFest, and it's bad, because:


yes, i'd be happier if i could have spent more time on it, being more 
responsive, especially in late December. But it wasn't possible, my fault :(

You're suggestion doesn't help with the problem that (like Joshua already 
mentioned) core developers are too busy with reviewing stuff during the 
CommitFest. Because of this it's really hard to get the necessary time of 
somebody who is able to evaluate the architecture of a new feature and 
(more important) its side-effects on the whole system. Especially the last 
CommitFest before Feature Freeze is becoming heavily busted with many many 
interesting patches, because people want to have their WIP reviewed at 
least for the upcoming release (like me). I don't see how postponing 
patches would make it better?

The lesson i've learned is to post more, post often and not waiting until 
the last CommitFest begins.

Bernd

> You're suggestion doesn't help with the problem that (like Joshua already
> mentioned) core developers are too busy with reviewing stuff during the
> CommitFest. Because of this it's really hard to get the necessary time of
> somebody who is able to evaluate the architecture of a new feature and (more
> important) its side-effects on the whole system. Especially the last
> CommitFest before Feature Freeze is becoming heavily busted with many many
> interesting patches, because people want to have their WIP reviewed at least
> for the upcoming release (like me). I don't see how postponing patches would
> make it better?

Hmm, well, I'm only talking about postponing patches that aren't ready
to be committed, and then only if their authors don't respond to
feedback in a timely fashion.  I agree that it would be nice if the
core developers had more time to provide more feedback to more people,
but that seems like an unrelated problem, and in any event I don't
know how to solve it, other than by hoping that we'll eventually have
more core developers.

On the other hand, it's easy to draw a line from the lax criteria for
resubmitting patches to the length of this CommitFest.  It now appears
that this CommitFest will be something like 3.5 months long and that
the next one will not occur before May.  That means we're essentially
closed to new patches for six months, which is a really long time.  To
put it another way, for every week the core team spends reworking the
existing patches, it will be another week before someone can get
feedback on any new patches submitted now.  At some point that becomes
a bad trade-off for the project as a whole.  Judging when exactly
you've hit that point is difficult, but I'm starting to think we may
have entered the zone of diminishing returns.

...Robert

Robert Haas wrote:
> On the other hand, it's easy to draw a line from the lax criteria for
> resubmitting patches to the length of this CommitFest.  It now appears
> that this CommitFest will be something like 3.5 months long and that
> the next one will not occur before May.  That means we're essentially
> closed to new patches for six months, which is a really long time.  To
> put it another way, for every week the core team spends reworking the
> existing patches, it will be another week before someone can get
> feedback on any new patches submitted now.  At some point that becomes
> a bad trade-off for the project as a whole.  Judging when exactly
> you've hit that point is difficult, but I'm starting to think we may
> have entered the zone of diminishing returns.

Well, also consider 8.3 was released in February, so we had 9 months of
development before the last commit-fest started.

Also, one thing we have always known is that many of the complex patches
bottle up on the last commit-fest because we kind of realize they are
not going to get much easier to deal with.

As long as we are mostly busy reviewing and committing stuff, and not
bottled up on 1-2 patches, I think the time in commit fest is being well
spent.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On Sun, Jan 25, 2009 at 12:28 AM, Bruce Momjian <bruce@momjian.us> wrote:
> Well, also consider 8.3 was released in February, so we had 9 months of
> development before the last commit-fest started.

Yes - that was good.  But what will happen for 8.5?

> Also, one thing we have always known is that many of the complex patches
> bottle up on the last commit-fest because we kind of realize they are
> not going to get much easier to deal with.

Yeah...  I'm not sure what to do about that, but as Tom pointed out,
it has the disadvantage that all of these massive changes are getting
put into the tree just before we start beta.

...Robert

Robert Haas wrote:
> On Sun, Jan 25, 2009 at 12:28 AM, Bruce Momjian <bruce@momjian.us> wrote:
> > Well, also consider 8.3 was released in February, so we had 9 months of
> > development before the last commit-fest started.
> 
> Yes - that was good.  But what will happen for 8.5?

Probably the same.

> > Also, one thing we have always known is that many of the complex patches
> > bottle up on the last commit-fest because we kind of realize they are
> > not going to get much easier to deal with.
> 
> Yeah...  I'm not sure what to do about that, but as Tom pointed out,
> it has the disadvantage that all of these massive changes are getting
> put into the tree just before we start beta.

Well, it is less a problem than in previous releases, so things are
getting better.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

>> Yeah...  I'm not sure what to do about that, but as Tom pointed out,
>> it has the disadvantage that all of these massive changes are getting
>> put into the tree just before we start beta.
>
> Well, it is less a problem than in previous releases, so things are
> getting better.

Well, that is good.

I wonder if it would be practical to have a "FeedbackFest" sometime
between now and the release of 8.4 - not to actually commit anything,
but to do preliminary reviews of patches for 8.5 while there's still
plenty of time for them to be revised and submitted earlier in the
cycle.  I would be willing to put in quite a bit of time to reviewing
other people's patches if it meant that someone knowledgeable would
take a quick look at mine.  (Unfortunately my current area of interest
is the optimizer, so the pool of sufficiently knowledgeable people is
pretty small.)

In any case, thanks for your thoughts on this subject - I appreciate
your taking the time to write back.

...Robert

Bruce Momjian <bruce@momjian.us> writes:
> Robert Haas wrote:
>> Yeah...  I'm not sure what to do about that, but as Tom pointed out,
>> it has the disadvantage that all of these massive changes are getting
>> put into the tree just before we start beta.

> Well, it is less a problem than in previous releases, so things are
> getting better.

Are they?  If we stay in commitfest mode until every single thing on the
list has been dealt with, I am not sure this release cycle will be any
shorter than 8.3's was.

We said back at the beginning that

>> The hope here is that we will not have enormous, previously unreviewed
>> patches landing on us at the end of October --- if that happens, we'll
>> be back in the same position we were in at 8.3 feature freeze. Although
>> this schedule allows for the final commit fest to take a good deal of
>> time, we'll reserve the right to reject patches that are too large to be
>> reviewed in a timely fashion. We want to encourage people to do
>> development of large features in an incremental fashion, with a new
>> increment landing during each commit fest.

and I'm beginning to think that we need to invoke that provision.
Particularly with regard to hot standby, which by any sane reading was
not close to being committable on 1 November (a fortiori from the fact
that it's *still* not committable despite large amounts of later work).
I'm also feeling that we are not in a position to commit SE-Postgres in
a timely fashion; which is not KaiGai-san's fault, rather that of the
community which has taken nearly zero interest in that patch.

If we want to ensure that 8.5 development opens soon, what we have to do
is reject those two patches, revert updatable views, and finish up the
other stuff (which is all small and could likely be dealt with in a week
or two).  That puts us in position to go beta by perhaps mid-February
with release perhaps on May 1.  If we don't, I hereby predict that 8.4
release will not happen before September.  Trying to deal with those
late, large features will add *at least* one more month to commitfest
and *at least* one more month to beta (you think they'll be bug-free?).

As our new president has been reminding us, it's time to start making
hard choices.
        regards, tom lane

> and I'm beginning to think that we need to invoke that provision.
> Particularly with regard to hot standby, which by any sane reading was
> not close to being committable on 1 November (a fortiori from the fact
> that it's *still* not committable despite large amounts of later work).
> I'm also feeling that we are not in a position to commit SE-Postgres in
> a timely fashion; which is not KaiGai-san's fault, rather that of the
> community which has taken nearly zero interest in that patch.
>
> If we want to ensure that 8.5 development opens soon, what we have to do
> is reject those two patches, revert updatable views, and finish up the
> other stuff (which is all small and could likely be dealt with in a week
> or two).  That puts us in position to go beta by perhaps mid-February
> with release perhaps on May 1.  If we don't, I hereby predict that 8.4
> release will not happen before September.  Trying to deal with those
> late, large features will add *at least* one more month to commitfest
> and *at least* one more month to beta (you think they'll be bug-free?).

As much as I hate to say it, I agree with all of this.  I think it
would be a good to see whether some subset of the Hot Standby can be
committed - perhaps the "infrastructure changes for recovery" portion.With respect to the remaining patches, I think
thatanything that is
 
basically committable now should be committed and everything else
should be considered returned with feedback (or reverted, in the case
of updatable views).  Nearly every patch that is still on the
CommitFest wiki has undergone significant changes since the CommitFest
started, and I don't think it's the purpose of CommitFest to give
people another 3 months to work the bugs out of a patch that basically
isn't finished, or to have the committers finish them in lieu of the
original submitters.  And if it's really true that we will be in
feature freeze for 10 months (11/1/08-9/1/09) then I don't think
that's remotely a good idea.

I would, however, like to see us make a commitment to actually review
SE-PostgreSQL.  There was some talk that we might not want to include
this feature in core at all, and if that is the case then I think it
is long past time to make that decision.  Assuming that isn't the
case, then we need to get past the stage where we make occasional
comments on the overall architecture and get down to really reading
the code.  I am willing to help with this but I don't have either the
time or the qualifications to do it single-handedly.  To be brutally
honest, I don't care about the feature at all: the only thing I ever
do with SELinux is turn it off (row-level DAC is mildly interesting to
me).  But I think that if we want to build a community of developers
around PostgreSQL, we'd better at least look at the work they submit.

...Robert

On Sun, Jan 25, 2009 at 12:06 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> and I'm beginning to think that we need to invoke that provision.
> Particularly with regard to hot standby, which by any sane reading was
> not close to being committable on 1 November (a fortiori from the fact
> that it's *still* not committable despite large amounts of later work).

maybe what we need is to put a policy for large patches... something
like: if you submit a large patch that introduce fairly complex new
features very late in the release cycle (maybe at the last commit fest
or previous to that one) then there are no promises about committers
to review them... maybe that will enforce authors to send patches more
often and more early...

and of course, the rest of us that make some kind of review (call it:
testing or code review) should start making those reviews on large
patches (just like someone suggests)

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

Tom Lane wrote:
> and I'm beginning to think that we need to invoke that provision.
> Particularly with regard to hot standby, which by any sane reading was
> not close to being committable on 1 November (a fortiori from the fact
> that it's *still* not committable despite large amounts of later work).
> I'm also feeling that we are not in a position to commit SE-Postgres in
> a timely fashion; which is not KaiGai-san's fault, rather that of the
> community which has taken nearly zero interest in that patch.

Yes, few feedbacks have been a concern for me, though I can understand
that reviewers/commiters had to deal with various kind of many patches
for the v8.4 development cycle. (Needless to say, I remember Tom commented
a lot on CommitFest:May and it made SE-PostgreSQL well.)
If SE-PostgreSQL is not still committable phase, I want to discuss what
code should be reworked and what items are remained.
Because it was unclear, I had to review patches by myself in this January. :(

In addition, I would like to mention about the scale of patches.|  % diffstat
sepostgresql-sepgsql-8.4devel-3-r1467.patch
says,| 110 files changed, 9813 insertions(+), 16 deletions(-), 924 modifications(!)
However, the total number of insertions under "src/backend/security" and
"src/include/security" is 8207 lines of 9813. It modifies 924 lines, but
504 lines of 924 is due to a new field of pg_attribute system catalog.
Please note that it never applies massive rough-mannered fixes all over
the core implementation, so it is informidable.

> If we want to ensure that 8.5 development opens soon, what we have to do
> is reject those two patches, revert updatable views, and finish up the
> other stuff (which is all small and could likely be dealt with in a week
> or two).  That puts us in position to go beta by perhaps mid-February
> with release perhaps on May 1.  If we don't, I hereby predict that 8.4
> release will not happen before September.  Trying to deal with those
> late, large features will add *at least* one more month to commitfest
> and *at least* one more month to beta (you think they'll be bug-free?).

I can understand we cannot postpone to release v8.4 forever and it is
important to see a calendar. However, I would like folks to see not
only a calendar, but a current *trend* also.

Nowaday, we have no days without looking a term such as SaaS/PaaS or
cloud-computing. It consists of various kind of web applications and
highly-integrated database system on cluster systems.
In this movement, end-users pay their attention on "availability" and
"security" most. If we can include these features in a timely fashion,
it makes PostgreSQL more attractive compared to other DBMSs.
Needless to say, I'll pay my maximum effort to reduce the additional
days, even if it is estimated one more month is necessary.

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Robert Haas wrote:
> I would, however, like to see us make a commitment to actually review
> SE-PostgreSQL.  There was some talk that we might not want to include
> this feature in core at all, and if that is the case then I think it
> is long past time to make that decision.  Assuming that isn't the
> case, then we need to get past the stage where we make occasional
> comments on the overall architecture and get down to really reading
> the code.  I am willing to help with this but I don't have either the
> time or the qualifications to do it single-handedly.  To be brutally
> honest, I don't care about the feature at all: the only thing I ever
> do with SELinux is turn it off (row-level DAC is mildly interesting to
> me).  But I think that if we want to build a community of developers
> around PostgreSQL, we'd better at least look at the work they submit.

I would like to call for SELinux folks to join reviewing SE-PostgreSQL
from the point of view of security expert. If folks in pgsql-hackers
have questions, I belive they can provide an answer to the questions.

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote:

> If we want to ensure that 8.5 development opens soon, what we have to
> do is reject those two patches, revert updatable views, and finish up
> the other stuff (which is all small and could likely be dealt with in
> a week or two).  That puts us in position to go beta by perhaps
> mid-February with release perhaps on May 1.  If we don't, I hereby
> predict that 8.4 release will not happen before September.  Trying to
> deal with those late, large features will add *at least* one more
> month to commitfest and *at least* one more month to beta (you think
> they'll be bug-free?).

> As our new president has been reminding us, it's time to start making
> hard choices.

We're clearly ahead of where we were in 8.3. Any release with big
features in it will take longer, whether you wait a year, or not.

I think the choice here is currently between features or release
schedule, but we could look at things differently and generate some
other options.

What would our users want?

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote:
> If we want to ensure that 8.5 development opens soon, what we have to
> do is reject those two patches, revert updatable views, and finish up
> the other stuff (which is all small and could likely be dealt with in
> a week or two).  That puts us in position to go beta by perhaps
> mid-February with release perhaps on May 1.  If we don't, I hereby
> predict that 8.4 release will not happen before September.

Expecting a release around September with those patches a bit
optimistic, IMHO. We already experienced that a release just after
summer is not possible. So, unless there is a way to commit SELinux+Hot
Standby+Sync Replication in the next two weeks (or so), as you wrote, we
should reject those patches and release 8.4 on time.

(Oh, personally I'd like to see all these features at 8.4, from advocacy
side).
--
Devrim GÜNDÜZ, RHCE
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr                  http://www.gunduz.org

Simon Riggs wrote:
> On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote:
> 
>> Any release with big
> features in it will take longer, whether you wait a year, or not.

Well, big features that land early in the release cycle don't delay the 
release. Just the ones that are submitted in the last commit fest.

> I think the choice here is currently between features or release
> schedule, but we could look at things differently and generate some
> other options.

Yeah, that's the tradeoff. features vs. release schedule. I don't see 
how else you could look at things.

> What would our users want?

I'm sure it depends on the user. Users that are more interested in the 
features we already have in the bag like window functions and 
WITH-clause, will obviously prefer to release earlier without hot 
standby. And users that want hot standby (or SE-postgresql) will prefer 
to delay the release and have those features included.

I don't personally have an opinion, because I don't work with any 
real-world databases. It's a tradeoff and both options seem good to me. 
I'm going to stick around and keep reviewing Hot Standby as long as it 
takes, and when it's ready I can commit it to either 8.4 or 8.5. It's 
the same amount of work for me either way.

There's still a list of non-resolved issues that I estimate will take 
about two weeks to address. And there's a good possibility that new 
issues arise, which require yet more time.

--   Heikki Linnakangas  EnterpriseDB   http://www.enterprisedb.com

On Mon, Jan 26, 2009 at 11:35 AM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:

> I'm sure it depends on the user. Users that are more interested in the
> features we already have in the bag like window functions and WITH-clause,
> will obviously prefer to release earlier without hot standby. And users that
> want hot standby (or SE-postgresql) will prefer to delay the release and
> have those features included.

At LinuxLive (UK) the overwhelming majority of people I spoke to over
three days wanted hot standby and replication (preferably
multi-master, but thats another story). Window functions, recursive
queries, SE PostgreSQL, updatable views and other new features were
barely mentioned.

> There's still a list of non-resolved issues that I estimate will take about
> two weeks to address. And there's a good possibility that new issues arise,
> which require yet more time.

That doesn't seem like it's worth deferring such a useful feature for
12+ months for - especially as the work is clearly ongoing and can
happen in parallel with work on other patches.

As I've pointed out before, we're not a commercial company working for
our shareholders, we're a FOSS project working for our end users. If
we can include an important and popular feature like this at the
expense of a few weeks extra wait for the release, it seems to me that
we'll be serving our users far better overall than making a fair
percentage of them wait another 12 months for work that is more or
less complete.

-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote:

> Particularly with regard to hot standby, which by any sane reading was
> not close to being committable on 1 November (a fortiori from the fact
> that it's *still* not committable despite large amounts of later
> work).

I've wasted much time in two main areas in recent months: 

The first area I've wasted time on is patch review. Had I known that
some patches on the queue would not be properly reviewed and that time
was going to be a factor then I would not have wasted a single second on
patch review. My main review item has been Sync Replication, which I did
because of Core's statement that we would include that feature in this
release. It would appear that my time on that has been both pointless
and damaging to my own situation.

The second is in developing an answer to the FATAL errors problem, which
I asked a direct question on and received a very specific answer:
http://archives.postgresql.org/pgsql-hackers/2008-09/msg01809.php
I spent two weeks working out a difficult solution to that, a week
subsequently arguing with Heikki (possibly longer) about whether it was
necessary and then two weeks refactoring it back out again when you were
silent.

>From my side, I've spent time and money supporting your views and
decisions, without argument with you, though most would not see that. I
learned on Jan 12 that you'd been too busy to consider this work and
that a priori you argued for patch rejection.

It would be much simpler if I had badly screwed up or if we'd found an
awkward showstopper. That hasn't happened. In fact, I've done the full
dance so far.

I fail to see how rejecting unreviewed patches provides benefit for
users, developers or sponsors. How does de-committing from features
voted most-popular or promised by core sit with users? How will
developers react when they know that helping in patch review assists
their own demise? How will sponsors react when they see another set of
patches rejected?

If there is a problem with timing it's because we need more planning,
scheduling and prioritisation of resources. But it isn't possible to do
it retrospectively without causing problems, and those problems may be
worse than slipping a month, or three.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

> Simon has put a lot of time into Hot Standby and has followed the
> pseudo-defacto community process from design through what he believes to be
> near-completion; he can't be sure of completion until someone reviews his
> work.

I think this is a fair critique.

> Yet, albeit with almost no review from the committers, Simon has continually
> worked through testing, revising his patches, and requesting information and
> suggestions from the community.

There really was not a lot of review from mid-October through the end
of the year.  That is partly because Simon was out of commission for
about three weeks and did not respond (in particular) to several
requests to separate ICfR from HS.  That having been said, since this
is such an important feature to so many people, I think it would have
been better if more effort could have been put into doing what
additional reviewing was still possible.  However, since the turn of
the year, it looks to me like Heikki has actually put quite a bit of
time into reviewing and responding to issues.

Still, I agree that if there's anything we should be putting our
effort into as a community right now, it's this feature.  If we got
Hot Standby in the next release and everything else in the CommitFest
got bumped, I think a lot of people would consider that a good trade
(though probably not the authors of the patches that got bumped).  For
example, I would much rather see Tom revert the updatable views patch
and work on this than spend the next week fixing updatable views.

At a minimum, I think the following patches from the CommitFest wiki
should be returned with feedback or rejected:

1. SE-PostgreSQL.  We handled this one badly, but there's not enough
time to fix it now.  8.5.
2. rmgr hooks and contrib/rmgr_hook.  Reject because Tom and Heikki
don't believe it's the right approach.  Need better use cases.
3. Synchronous log-shipping replication.  We handled this one well,
but it's not in good enough shape.  8.5.
4. pg_upgrade script.  I haven't heard much about this in a while...
I am doubtful that it is production-quality, but maybe I'm wrong?
5. Reducing some DDL Locks to ShareLock.  No activity in a long time,
no time to wait for this to be finished.  8.5.

...Robert

On Mon, 2009-01-26 at 13:35 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote:
> > 
> >> Any release with big
> > features in it will take longer, whether you wait a year, or not.
> 
> Well, big features that land early in the release cycle don't delay the 
> release. Just the ones that are submitted in the last commit fest.

Has that ever happened? :-)

I don't think its chance we get big patches in last commit fest. No
sponsor I ever met was happy to both sponsor and to wait. The two seem
mutually exclusive.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

Simon Riggs wrote:
> 
> On Mon, 2009-01-26 at 13:35 +0200, Heikki Linnakangas wrote:
>
> > Well, big features that land early in the release cycle don't delay the 
> > release. Just the ones that are submitted in the last commit fest.
> 
> Has that ever happened? :-)
> 
> I don't think its chance we get big patches in last commit fest. No
> sponsor I ever met was happy to both sponsor and to wait. The two seem
> mutually exclusive.

How come it works for Linux?

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Jonah H. Harris wrote:
> Looking forward, if no one
> wanted to review these patches in November, 

I did, and many others were active in the discussion too.

> and seemingly no one wants to review them now,

I do. Thank you for your appreciation :-(.

> how can we expect this to change for 8.5?  Can anyone point
> out something Simon did wrong in this process?

Not really, except maybe started 6 months too late. Big patches simply 
take a long time to mature.

Looking back at the timeline for hot standby, it doesn't look 
unreasonable at all:

September: First discussion about the user-visible behavior, transaction 
isolation etc. Big architectural decisions are made, like where that 
snapshots are taken. "Infrastructure changes for recovery" patch is 
reviewed, and pushed to next commitfest because of issues. Discussion 
started on how to handle (heavy-weight) locks.

October: Discussion on various administration commands, how to handle 
b-tree splits, and some other stuff. More discussion on how to derive 
snapshots. First complete hot standby patch is submitted. A patch to 
change the way subtransactions is submitted, and committed.

November: Various people test and review the patch, including Mark 
Kirkwood and Pavan Deolasee. Bugs are found and fixed. Decision is made 
that SIGINT should kill an idle-in-transaction transaction as well.

December: Discussion on slotids and the B-tree killed items problem.

January: Major changes; slotids eliminated, conflict resolution 
mechanism rewritten. RestoreBkpBlocks refactoring committed. More bugs 
in conflict resolution unearthed. Discussion on adding a GUC.

There has been steady progress, starting from basic design discussions 
and decisions, moving on to implementation details. Progress never 
stalled for a long time. I can see no wrongdoing on Simon's part, and 
there is also no grounds to say that the community has neglected this patch.

--   Heikki Linnakangas  EnterpriseDB   http://www.enterprisedb.com

>>> Robert Haas <robertmhaas@gmail.com> wrote: 
> Still, I agree that if there's anything we should be putting our
> effort into as a community right now, it's this feature.  If we got
> Hot Standby in the next release and everything else in the
CommitFest
> got bumped, I think a lot of people would consider that a good trade
Based on discussions at Milwaukee BarCamp, improvements in replication
would do more to break down barriers to migration to PostgreSQL than
anything else.
-Kevin

Robert Haas wrote:
> At a minimum, I think the following patches from the CommitFest wiki
> should be returned with feedback or rejected:
> 
> 1. SE-PostgreSQL.  We handled this one badly, but there's not enough
> time to fix it now.  8.5.

Unacceptable.
Please make it clear how many items should be fixed/reworked before
rejecting it due to the lack of time, at least.
(I'm optimistic for its design and quality.)
In addition, why does it impossible to be worked in parallel?
Maybe, I cannot contribute Simon's work, but I can fix/rework
for SE-PostgreSQL in parallel, with my highest priority.

As I noted in another message, the current trend strongly
requires "availability" and "security" for the platform of
SaaS/PaaS or cloud-computing. So, we should also include
SE-PostgreSQL within v8.4. I believe it makes PostgreSQL
more widespreadly applied.

> 2. rmgr hooks and contrib/rmgr_hook.  Reject because Tom and Heikki
> don't believe it's the right approach.  Need better use cases.
> 3. Synchronous log-shipping replication.  We handled this one well,
> but it's not in good enough shape.  8.5.

IMO, this feature is also key factor for "availablity".
If author can have enough activity, we should not give up right now...

> 4. pg_upgrade script.  I haven't heard much about this in a while...
> I am doubtful that it is production-quality, but maybe I'm wrong?
> 5. Reducing some DDL Locks to ShareLock.  No activity in a long time,
> no time to wait for this to be finished.  8.5.
> 
> ...Robert

-- 
KaiGai Kohei <kaigai@kaigai.gr.jp>

On Mon, 2009-01-26 at 12:05 -0300, Alvaro Herrera wrote:
> Simon Riggs wrote:
> > 
> > On Mon, 2009-01-26 at 13:35 +0200, Heikki Linnakangas wrote:
> >
> > > Well, big features that land early in the release cycle don't delay the 
> > > release. Just the ones that are submitted in the last commit fest.
> > 
> > Has that ever happened? :-)
> > 
> > I don't think its chance we get big patches in last commit fest. No
> > sponsor I ever met was happy to both sponsor and to wait. The two seem
> > mutually exclusive.
> 
> How come it works for Linux?

Not sure. How come up it happens here is a more relevant question.

I can only tell you that people only get their cheque books out when
they are certain somebody else will not do it for them for free. 

This happened to you also, or did you have this terrible desire for CRC
checks on data blocks quite late in release cycle? If there hadn't been
blockers you'd be being lightly barbecued on the patch queue also. :-)

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

Simon Riggs <simon@2ndQuadrant.com> writes:

> I fail to see how rejecting unreviewed patches provides benefit for
> users, developers or sponsors. 

Nobody has suggested rejecting either sync replication or standby database.
The debate here is over whether to commit into 8.4 or into 8.5.

Put another way, the choice here is whether to have a half-baked delayed 8.4
release in 6 months or a polished on-time 8.5 release in 12 months. Either way
the feature ships and on a not terribly different timeline either.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's PostGIS support!

Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:

> Not really, except maybe started 6 months too late. Big patches simply take a
> long time to mature.
>
> Looking back at the timeline for hot standby, it doesn't look unreasonable at
> all:
>
> September: First discussion about the user-visible behavior, transaction
> isolation etc. 

Is that right? I had the impression Simon had started working on it
previously. If so then given that feature freeze was scheduled to be November
1st starting in September does seem like it was more than a bit unrealistic.

Large patches should really be targeted towards the *first* commitfest of the
cycle, not the last. Targeting the last is putting all your eggs in one basket
as far as getting everything right the first time.

Someone else asked why this works for Linux. The reason it works is because
work is *always* committed early in the release cycle. The first couple weeks
of the release cycle are the *only* time significant patches are taken. 

The entire rest of the cycle is spent in feature freeze with development
happening exclusively on subsystem maintainer's private branches. When the
next release cycle begins subsystem maintainers send up patches for all the
changes that have happened since the last cycle that they think are ready for
release.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production
Tuning

Gregory Stark <stark@enterprisedb.com> writes:
> Put another way, the choice here is whether to have a half-baked delayed 8.4
> release in 6 months or a polished on-time 8.5 release in 12 months. Either way
> the feature ships and on a not terribly different timeline either.

This is pretty much exactly how I see it.  *Hot standby is not ready*,
and committing it into 8.4 isn't going to magically make that better.
The earliest we are going to have a HS feature that I would trust my
data to is probably ten-twelve months off.  The decision we need to
make now is whether that release will be called 8.4 or 8.5; in the
former case meaning that all the stuff already in 8.4 will not reach
users' hands for close to a year more.
        regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> writes:

> Gregory Stark <stark@enterprisedb.com> writes:
>> Put another way, the choice here is whether to have a half-baked delayed 8.4
>> release in 6 months or a polished on-time 8.5 release in 12 months. Either way
>> the feature ships and on a not terribly different timeline either.
>
> This is pretty much exactly how I see it.  *Hot standby is not ready*,
> and committing it into 8.4 isn't going to magically make that better.
> The earliest we are going to have a HS feature that I would trust my
> data to is probably ten-twelve months off.  The decision we need to
> make now is whether that release will be called 8.4 or 8.5; in the
> former case meaning that all the stuff already in 8.4 will not reach
> users' hands for close to a year more.

I'm not even so much worried about the stuff already in 8.4. I'm worried about
the queue for the first 8.5 commitfest. Nobody's going to be reviewing new
patches until 8.4 is out which will leave all that stuff sitting in limbo for
months. Worse, it'll take months to sort through it and nobody will be able to
work on anything which depends on stuff in that queue for, uh, 2*months.

This is stuff like sync replication, join removal, the posix_fadvise for WAL
replay, index-only scans, etc. If this stuff isn't ready to be committed until
next September we'll be back in the same boat again.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's PostGIS support!

On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Gregory Stark <stark@enterprisedb.com> writes:
>> Put another way, the choice here is whether to have a half-baked delayed 8.4
>> release in 6 months or a polished on-time 8.5 release in 12 months. Either way
>> the feature ships and on a not terribly different timeline either.
>
> This is pretty much exactly how I see it.  *Hot standby is not ready*,
> and committing it into 8.4 isn't going to magically make that better.
> The earliest we are going to have a HS feature that I would trust my
> data to is probably ten-twelve months off.

So can you give us an idea of what parts of the code are in need of
rethinking etc? I assume you've looked at it now if you can estimate
it's going to take another 10 -12 months?

I ask because I've only seen Heikki doing any in depth public review,
and he suggested a couple of weeks for the outstanding issues he's
aware of.

If there are fundamental problems which will take 10 - 12 months to
resolve to our normal standards, then I do believe 8.5 would be more
appropriate.

-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:
> Jonah H. Harris wrote:
>> how can we expect this to change for 8.5?  Can anyone point
>> out something Simon did wrong in this process?

> Not really, except maybe started 6 months too late. Big patches simply 
> take a long time to mature.

Yeah, exactly.  When we decided at PgCon that we needed to accept
replication into the core, we thought that there was a realistic chance
of it being in 8.4 if full-tilt development started *immediately*.
For one reason or another nothing much happened until September.
Simon's certainly been working hard on it since then, but it just
takes longer than four months for something like this to happen.

The alternatives that we realistically have got are:
* ship 8.4 "now" (ie before the summer) without HS, plan for HS in 8.5
* delay 8.4 to late summer and ship it with an unpolished HS feature
* delay 8.4 long enough to have a stable, polished HS feature; I claim this will mean it's not out before Christmas.

I think anyone who thinks alternatives 2 or 3 will happen faster than
sketched here is living in dreamland.

So my feeling is that prudent project management would suggest that
HS go into 8.5 near the beginning of that development cycle, rather
than trying to push it into 8.4.
        regards, tom lane

On 1/26/09, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Gregory Stark <stark@enterprisedb.com> writes:
>  > Put another way, the choice here is whether to have a half-baked delayed 8.4
>  > release in 6 months or a polished on-time 8.5 release in 12 months. Either way
>  > the feature ships and on a not terribly different timeline either.
>
>
> This is pretty much exactly how I see it.  *Hot standby is not ready*,
>  and committing it into 8.4 isn't going to magically make that better.
>  The earliest we are going to have a HS feature that I would trust my
>  data to is probably ten-twelve months off.  The decision we need to
>  make now is whether that release will be called 8.4 or 8.5; in the
>  former case meaning that all the stuff already in 8.4 will not reach
>  users' hands for close to a year more.

What about a compromise solution: release 8.4 now, then focus on
wrapping up the big ticket items that didn't make it into 8.4 into a
quick (as possible) 8.5 release.  This means no fests.

merlin

Dave Page <dpage@pgadmin.org> writes:
> On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> This is pretty much exactly how I see it.  *Hot standby is not ready*,

> So can you give us an idea of what parts of the code are in need of
> rethinking etc? I assume you've looked at it now if you can estimate
> it's going to take another 10 -12 months?

No, I'm just estimating that based on the amount of design churn that's
still going on according to the mailing list discussions.  I haven't
looked at the code at all.  (If you expect me to sign off on it you can
figure it'll be a couple of months even for that to happen...)
        regards, tom lane

On Mon, Jan 26, 2009 at 4:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Dave Page <dpage@pgadmin.org> writes:
>> On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> This is pretty much exactly how I see it.  *Hot standby is not ready*,
>
>> So can you give us an idea of what parts of the code are in need of
>> rethinking etc? I assume you've looked at it now if you can estimate
>> it's going to take another 10 -12 months?
>
> No, I'm just estimating that based on the amount of design churn that's
> still going on according to the mailing list discussions.  I haven't
> looked at the code at all.  (If you expect me to sign off on it you can
> figure it'll be a couple of months even for that to happen...)

Well there is one of the problems imho - the project is getting too
big and the patches are getting too complex for us to be able to rely
on you to sign off on every feature or major patch. We need put more
trust in the judgement of the other committers and senior developers
otherwise the project will simply bottleneck as you get more and more
overworked.

/me hopes that comes across as intended :-)

-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

"Jonah H. Harris" <jonah.harris@gmail.com> writes:
> On Mon, Jan 26, 2009 at 11:11 AM, Merlin Moncure <mmoncure@gmail.com> wrote:
>> What about a compromise solution: release 8.4 now, then focus on
>> wrapping up the big ticket items that didn't make it into 8.4 into a
>> quick (as possible) 8.5 release.  This means no fests.

> That would depend on timing then.  Trying to get people to upgrade to 8.4 is
> going to be difficult if they're waiting on Hot Standby, which means less
> in-the-field testing of the 8.4 code base until the 8.5 release.

[ deja vu... ]  Just like no one was going to bother upgrading to 8.3
because what they wanted wouldn't be there till 8.4, and the similar
claims we heard about 8.2 and 8.1 before that ...

> Similarly,
> if we're looking at a quick 8.5 around September/October (having no commit
> fests), that means it will probably be early 2011 for 8.6, which is fairly
> unacceptable for the other patches currently in the queue.

Right, one of the major considerations here is allowing other
development to get started again (and not be looking at two years wait
to see the light of day).
        regards, tom lane

On 1/26/09, Jonah H. Harris <jonah.harris@gmail.com> wrote:
> On Mon, Jan 26, 2009 at 11:11 AM, Merlin Moncure <mmoncure@gmail.com> wrote:
> > What about a compromise solution: release 8.4 now, then focus on
> > wrapping up the big ticket items that didn't make it into 8.4 into a
> > quick (as possible) 8.5 release.  This means no fests.
>
> That would depend on timing then.  Trying to get people to upgrade to 8.4 is
> going to be difficult if they're waiting on Hot Standby, which means less
> in-the-field testing of the 8.4 code base until the 8.5 release.  Similarly,
> if we're looking at a quick 8.5 around September/October (having no commit
> fests), that means it will probably be early 2011 for 8.6, which is fairly
> unacceptable for the other patches currently in the queue.

I think that's baked in the cards no matter what.  IMO, the issue is
that code to review is building up faster than it is getting reviewed,
so maybe a festless release is nesessary to flush out the difference
(along with the other held over patches from 8.4).

The other thing going on here is that HS is extremely important to the
project.  If it was up to me, 100% effort would be geared to getting
it out as quickly as possible.  I'm just looking for a way to get HS
in the hands of people as quickly as possible that is fair to
everybody.  (I would actually prefer to hold 8.4 for it, but that's
starting to sound unrealistic based on Tom's comments).

merlin

Tom Lane <tgl@sss.pgh.pa.us> writes:

> Dave Page <dpage@pgadmin.org> writes:
>> On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> This is pretty much exactly how I see it.  *Hot standby is not ready*,
>
>> So can you give us an idea of what parts of the code are in need of
>> rethinking etc? I assume you've looked at it now if you can estimate
>> it's going to take another 10 -12 months?
>
> No, I'm just estimating that based on the amount of design churn that's
> still going on according to the mailing list discussions.  

That's, uh, not a very useful metric. It should argue in *favour* of the patch
being near completion that the patch is getting feedback and Simon is rapidly
taking such suggestions to heart and making changes responsively.

And incidentally most of the changes Heikki's been making have been
simplifications. Every bit which is simplified is one fewer area to find
further problems in. It's not like he's been finding problems which require
new code which in turn could have more problems.

> I haven't looked at the code at all. (If you expect me to sign off on it you
> can figure it'll be a couple of months even for that to happen...)

Well, given that you're basing your judgements on the mailing list traffic and
haven't read the code I guess we have to go on Heikki's judgement that there
are about two known weeks of work and possibly more afterwards.

The real question is how long will the beta period be with or without Hot
Standby? How many users have been pounding on it so far and how many more will
we get in beta?

If we assume it's about a month before the patch is committed, and then the
beta period is an extra month or two long (on top of what I assume will be
about a month) then we're looking at 3-4 months before 8.4 or about May.

I fear that 3-4 months will already be long enough to back up development on
8.5 and cause a big queue for the first commitfest. Frankly, at 3 months I
think feature freeze has _already_ gone on too long, I don't think we need the
doom scenario of 10-12 months before we create a serious problem again.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production
Tuning

Dave Page wrote:
> On Mon, Jan 26, 2009 at 4:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Dave Page <dpage@pgadmin.org> writes:
>>> On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>> This is pretty much exactly how I see it.  *Hot standby is not ready*,
>>> So can you give us an idea of what parts of the code are in need of
>>> rethinking etc? I assume you've looked at it now if you can estimate
>>> it's going to take another 10 -12 months?
>> No, I'm just estimating that based on the amount of design churn that's
>> still going on according to the mailing list discussions.  I haven't
>> looked at the code at all.  (If you expect me to sign off on it you can
>> figure it'll be a couple of months even for that to happen...)
> 
> Well there is one of the problems imho - the project is getting too
> big and the patches are getting too complex for us to be able to rely
> on you to sign off on every feature or major patch. We need put more
> trust in the judgement of the other committers and senior developers
> otherwise the project will simply bottleneck as you get more and more
> overworked.

About another issued patch, SE-PostgreSQL has been also said
large and complex. However, I cannot believe the reputation
reflects the correct shape.

As I noted in another message, the main patch of SE-PostgreSQL
changes 110 files, inserts 9813 lines, deletes 16 lines and
modifies 924 lines. It might seem the patch is large/complex.

However, 8207 of 9813 lines are on "src/backend/security/*" and
"src/include/security/*" because of its modular design.
It means most of patch simply adds new files.
The 504 of 924 lines modifications are due to a new field in
pg_attribute system catalog, so it is obvious change.
Rest of relatively-large changes are 293 lines in copy.c and
209 lines in execMain.c.

I don't think it is too complex patch to hesiate reviewing.
So, I believe we can move it to "ready to commit" state
within reasonable term, as long as we make it progress.

Thanks,

[kaigai@fedora10 ~]$ diffstat sepostgresql-sepgsql-8.4devel-3-r1467.patch configure
| 113 + configure.in                                  |   13 src/Makefile.global.in                        |    1
src/backend/Makefile                         |    7 src/backend/access/common/heaptuple.c         |   35
src/backend/access/common/reloptions.c       |   22 src/backend/access/common/tupdesc.c           |   12
src/backend/access/heap/heapam.c             |   19 src/backend/access/heap/tuptoaster.c          |   19
src/backend/bootstrap/bootparse.y            |   13 src/backend/bootstrap/bootstrap.c             |    8
src/backend/catalog/Makefile                 |    1 src/backend/catalog/aclchk.c                  |    2
src/backend/catalog/catalog.c                |    4 src/backend/catalog/heap.c                    |   91 !
src/backend/catalog/index.c                  |   16 src/backend/catalog/pg_aggregate.c            |    3
src/backend/catalog/pg_largeobject.c         |    5 src/backend/catalog/pg_proc.c                 |    6
src/backend/catalog/toasting.c               |    3 src/backend/commands/cluster.c                |   11
src/backend/commands/copy.c                  |  293 +++! src/backend/commands/dbcommands.c             |   20
src/backend/commands/functioncmds.c          |   29 src/backend/commands/lockcmds.c               |    3
src/backend/commands/proclang.c              |    6 src/backend/commands/tablecmds.c              |   25
src/backend/commands/trigger.c               |   25 src/backend/executor/execJunk.c               |    6
src/backend/executor/execMain.c              |  209 +++ src/backend/executor/execQual.c               |    4
src/backend/executor/execScan.c              |   40 src/backend/executor/execTuples.c             |   19
src/backend/executor/execUtils.c             |   10 src/backend/executor/functions.c              |    6
src/backend/executor/nodeAgg.c               |    5 src/backend/executor/nodeMergejoin.c          |    2
src/backend/executor/nodeSubplan.c           |    4 src/backend/executor/nodeWindowAgg.c          |    4
src/backend/executor/spi.c                   |    4 src/backend/libpq/be-fsstubs.c                |   16
src/backend/nodes/copyfuncs.c                |   30 src/backend/nodes/equalfuncs.c                |   22
src/backend/nodes/outfuncs.c                 |   34 src/backend/nodes/readfuncs.c                 |   43
src/backend/optimizer/plan/createplan.c      |    6 src/backend/optimizer/plan/planner.c          |    1
src/backend/optimizer/util/clauses.c         |    5 src/backend/optimizer/util/relnode.c          |    1
src/backend/parser/analyze.c                 |   46 src/backend/parser/gram.y                     |   64 !
src/backend/parser/parse_target.c            |   64 ! src/backend/postmaster/postmaster.c           |   43
src/backend/rewrite/rewriteHandler.c         |    3 src/backend/security/Makefile                 |   23
src/backend/security/pgaceCommon.c           |  729 ++++++++++++ src/backend/security/pgaceHooks.c             | 1547
++++++++++++++++++++++++++src/backend/security/rowacl/rowacl.c          |  721 ++++++++++++
src/backend/security/sepgsql/avc.c           | 1200 ++++++++++++++++++++ src/backend/security/sepgsql/core.c
| 623 ++++++++++ src/backend/security/sepgsql/hooks.c          | 1019 +++++++++++++++++
src/backend/security/sepgsql/permissions.c   |  785 +++++++++++++ src/backend/security/sepgsql/proxy.c          | 1097
++++++++++++++++++src/backend/storage/file/fd.c                 |    7 src/backend/storage/ipc/ipci.c                |
 2 src/backend/tcop/fastpath.c                   |    2 src/backend/tcop/pquery.c                     |    2
src/backend/tcop/utility.c                   |    3 src/backend/utils/adt/acl.c                   |    6
src/backend/utils/adt/ri_triggers.c          |   25 src/backend/utils/adt/trigfuncs.c             |   11
src/backend/utils/cache/catcache.c           |   32 src/backend/utils/cache/plancache.c           |   12
src/backend/utils/cache/relcache.c           |   38 src/backend/utils/cache/syscache.c            |   40
src/backend/utils/fmgr/dfmgr.c               |   10 src/backend/utils/init/postinit.c             |    4
src/backend/utils/misc/guc.c                 |   58 src/backend/utils/misc/postgresql.conf.sample |    6
src/include/access/htup.h                    |   68 + src/include/access/sysattr.h                  |    9
src/include/access/tupdesc.h                 |    2 src/include/catalog/heap.h                    |   11
src/include/catalog/indexing.h               |    5 src/include/catalog/pg_attribute.h            |  504 !!!!!!!!
src/include/catalog/pg_class.h               |    2 src/include/catalog/pg_proc.h                 |   21
src/include/catalog/pg_proc_fn.h             |    3 src/include/catalog/pg_security.h             |   31
src/include/catalog/pg_type.h                |    1 src/include/executor/executor.h               |   11
src/include/executor/tuptable.h              |    4 src/include/fmgr.h                            |    3
src/include/libpq/be-fsstubs.h               |    3 src/include/nodes/nodes.h                     |    3
src/include/nodes/parsenodes.h               |   17 src/include/nodes/plannodes.h                 |   10
src/include/nodes/relation.h                 |    2 src/include/nodes/security.h                  |   40
src/include/pg_config.h.in                   |    3 src/include/security/pgace.h                  |  181 +++
src/include/security/rowacl.h                |   41 src/include/security/sepgsql.h                |  241 ++++
src/include/storage/fd.h                     |    1 src/include/storage/lwlock.h                  |    1
src/include/utils/acl.h                      |    7 src/include/utils/catcache.h                  |    1
src/include/utils/errcodes.h                 |    7 src/include/utils/rel.h                       |   18
src/include/utils/syscache.h                 |    4 110 files changed, 9813 insertions(+), 16 deletions(-), 924
modifications(!)

-- 
KaiGai Kohei <kaigai@kaigai.gr.jp>

Dave Page <dpage@pgadmin.org> writes:
> On Mon, Jan 26, 2009 at 4:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> ...  (If you expect me to sign off on it you can
>> figure it'll be a couple of months even for that to happen...)

> Well there is one of the problems imho - the project is getting too
> big and the patches are getting too complex for us to be able to rely
> on you to sign off on every feature or major patch. We need put more
> trust in the judgement of the other committers and senior developers
> otherwise the project will simply bottleneck as you get more and more
> overworked.

Well, actually, Bruce and I are pretty much leaving it to Heikki's
judgement as to when this patch will be ready to commit.  He's spent
far more time on it than either of us.  The reason I started this thread
is not to debate when it'll be ready but whether we are prepared to
delay 8.4 for however long it will take, realizing that optimistic
estimates are very likely to be wrong.
        regards, tom lane

On Mon, Jan 26, 2009 at 11:44 AM, Jonah H. Harris
<jonah.harris@gmail.com> wrote:
> On Mon, Jan 26, 2009 at 11:33 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>
>> > That would depend on timing then.  Trying to get people to upgrade to
>> > 8.4 is
>> > going to be difficult if they're waiting on Hot Standby, which means
>> > less
>> > in-the-field testing of the 8.4 code base until the 8.5 release.
>>
>> [ deja vu... ]  Just like no one was going to bother upgrading to 8.3
>> because what they wanted wouldn't be there till 8.4, and the similar
>> claims we heard about 8.2 and 8.1 before that ...
>
> I'm not trying to be an alarmist, I'm just stating what I saw when I was @
> EDB.  Customers, especially those with large databases or small admin teams,
> would definitely wait for features before upgrading.  Some people waited
> specifically for HOT or features that would benefit them specifically.  My
> only gripe with a small window between 8.4 and 8.5 was just that I believe
> people would be more likely to wait until 8.5 rather than upgrading twice in
> the same year.  Though, as I generally like people to be using the latest
> version of PG, I'd certainly be happy to be wrong on this.

We've got folks working on the upgrade to 8.3; 8.4 isn't on our radar
yet, particularly in view of the fact that it's getting pretty
nebulous when 8.4 will be production-worthy :-).

It would surprise me not at all if we never got around to an 8.4
deployment, irrespective of whether we hold off another 6 months to
get more of hot standby into it or not.

Based on that metric, I suppose I ought to prefer for us to get 8.4
out the door more quickly, and start seeing work progress on the 8.5
backlog, which would presumably include the two large O/S patches
(e.g. - hot standby, SE-PG).

I suppose there are two, possibly 3 directions to consider, with
respective merits and demerits:

1.  Suppose we cut things off now, and say "push that stuff to 8.5"...

- Simon and KaiGai will be justifiably angry since they *have* been
doing the right kinds of things, and were expecting to see their
material in 8.4.

- On the other hand, their work would be among the first considered
for 8.5, so that they'd not be behind the way they were for 8.4.

- We *won't* irritate the somewhat numerous set of people with patches
already awaiting
8.5,<http://wiki.postgresql.org/wiki/CommitFest_2009-First> and allow
those patches to "bit rot" in the interim.

2.  Alternatively, we may press on...

- We may put off 8.4 for an extra 4-6 months.

- Everyone loses the utility of the features already committed in 8.4
for that extra period of time.

- We irritate everyone that was accepting that their contributions
would be waiting until the first 8.5 commitfest.

3.  There's always the possibility of a "worst of all worlds."

- All the irritations and losses of #2

- For whatever reason, we don't get usable hot standby / SEPostgreSQL
at the end of it.

I'm getting increasingly scared of #2 and #3.

From a purely selfish perspective, neither of these features are ones
I was expecting to be using in the short or medium term (I don't
actually see a use case for SELinux, myself).  I'll note that as my
bias, but I don't think that invalidates the analysis of the 3
directions.
-- 
http://linuxfinances.info/info/linuxdistributions.html
Robert Benchley  - "Drawing on my fine command of the English
language, I said nothing."

> I think that's baked in the cards no matter what.  IMO, the issue is
> that code to review is building up faster than it is getting reviewed,
> so maybe a festless release is nesessary to flush out the difference
> (along with the other held over patches from 8.4).

How is that going to help?  People will still write new code in the
meanwhile and some of it may be better or more important than the
stuff that doesn't get committed to 8.4.  Artificially saying that
nothing will be allowed for 8.5 that wasn't planned for 8.4 doesn't
seem to me to solve anything.

I think we've actually done a very fine job reviewing code for 8.4.
Take a look at how many patches were reviewed and committed.  The
problem is that we didn't review the big ones.  But at least a part of
that is due to some oversights in assigning reviewers, rather than an
absolute lack of manpower.

> The other thing going on here is that HS is extremely important to the
> project.  If it was up to me, 100% effort would be geared to getting
> it out as quickly as possible.  I'm just looking for a way to get HS
> in the hands of people as quickly as possible that is fair to
> everybody.  (I would actually prefer to hold 8.4 for it, but that's
> starting to sound unrealistic based on Tom's comments).

+1.

...Robert

Robert Haas wrote:
>> The other thing going on here is that HS is extremely important to the
>> project.  If it was up to me, 100% effort would be geared to getting
>> it out as quickly as possible.  I'm just looking for a way to get HS
>> in the hands of people as quickly as possible that is fair to
>> everybody.  (I would actually prefer to hold 8.4 for it, but that's
>> starting to sound unrealistic based on Tom's comments).
> 
> +1.

+1 Too.

Really, the most interesting part of 8.4 for us is sync_rep and hot_standby!

Best regards,
-- 
Laurent COUSTET

On Mon, 2009-01-26 at 12:47 -0500, Robert Haas wrote:

> How is that going to help?  People will still write new code in the
> meanwhile and some of it may be better or more important than the
> stuff that doesn't get committed to 8.4.  Artificially saying that
> nothing will be allowed for 8.5 that wasn't planned for 8.4 doesn't
> seem to me to solve anything.
> 
> I think we've actually done a very fine job reviewing code for 8.4.

We have. This release has been better than any other release in recent
memory. We have had more reviewers, and a more transparent process. That
is a great thing and should only stand to improve for the next release.

> Take a look at how many patches were reviewed and committed.  The
> problem is that we didn't review the big ones.  

I am not sure this is entirely true. SE Postgres for example has been in
constant review by Bruce and Hot Standby has been in constant review by
Heikki.

I think specifically these two patches suffer from two very distinct
problems.

1. This community in general doesn't understand SE Linux/Postgres. It
also struggles with the very Linux centric nature of it. 

2. Hot Standby has two very distinct problems of its own.
  A. It is a very much buzzword compliant feature. 
  B. It *appears* to be very much in dynamic development. The chatter
on the list alone is enough to make conservative hackers very nervous.
Myself included.

I am personally not in favor of delaying 8.4. To me Hot Standby is patch
#8762. It isn't going to make the cut because of timing. That is life.
It will be one of the first in the queue for 8.5. 

Or put another way. Nobody would blink about cutting the new \dfU stuff
in favor of getting 8.4 out on time. Why are we delaying for Hot
Standby? 


Sincerely,

Joshua D. Drake



-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

All,

So, some feedback to make this decision more difficult:

Users: care about HS more than anything else in the world.  I'm 
convinced that if we took a staw poll, 80% of our users would be in 
favor of waiting for HS.  This one feature will make more of a 
difference in the number of PG users than any feature since the Windows 
port.  Maybe more.

on the other hand:

We held back version 4 months 7.4 for Windows, before it became apparent 
that there was at least a year more work to do.  That was a mistake, and 
in many ways HS seems like a similar case.


SE-Linux:  this patch has effectively been in development for 2 years 
ourside the core process before putting it in; the forked SEPostgres is 
in use in production. KaiGai has been available for 20 hours a week (or 
more) to troubleshoot issues and change APIs.  I really don't see what 
the problem is with committing it.

pg_upgrade hasn't recieved a lot of testing because 8.4 has been such a 
moving target.  I've been waiting for it to settle down so that we can 
see if upgrade works.  It was always true that pg_upgrade would be among 
the last patches tested; we discussed this at pgCon.

==============

Regarding the Commitfests in general:  we still haven't perfected this 
process.  There are a number of issues with it which are hampered by 
technology, but a lot more by people.  Here's my analysis of what's 
changed over the last year:

1) having the last CF on Nov. 1 was a mistake.  That put us square in 
the path of the US & Christian holidays during the critical integration 
phase .. which means we haven't really had 3 months of integration, 
we've had *two*.

2) Having the CFs improves visibility.  However, as SEPostgres shows, it 
doesn't eliminate the ability of major committers to put off dealing 
with large complex patches which junior reviewers can't be assigned to.  Particularly if a major reviewer "claims" a
patch,but then doesn't 
 
seem to do a lot of review.

3) I don't feel like I got a real handle on the "Round Robin Reviewers" 
and got them processing small patches efficiently until the November 
review.  It just took several cycles to work out how to do it, 
particularly given my job change this year.

Better technology would also help, such as automated tracking of patch 
changes and when the last time a reviewer spoke up was.  Currently, Dave 
and I have been doing these things by hand and I know we missed a lot of 
patches which stalled.  But the main issue is (and will remain) people 
and procrastination.

--Josh

On Mon, 2009-01-26 at 11:36 -0800, Josh Berkus wrote:
> All,
> 
> So, some feedback to make this decision more difficult:
> 
> Users: care about HS more than anything else in the world.  I'm 
> convinced that if we took a staw poll, 80% of our users would be in 
> favor of waiting for HS.  This one feature will make more of a 
> difference in the number of PG users than any feature since the Windows 
> port.  Maybe more.

Maybe. Most of the people I talk to are more interested in Recursive
Queries and Parallel restore. Not that they don't see the benefit of HS
of course just that they already have in place solutions for that
problem.

> 
> on the other hand:
> 
> We held back version 4 months 7.4 for Windows, before it became apparent 
> that there was at least a year more work to do.  That was a mistake, and 
> in many ways HS seems like a similar case.

Had to read this twice. I think you mean we held back 7.4 four 4 months.
Which I had actually forgotten.

> SE-Linux:  this patch has effectively been in development for 2 years 
> ourside the core process before putting it in; the forked SEPostgres is 
> in use in production. KaiGai has been available for 20 hours a week (or 
> more) to troubleshoot issues and change APIs.  I really don't see what 
> the problem is with committing it.

As I posted earlier I think it is mostly understanding of the
technology. 

> 
> pg_upgrade hasn't recieved a lot of testing because 8.4 has been such a 
> moving target.  I've been waiting for it to settle down so that we can 
> see if upgrade works.  It was always true that pg_upgrade would be among 
> the last patches tested; we discussed this at pgCon.
> 

I thought pg_upgrade was dead. I am happy to hear it isn't. When can we
see a patch that works on anything?

> 
> 1) having the last CF on Nov. 1 was a mistake.  That put us square in 
> the path of the US & Christian holidays during the critical integration 
> phase .. which means we haven't really had 3 months of integration, 
> we've had *two*.
> 

Its always something though. Its either Christmas or summer or school
starting or school ending or spring break etc...

> Better technology would also help, such as automated tracking of patch 
> changes and when the last time a reviewer spoke up was.  Currently, Dave 
> and I have been doing these things by hand and I know we missed a lot of 
> patches which stalled.  But the main issue is (and will remain) people 
> and procrastination.
> 

Nod.

Sincerely,

Joshua D. Drake


> --Josh
> 
> 
> 
> 
-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

On 1/26/09, Josh Berkus <josh@agliodbs.com> wrote:
> All,
>
>  So, some feedback to make this decision more difficult:
>
>  Users: care about HS more than anything else in the world.  I'm convinced
> that if we took a staw poll, 80% of our users would be in favor of waiting
> for HS.  This one feature will make more of a difference in the number of PG
> users than any feature since the Windows port.  Maybe more.
>
>  on the other hand:
>
>  We held back version 4 months 7.4 for Windows, before it became apparent
> that there was at least a year more work to do.  That was a mistake, and in
> many ways HS seems like a similar case.

Not completely.  HS is in much better shape than win32 was when it was
pulled from 7.4...the build system wasn't even in place yet nor any of
the major challenges solved (like fork/exec).

HS is working very well (Simon's ongoing work aside).  I am pretty
confident based on my personal testing that it would represent the
project well if committed today.

merlin

Merlin,

> Not completely.  HS is in much better shape than win32 was when it was
> pulled from 7.4...the build system wasn't even in place yet nor any of
> the major challenges solved (like fork/exec).
> 
> HS is working very well (Simon's ongoing work aside).  I am pretty
> confident based on my personal testing that it would represent the
> project well if committed today.

OK.  I haven't tried it since early December; that's why I was just 
presenting alternatives.

--Josh

On Mon, Jan 26, 2009 at 2:36 PM, Josh Berkus <josh@agliodbs.com> wrote:
> All,
>
> So, some feedback to make this decision more difficult:
>
> Users: care about HS more than anything else in the world.  I'm convinced
> that if we took a staw poll, 80% of our users would be in favor of waiting
> for HS.  This one feature will make more of a difference in the number of PG
> users than any feature since the Windows port.  Maybe more.
>

if we think we will need a lot of time to get this in form, there will
be no difference in the time of the release just in the numbers in
which HS comes

>
> SE-Linux:  this patch has effectively been in development for 2 years
> ourside the core process before putting it in; the forked SEPostgres is in
> use in production. KaiGai has been available for 20 hours a week (or more)
> to troubleshoot issues and change APIs.  I really don't see what the problem
> is with committing it.
>

it hasn't been testing by ours in different platforms (ie: ubuntu has
selinux and i want to give it a try, badly enough i have never used
selinux so this is new to me)...

nor we have any evidence that it doesn't affect to users that doesn't
have any variant of selinux (ie: windows)... the real problem here is
the base of users we enough knowledge of the tool to make some usefull
tests

> ==============
>
> Regarding the Commitfests in general:  we still haven't perfected this
> process.  There are a number of issues with it which are hampered by
> technology, but a lot more by people.  Here's my analysis of what's changed
> over the last year:
>
> 1) having the last CF on Nov. 1 was a mistake.  That put us square in the
> path of the US & Christian holidays during the critical integration phase ..
> which means we haven't really had 3 months of integration, we've had *two*.
>

+1

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

All,

>> 1) having the last CF on Nov. 1 was a mistake.  That put us square in the
>> path of the US & Christian holidays during the critical integration phase ..
>> which means we haven't really had 3 months of integration, we've had *two*.

Actually, I'm thinking about this again, and made a mistake about the 
mistake.  The *original plan* was that we were not going to accept any 
new patches for Nov-CF.  Just revised patches from eariler Fests.  We 
didn't stick to that, which is mostly why we are still reviewing now.

--Josh

Josh Berkus <josh@agliodbs.com> writes:
> So, some feedback to make this decision more difficult:

> Users: care about HS more than anything else in the world.

I don't think this is correct.  There are certainly a lot of users who
would like an in-core replication solution, but HS by itself is not that
--- you also need (near) real-time log shipping, which we have already
decided to punt to 8.5.  That being the case, I think the argument
that HS is a must-have feature for 8.4 is actually rather weak.

> SE-Linux:  this patch has effectively been in development for 2 years 
> ourside the core process before putting it in; the forked SEPostgres is 
> in use in production. KaiGai has been available for 20 hours a week (or 
> more) to troubleshoot issues and change APIs.  I really don't see what 
> the problem is with committing it.

The problem, in words of one syllable, is that we are not sure we want
it.  Do you see a user community clamoring for SEPostgres, or a hacker
community that is willing or able to maintain it?  If KaiGai-san got run
over by a bus tomorrow, this patch would be a dead letter, because there
just isn't anyone else who is taking sufficient (any?) interest in it.
That doesn't bode well for its future viability.  Compare the likely
audience for it to previous patches of roughly similar complexity,
such as integrated text search or the Windows port, and it's just not
in the ballpark.

The second problem is that we're not sure it's really the right thing,
because we have no one who is competent to review the design from a
security standpoint.  But unless we get past the first problem the
second one is moot.
        regards, tom lane

Merlin Moncure <mmoncure@gmail.com> writes:

> HS is working very well (Simon's ongoing work aside).  I am pretty
> confident based on my personal testing that it would represent the
> project well if committed today.

I think a lot of people weren't aware there was anybody testing this patch
other than Simon and Heikki -- I wasn't until just today. I wonder how many
more people are trying it out?

This is one of the changes for the better from the past. Though I still think
a lot more would try it out once it's committed.

Here's a thought experiment. If it was committable *today* would we be willing
to go with it and plan to release with it? Assume that it would *still* mean a
longer beta process, so it would still mean releasing in, say April instead of
February or March.

While personally I want us to switch to a mode where we commit large patches
early in the development cycle I don't believe we would refuse to commit it
today if it was ready. And I can't imagine two weeks would make the
difference either.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Get trained by Bruce Momjian - ask me about
EnterpriseDB'sPostgreSQL training!
 

On 1/26/09, Josh Berkus <josh@agliodbs.com> wrote:
> All,
>
>>> 1) having the last CF on Nov. 1 was a mistake.  That put us square in the
>>> path of the US & Christian holidays during the critical integration phase
>>> ..
>>> which means we haven't really had 3 months of integration, we've had
>>> *two*.
>
> Actually, I'm thinking about this again, and made a mistake about the
> mistake.  The *original plan* was that we were not going to accept any
> new patches for Nov-CF.  Just revised patches from eariler Fests.  We
> didn't stick to that, which is mostly why we are still reviewing now.

I don't recall us discussing that, but it sounds like it might help next cycle.

-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

On 1/26/09, Gregory Stark <stark@enterprisedb.com> wrote:
>
>  Merlin Moncure <mmoncure@gmail.com> writes:
>
>  > HS is working very well (Simon's ongoing work aside).  I am pretty
>  > confident based on my personal testing that it would represent the
>  > project well if committed today.
>
>
> I think a lot of people weren't aware there was anybody testing this patch
>  other than Simon and Heikki -- I wasn't until just today. I wonder how many
>  more people are trying it out?
>
>  This is one of the changes for the better from the past. Though I still think
>  a lot more would try it out once it's committed.
>
>  Here's a thought experiment. If it was committable *today* would we be willing
>  to go with it and plan to release with it? Assume that it would *still* mean a
>  longer beta process, so it would still mean releasing in, say April instead of
>  February or March.

I think the best person to answer that is Simon.  He got a small
reprieve the last time this discussion came up, but it's 'put up or
shut up time'...time's up.  Is the thing ready to go?

While i'm 'pro HS', I'm also very much 'pro get 8.4 out asap'.  I
think both _may_ be possible.  This is probably horribly optimistic.
What I can do however is offer to summarize my own experiences testing
and invite others to do the same.  I'd also like to see Simon and or
Heikki make a strong statement on where things stand _right now_ (not
in two weeks) :-)

merlin

Gregory Stark <stark@enterprisedb.com> writes:
> Here's a thought experiment. If it was committable *today* would we be
> willing to go with it and plan to release with it? Assume that it
> would *still* mean a longer beta process, so it would still mean
> releasing in, say April instead of February or March.

"April instead of February or March"?  If we cut off *all* further patch
acceptance *today*, we might conceivably be ready to release May 1.
It is not physically possible to get to beta from where we are before
mid-February, given the open issues, lack of release notes, and
already-scheduled back-branch updates that will be consuming some of
core's time this week.  And I doubt anyone thinks we need less than 2
months in beta.

(Or did you mean April of 2010?)

> While personally I want us to switch to a mode where we commit large patches
> early in the development cycle I don't believe we would refuse to commit it
> today if it was ready. And I can't imagine two weeks would make the
> difference either.

Well, if it were ready, we'd not be having this discussion.  The problem
is that it's not ready and no one is very sure about when it will be.
        regards, tom lane

On Mon, 2009-01-26 at 20:12 +0000, Gregory Stark wrote:
> Merlin Moncure <mmoncure@gmail.com> writes:
> 
> > HS is working very well (Simon's ongoing work aside).  I am pretty
> > confident based on my personal testing that it would represent the
> > project well if committed today.
> 
> I think a lot of people weren't aware there was anybody testing this patch
> other than Simon and Heikki -- I wasn't until just today. I wonder how many
> more people are trying it out?

OK, so maybe its been unclear: Gianni Ciolli has been leading a test
team on this, with other people swapping in and out as they go to work
on other things. There's been multiple test machines running various
kinds of test on different OS. Gianni and I have done many midnight
sessions together on this and his help and support has been invaluable.
If some bugs have got passed that test screening in recent weeks, I
think that's down to the sheer volume of tests and my own descriptions
of how things ought to work and exactly where to focus. Gianni has
located more issues than Heikki has, just.

Hannu provided a lot of early input on the problems of conflict
resolution and various alternatives. 

I'm the main developer, but I don't underrate their contributions.

I've had specific bug reports from 6 people and test feedback/questions
from about another 4-5 people; Mark isn't credited because I wasn't
keeping track of them in earl stages of review.

Recent refactoring has been a rich source of regressions and those don't
go away overnight, but they don't take months either.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
>> So, some feedback to make this decision more difficult:
> 
>> Users: care about HS more than anything else in the world.
> 
> I don't think this is correct.  There are certainly a lot of users who
> would like an in-core replication solution, but HS by itself is not that
> --- you also need (near) real-time log shipping, which we have already
> decided to punt to 8.5.  That being the case, I think the argument
> that HS is a must-have feature for 8.4 is actually rather weak.
> 
>> SE-Linux:  this patch has effectively been in development for 2 years 
>> ourside the core process before putting it in; the forked SEPostgres is 
>> in use in production. KaiGai has been available for 20 hours a week (or 
>> more) to troubleshoot issues and change APIs.  I really don't see what 
>> the problem is with committing it.
> 
> The problem, in words of one syllable, is that we are not sure we want
> it.  Do you see a user community clamoring for SEPostgres, or a hacker
> community that is willing or able to maintain it?  If KaiGai-san got run
> over by a bus tomorrow, this patch would be a dead letter, because there
> just isn't anyone else who is taking sufficient (any?) interest in it.
> That doesn't bode well for its future viability.  Compare the likely
> audience for it to previous patches of roughly similar complexity,
> such as integrated text search or the Windows port, and it's just not
> in the ballpark.
> 
> The second problem is that we're not sure it's really the right thing,
> because we have no one who is competent to review the design from a
> security standpoint.  But unless we get past the first problem the
> second one is moot.
> 


I've never posted to this list before, but I am an SELinux upstream maintainer.

I'd just like to interject here, we (the SELinux community) are very interested 
in KaiGai's work and have been looking forward to it being upstreamed for quite 
some time.

While we haven't been able to analyze the patches directly to determine whether 
the security goals are indeed being met we have had much discussion and 
eventually community agreement on the security model being implemented. This 
happened years ago and has since been merged into the SELinux reference policy 
that practically all SELinux users use (distributions start with the reference 
policy and add rules/modules suitable for them).

So the security model has been looked at, though not the implementation and we 
do have a community of developers, users and customers interested in this work.

Joshua Brindle

On Mon, 2009-01-26 at 15:47 -0500, Merlin Moncure wrote:

> I'd also like to see Simon and or
> Heikki make a strong statement on where things stand _right now_ (not
> in two weeks) :-)

Well, we just found 2 bugs over the weekend, one of which is a
regression from refactoring. 

The second bug is being fixed as part of requested refactoring from
Heikki, discussed today.

I've insisted on full visibility, so all known bugs have been recorded
on Wiki as soon as they are reported, even internally located ones. Most
have been fixed on same day found. Check out the Wiki, any day.

I still have not completed Prepared Transaction support, but if I had
then it just would have slowed down the earlier rework. This is probably
the only hanging offence. Please remember that Heikki's rework proposals
were still under discussion only 2 weeks ago.

Personally, I've very happy with the code as stands, generally, but
Heikki may wish to move a few things around yet. The rework has meant
I've reread most of my own code, so I have found and fixed many of my
own bugs and rewritten comments to better explain things. I found,
reported and fixed the issue of drop tablespace for example. If cynical,
I could have hidden that and waited a month for people to find it.

Going Beta will reveal further feedback on usability, so we might expect
a few tweaks there as people moan. We'll see.

I won't put a time limit on this since people will just add their own
fudge factors to that and we'll be no nearer to a true figure. Without
everybody breathing down my neck it might be 2-3 days, but I'm spending
time reading email and trying to keep my patches alive.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

On Mon, 2009-01-26 at 15:49 -0500, Tom Lane wrote:

> The problem is that it's not ready and no one is very sure about when
> it will be.

With respect, I've done more than any other developer has done to give
you and the community full information on the patch as it develops. I'm
sorry you don't believe what I tell you, but if you don't you should see
the patch for yourself or ask the person reviewing it.

The Wiki has been there for 4 months now, updated daily for last 8 weeks
apart from illness.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

Josh Berkus wrote:
> All,
>
> So, some feedback to make this decision more difficult:
>
> Users: care about HS more than anything else in the world.  I'm 
> convinced that if we took a staw poll, 80% of our users would be in 
> favor of waiting for HS.  This one feature will make more of a 
> difference in the number of PG users than any feature since the 
> Windows port.  Maybe more.
>
> on the other hand:
>
> We held back version 4 months 7.4 for Windows, before it became 
> apparent that there was at least a year more work to do.  That was a 
> mistake, and in many ways HS seems like a similar case.
>

I can only confirm what Josh is saying here.
We would also assume that 80% have been waiting for Simon's work for 
years. In fact, I have been dealing fulltime with PostgreSQL since 1999 
and it has been a missing issue since than.
Now that we are so close to fixing this issue for so many people out 
there, we should give it all the attention we have and support Simon + 
team wherever we can.
I think Simon has responded to all question is almost realtime. We 
should take that into consideration.
Also, Simon is focuing on a very open development model - this naturally 
means a lot of mailing list traffic. Isn't this what this project is all 
about?

I am in favor of giving this patch a useful timeframe for completion.
If people decide to give this patch a chance, we will definitely agree 
on putting some significant manpower in here as well.
We are not the only ones who want to see that in.
We already see people saying that they delay migrations because they are 
hoping for readable slaves to go in.
Also, in the past 10 years I have been tortured with "when can we have 
replication" each and every day ...
I am fed up :). i cannot hear it anymore ("... but MySQL has 
replication" *aaaaaaaaaaaaaaaaaaaaaaaaaargh*).
   best regards,
      hans

-- 
--
Cybertec Schönig & Schönig GmbH
Professional PostgreSQL Consulting, Support, Training
Gröhrmühlgasse 26, A-2700 Wiener Neustadt
Web: www.postgresql-support.de

Joshua,

> So the security model has been looked at, though not the implementation 
> and we do have a community of developers, users and customers interested 
> in this work.

Can you please take a look at it ASAP, then?  In the next week, we will 
probably decide on whether or not to defer SEPostgres until 8.5.  The 
fact that we haven't gotten a sign-off from any security expert anywhere 
is leaning the whole community towards "defer".

--Josh

On Monday 26 January 2009 2:12:02 pm Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
> > So, some feedback to make this decision more difficult:
> >
> > Users: care about HS more than anything else in the world.
>
> I don't think this is correct.  There are certainly a lot of users who
> would like an in-core replication solution, but HS by itself is not that
> --- you also need (near) real-time log shipping, which we have already
> decided to punt to 8.5.
Precisely.  Thank you.

> That being the case, I think the argument that HS is a must-have feature for 
8.4 is actually rather weak.
I am just one person, representing one company...but I do agree, fwiw.

On 1/26/09 4:28 PM, "Joshua Brindle" <method@manicmethod.com> wrote:

> Tom Lane wrote:
>> Josh Berkus <josh@agliodbs.com> writes:
<snip>
>>> SE-Linux:  this patch has effectively been in development for 2 years
>>> ourside the core process before putting it in; the forked SEPostgres is
>>> in use in production. KaiGai has been available for 20 hours a week (or
>>> more) to troubleshoot issues and change APIs.  I really don't see what
>>> the problem is with committing it.
>>
>> The problem, in words of one syllable, is that we are not sure we want
>> it.  Do you see a user community clamoring for SEPostgres, or a hacker
>> community that is willing or able to maintain it?  If KaiGai-san got run
>> over by a bus tomorrow, this patch would be a dead letter, because there
>> just isn't anyone else who is taking sufficient (any?) interest in it.
>> That doesn't bode well for its future viability.  Compare the likely
>> audience for it to previous patches of roughly similar complexity,
>> such as integrated text search or the Windows port, and it's just not
>> in the ballpark.
>>
>> The second problem is that we're not sure it's really the right thing,
>> because we have no one who is competent to review the design from a
>> security standpoint.  But unless we get past the first problem the
>> second one is moot.
>>
>
>
> I've never posted to this list before, but I am an SELinux upstream
> maintainer.
>
> I'd just like to interject here, we (the SELinux community) are very
> interested
> in KaiGai's work and have been looking forward to it being upstreamed for
> quite
> some time.
>
> While we haven't been able to analyze the patches directly to determine
> whether
> the security goals are indeed being met we have had much discussion and
> eventually community agreement on the security model being implemented. This
> happened years ago and has since been merged into the SELinux reference policy
> that practically all SELinux users use (distributions start with the reference
> policy and add rules/modules suitable for them).
>
> So the security model has been looked at, though not the implementation and we
> do have a community of developers, users and customers interested in this
> work.
>
I'd just like to echo Josh's sentiments. I'm also active in the SELinux
community, and have been involved in several developments that really needed
a database with mandatory access control mechanisms. Unfortunately, these
developments have all had maintenance requirements that precluded using
KaiGai's code as it was outside not in a commercial distribution. We've been
waiting anxiously for it to be merged upstream.

Additionally, I've talked to many other end users that really want to deploy
LAPP stacks with these security features. They often came to us looking for
us to help them build such systems, but we've had to turn them away as there
was no supported way to build it.

Thanks,
Chad Sellers

Gregory Stark wrote:
> 
> I think a lot of people weren't aware there was anybody testing this patch
> other than Simon and Heikki -- I wasn't until just today. I wonder how many
> more people are trying it out?

I've been running the patch (I think since Jan 5) on a couple dev instances
that were used primarily to test if our software worked with what we expected
to turn into 8.4.   I can't say I've really been testing the patch specifically,
but rather testing my workplace's software against a version with this patch,
and hadn't noticed any problems.

Josh Berkus wrote:
> Joshua,
> 
>> So the security model has been looked at, though not the 
>> implementation and we do have a community of developers, users and 
>> customers interested in this work.
> 
> Can you please take a look at it ASAP, then?  In the next week, we will 
> probably decide on whether or not to defer SEPostgres until 8.5.  The 
> fact that we haven't gotten a sign-off from any security expert anywhere 
> is leaning the whole community towards "defer".
> 

Yes, I will look at them to the extent I am able. As I am not familiar with the 
postgresql codebase I won't be able to assert the correctness of the hook 
placement (that is, where the security functions are called with respect to the 
data they are protecting being accessed). The postgresql community should be 
more familiar with the hook call sites and hopefully can assist there.

I should be able to handle the security backend and determining whether it 
matches the security model we agreed on, but the hook placement is just as 
important since a misplaced or missing hook will allow access that should not be 
granted.

Joshua Brindle

Gregory Stark wrote:
> I think a lot of people weren't aware there was anybody testing this patch
> ...I wonder how many more people are trying it out?

I think I have an idea to improve this aspect for future commit fests.

For a long time at each of my workplaces I've been running a development
instance against CVS-HEAD just to make sure our software is more future-proof
against up-and-coming releases.   We run this system with -enable-debug, asserts,
etc, and accept that it's just a development system not expected to be totally
stable.

If it were just as easy for us to pull from a "all 'pending-patches' for-commit-fest-nov that pass regression tests"
branch, I'd happily pull from that instead.

I realize in the current system (emailed patches), this would be a horrible
pain to maintain such a branch; but perhaps some of the burden could be
pushed down to the patch submitters (asking them to merge their own changes
into this merged branch).   And I hate bringing up the version control
flame war again; but git really would make this easier.   If all patches
were on their own branches; the painful merges into this shared branch
would be rare, as the source control system would remember the painful
parts of the merges.

Ron Mayer <rm_pg@cheapcomplexdevices.com> writes:

> I realize in the current system (emailed patches), this would be a horrible
> pain to maintain such a branch; but perhaps some of the burden could be
> pushed down to the patch submitters (asking them to merge their own changes
> into this merged branch).   

I've considered maintaining such a repository a few times and dismissed it
when I realized how much work it would be to maintain.

> And I hate bringing up the version control flame war again; but git really
> would make this easier. If all patches were on their own branches; the
> painful merges into this shared branch would be rare, as the source control
> system would remember the painful parts of the merges.

We have git repositories, I still think maintaining a merged tree with dozens
of patches would be a lot of work.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production
Tuning

Joshua Brindle <method@manicmethod.com> writes:

> Yes, I will look at them to the extent I am able. As I am not familiar with the
> postgresql codebase I won't be able to assert the correctness of the hook
> placement (that is, where the security functions are called with respect to the
> data they are protecting being accessed). The postgresql community should be
> more familiar with the hook call sites and hopefully can assist there.

I would suggest looking at the documentation (which I assume the patch does
update). If it's not clear that things are in order from the documentation
then either it needs better documentation or something's wrong...

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support!

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Josh Berkus <josh@agliodbs.com> writes:
> > Users: care about HS more than anything else in the world.
>
> I don't think this is correct.  There are certainly a lot of users who
> would like an in-core replication solution, but HS by itself is not that
> --- you also need (near) real-time log shipping, which we have already
> decided to punt to 8.5.  That being the case, I think the argument
> that HS is a must-have feature for 8.4 is actually rather weak.

HS would still be really nice, and I'm a bit concerned that we'll end up
in 8.5 with a similar discussion along the lines of "well, we've only
just committed HS, why don't we release 8.5 with that and wait on
replication till 8.6".  I agree that more folks are after replication
than HS, but there is still a user base for it.

> > SE-Linux:  this patch has effectively been in development for 2 years
> > ourside the core process before putting it in; the forked SEPostgres is
> > in use in production. KaiGai has been available for 20 hours a week (or
> > more) to troubleshoot issues and change APIs.  I really don't see what
> > the problem is with committing it.
>
> The problem, in words of one syllable, is that we are not sure we want
> it.  Do you see a user community clamoring for SEPostgres, or a hacker
> community that is willing or able to maintain it?  If KaiGai-san got run
> over by a bus tomorrow, this patch would be a dead letter, because there
> just isn't anyone else who is taking sufficient (any?) interest in it.

I'm certainly interested in it and would like to see it in core.  Would
I use it immediately?  No, because I've so far avoided having to have it
for my systems.  I don't expect that to last forever though, at some
point I'll have to implement a system which requires the seperation
provided by SELinux or move to something like Solaris Trusted Extensions
and Oracle Label Security.

> That doesn't bode well for its future viability.  Compare the likely
> audience for it to previous patches of roughly similar complexity,
> such as integrated text search or the Windows port, and it's just not
> in the ballpark.

No, it doesn't have as large a user base as the Windows port or
integrated text search.  On the other hand, there *are* users out there,
and hackers, who are willing and interested in it for PostgreSQL because
it would give them an alternative to the de-facto standards.  Perhaps
it's just me, but historically it seems like there's also been a fair
bit of aid given to trusted/SE implementations by various organizations
who need it, both in terms of funding for others to work on it and in
actual direct development.  I realize this is a trivially defeated POV,
but I really feel that 'if you build it, they will come' in this case.

> The second problem is that we're not sure it's really the right thing,
> because we have no one who is competent to review the design from a
> security standpoint.  But unless we get past the first problem the
> second one is moot.

I think the database design for SELinux is pretty well defined..  The
implementation needs to be reviewed, but I think there's few who can
do that, unfortunately.
Thanks,
    Stephen

Stephen Frost <sfrost@snowman.net> writes:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> The problem, in words of one syllable, is that we are not sure we want
>> it.  Do you see a user community clamoring for SEPostgres, or a hacker
>> community that is willing or able to maintain it?

> No, it doesn't have as large a user base as the Windows port or
> integrated text search.  On the other hand, there *are* users out there,
> and hackers, who are willing and interested in it for PostgreSQL because
> it would give them an alternative to the de-facto standards.

Then why has *nobody* stepped up to review the design, much less the
whole patch?  The plain truth is that no one appears to care enough to
expend any real effort.  But this patch is far too large and invasive
to accept on the basis that only one guy understands it and will/might
continue to maintain it.

I'll risk being rude to make my point: those who want SEPostgres in core
need to put up or shut up.  Now, not at some future time.  We need
people to sign off that this patch implements the features they want
(not "sounds roughly like some vague future need I might have") and does
so correctly.  An incorrect security feature is considerably worse than
useless.  And once it's in core we aren't going to have a whole lot of
elbow room to change the definition later.
        regards, tom lane

Tom Lane wrote:
> The problem, in words of one syllable, is that we are not sure we want
> it.  Do you see a user community clamoring for SEPostgres, or a hacker

This is a chicken-and-egg type of problem.

Security-conscious users, applications, hackers, and customers will
flock towards whichever database product leads in that area.

If some hypothetical database has only minimal security features, I
imagine few security experts would spend a lot of time with the database.

> The second problem is that we're not sure it's really the right thing,
> because we have no one who is competent to review the design from a
> security standpoint.  But unless we get past the first problem the
> second one is moot.

Are we underestimating Kaigai Kohei?  I seem to see him credited on
the NSA's SELinux pages:  http://www.nsa.gov/research/selinux/contrib.shtml

and it seems his patches there related to postgresql were pretty widely
discussed on the SELinux lists: http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163

On Monday 26 January 2009 6:31:48 pm Ron Mayer wrote:
> Tom Lane wrote:
[...snip...]
> > The second problem is that we're not sure it's really the right thing,
> > because we have no one who is competent to review the design from a
> > security standpoint.  But unless we get past the first problem the
> > second one is moot.
>
> Are we underestimating Kaigai Kohei?  I seem to see him credited on
> the NSA's SELinux pages:
>    http://www.nsa.gov/research/selinux/contrib.shtml
>
> and it seems his patches there related to postgresql were pretty widely
> discussed on the SELinux lists:
>   http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163

I'm pretty sure that when he says "review" that it is implied to be a "peer 
review"

On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote:

> Then why has *nobody* stepped up to review the design, much less the
> whole patch?  The plain truth is that no one appears to care enough to
> expend any real effort.  But this patch is far too large and invasive
> to accept on the basis that only one guy understands it and will/might
> continue to maintain it.

It was only today that I saw an announcement go out to our announce list
to try and get people to pop their heads up and help. It looks like
today is the day that someone actually reached out to the SELinux folks
and asked for help. We really aren't very good at reaching out for help.
We have a tendency to stick to our little pods, in this case -hackers.

> 
> I'll risk being rude to make my point: those who want SEPostgres in core
> need to put up or shut up.  Now, not at some future time.  We need
> people to sign off that this patch implements the features they want
> (not "sounds roughly like some vague future need I might have") and does
> so correctly.  An incorrect security feature is considerably worse than
> useless.  And once it's in core we aren't going to have a whole lot of
> elbow room to change the definition later.

I think this is a fair statement to make. I would also note that at
least one other SELinux person has offered today to review at least the
SE portions of this. I quote:

"
Yes, I will look at them to the extent I am able. As I am not familiar
with the postgresql codebase I won't be able to assert the correctness
of the hook placement (that is, where the security functions are called
with respect to the data they are protecting being accessed). The
postgresql community should be more familiar with the hook call sites
and hopefully can assist there.

I should be able to handle the security backend and determining whether
it matches the security model we agreed on, but the hook placement is
just as important since a misplaced or missing hook will allow access
that should not be granted.

Joshua Brindle
"

I do think its important to recognize that SEPostgres is a specialty
feature. Like full disjunctions was, it is a feature that very, very few
will use but those that do really do want and will very much make use of
it.

If I put on my pragmatic helmet it is a high cost, low benefit feature
for the vast majority of our community.

If I put on my advocacy hat it is a, "Hah take that you little blue
mammal!" and frankly "There is no big O in this equation... if you want
this kind of power you gotta ride the elephant!".

There is a strategic element though. Adding features for the too smart
for their own good lures the too smart for their own good to use (and
enhance) a product. This could help us in the long run.

Sincerely,

Joshua D. Drake


-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

Ron Mayer <rm_pg@cheapcomplexdevices.com> writes:
> Tom Lane wrote:
>> The second problem is that we're not sure it's really the right thing,
>> because we have no one who is competent to review the design from a
>> security standpoint.

> Are we underestimating Kaigai Kohei?

Perhaps he walks on water, but still I'd like to have more than one
person who has confidence that this design and implementation are correct.

> and it seems his patches there related to postgresql were pretty widely
> discussed on the SELinux lists:
>   http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163

Well, a quick look through that thread shows a lot of discussion of the
selinux policy code that's in the patch, which is good as far as it goes
because for sure there's no one in *this* list who understands a line of
that stuff.  But to be blunt there's no evidence there that anyone in
that discussion has heard of a foreign key, much less understands why
it might be an issue for this patch.  I see a lot of reasoning by
analogy to X servers, and little if any database-specific knowledge.

Mind you, I'd like nothing better than to have some NSA database
security experts (I'm sure there are some) show up here and tell us that
this design is good, secure, and useful --- and why.  But right now we
have no evidence for that proposition.  And we really need to understand
*why* it's a useful design and what the critical security issues are,
because otherwise we are 100% certain to break it in future maintenance
(even granting the improbable supposition that there are no bugs in the
patch today).
        regards, tom lane

Tom Lane wrote:
> Ron Mayer <rm_pg@cheapcomplexdevices.com> writes:
>> Are we underestimating Kaigai Kohei?
> Perhaps he walks on water, but still I'd like to have more than one
> person who has confidence that this design and implementation are correct.

Totally fair.  I know I'm totally unqualified to review his buoyancy on
water, though.

>> and it seems his patches there related to postgresql were pretty widely
>> discussed on the SELinux lists:
>>   http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163
> 
> Well, a quick look through that thread shows a lot of discussion of the
> selinux policy code that's in the patch, which is good as far as it goes
> because for sure there's no one in *this* list who understands a line of
> that stuff. 

Totally fair too.

> Mind you, I'd like nothing better than to have some NSA database
> security experts (I'm sure there are some) show up here and tell us that
> this design is good, secure, and useful --- and why.  But right now we

What's the right way for us to ask them?  No doubt there are some,
but how do we expect them to find & join our email list?  If we
wanted more feedback would it make sense for someone who can speak
for the project to call them and ask if they'd be interested
in getting involved?

* Ron Mayer (rm_pg@cheapcomplexdevices.com) wrote:
> What's the right way for us to ask them?  No doubt there are some,
> but how do we expect them to find & join our email list?  If we
> wanted more feedback would it make sense for someone who can speak
> for the project to call them and ask if they'd be interested
> in getting involved?

Sorry if it wasn't the 'right way', but I've forwarded Bruce's post to
-announce (with some additional links to this discussion) to the mailing
list up thread, to which I've also subscribed.
Thanks,
    Stephen

"Joshua D. Drake" <jd@commandprompt.com> writes:
> On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote:
>> Then why has *nobody* stepped up to review the design, much less the
>> whole patch?

> It was only today that I saw an announcement go out to our announce list
> to try and get people to pop their heads up and help. It looks like
> today is the day that someone actually reached out to the SELinux folks
> and asked for help.

Hmm, you think selinux people read pgsql-announce?  But seriously,
we *have* been trying to get people's attention for this patch, both
inside and outside the postgres community, for well over a year now.
The lack of response has been depressing and (IMHO) telling.  Nowhere
have we gotten anything more concrete than "ooh, that's cool ... maybe
I might use it someday, but I can't be bothered right now".
        regards, tom lane

On Mon, 2009-01-26 at 20:27 -0500, Stephen Frost wrote:
> * Ron Mayer (rm_pg@cheapcomplexdevices.com) wrote:
> > What's the right way for us to ask them?  No doubt there are some,
> > but how do we expect them to find & join our email list?  If we
> > wanted more feedback would it make sense for someone who can speak
> > for the project to call them and ask if they'd be interested
> > in getting involved?
> 
> Sorry if it wasn't the 'right way', but I've forwarded Bruce's post to
> -announce (with some additional links to this discussion) to the mailing
> list up thread, to which I've also subscribed.

As have I. Perhaps some humble requests for help that are directed at
the pin point will help.

Joshua D. Drake


> 
>     Thanks,
> 
>         Stephen
-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

Tom Lane wrote:
> Hmm, you think selinux people read pgsql-announce?  But seriously,
> we *have* been trying to get people's attention for this patch, both
> inside and outside the postgres community, for well over a year now.
> The lack of response has been depressing and (IMHO) telling.  Nowhere
> have we gotten anything more concrete than "ooh, that's cool ... maybe
> I might use it someday, but I can't be bothered right now".

Ah!  Then yes, that does say something about the lack of interest.
It wasn't obvious to me that people were reaching out beyond these lists.

* Ron Mayer (rm_pg@cheapcomplexdevices.com) wrote:
> Tom Lane wrote:
> > Hmm, you think selinux people read pgsql-announce?  But seriously,
> > we *have* been trying to get people's attention for this patch, both
> > inside and outside the postgres community, for well over a year now.
> > The lack of response has been depressing and (IMHO) telling.  Nowhere
> > have we gotten anything more concrete than "ooh, that's cool ... maybe
> > I might use it someday, but I can't be bothered right now".
>
> Ah!  Then yes, that does say something about the lack of interest.
> It wasn't obvious to me that people were reaching out beyond these lists.

Where were we reaching outside the postgres community..?  Just curious
what other SELinux or related lists are out there.  As mentioned up
thread, there was interest for a couple of months this past summer on
the NSA SELinux mailing list.
Thanks,
    Stephen

On Mon, 2009-01-26 at 20:28 -0500, Tom Lane wrote:
> "Joshua D. Drake" <jd@commandprompt.com> writes:
> > On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote:
> >> Then why has *nobody* stepped up to review the design, much less the
> >> whole patch?
> 
> > It was only today that I saw an announcement go out to our announce list
> > to try and get people to pop their heads up and help. It looks like
> > today is the day that someone actually reached out to the SELinux folks
> > and asked for help.
> 
> Hmm, you think selinux people read pgsql-announce? 

*shrug* they very well might if they are also postgres people.

>  But seriously,
> we *have* been trying to get people's attention for this patch, both
> inside and outside the postgres community, for well over a year now.
> The lack of response has been depressing and (IMHO) telling.  Nowhere
> have we gotten anything more concrete than "ooh, that's cool ... maybe
> I might use it someday, but I can't be bothered right now".

Well that is certainly unfortunate. If the very community that invented
the tech isn't willing to help.... :(

Joshua D. Drake

> 
>             regards, tom lane
> 
-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

Tom Lane wrote:
> Ron Mayer <rm_pg@cheapcomplexdevices.com> writes:
>> Tom Lane wrote:
>>> The second problem is that we're not sure it's really the right thing,
>>> because we have no one who is competent to review the design from a
>>> security standpoint.
> 
>> Are we underestimating Kaigai Kohei?
> 
> Perhaps he walks on water, but still I'd like to have more than one
> person who has confidence that this design and implementation are correct.
> 
>> and it seems his patches there related to postgresql were pretty widely
>> discussed on the SELinux lists:
>>   http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163
> 
> Well, a quick look through that thread shows a lot of discussion of the
> selinux policy code that's in the patch, which is good as far as it goes
> because for sure there's no one in *this* list who understands a line of
> that stuff.  But to be blunt there's no evidence there that anyone in
> that discussion has heard of a foreign key, much less understands why
> it might be an issue for this patch.  I see a lot of reasoning by
> analogy to X servers, and little if any database-specific knowledge.
> 

http://marc.info/?l=selinux&m=115762285013528&w=2

Is the original discussion thread for the security model used in the 
sepostgresql work. Hopefully you'll see some of the evidence you speak of there.

For some additional resources there is a good chapter on database security 
(specifically multilevel databases) in Pfleeger's Security in Computing:
http://www.amazon.com/Security-Computing-Second-Charles-Pfleeger/dp/0133374866/ref=si3_rdr_bb_product


> Mind you, I'd like nothing better than to have some NSA database
> security experts (I'm sure there are some) show up here and tell us that
> this design is good, secure, and useful --- and why.  But right now we
> have no evidence for that proposition.  And we really need to understand
> *why* it's a useful design and what the critical security issues are,
> because otherwise we are 100% certain to break it in future maintenance
> (even granting the improbable supposition that there are no bugs in the
> patch today).
> 

This capability puts PostgreSQL in a unique position. Not only is the security 
model more fine grained than any previous trusted database but it also allows 
interesting things like trusted stored procedures that can access data (and do 
any necessary fuzzing, modifying or filtering) that the client himself cannot read.

It also integrates well with the labeled networking supplied by SELinux, as 
outlined on KaiGai's wiki page. There can be multiple clients running on one 
machine (such as a web server) that have different levels of database access, in 
a more fine grained and flexible way than using different database credentials 
allows.

I'm happy to discuss this at length if it will help the patches get upstreamed.

Stephen Frost <sfrost@snowman.net> writes:
> * Ron Mayer (rm_pg@cheapcomplexdevices.com) wrote:
>> Ah!  Then yes, that does say something about the lack of interest.
>> It wasn't obvious to me that people were reaching out beyond these lists.

> Where were we reaching outside the postgres community..?

Well, I've been trying to get Red Hat to interest their NSA contacts in
it, and Bruce and Josh B have been making efforts via EDB and Sun that
I don't have details about, but nothing much has come of any of that.
Maybe waving a red flag in front of the selinux list will have some
effect.
        regards, tom lane

Tom Lane wrote:
> Stephen Frost <sfrost@snowman.net> writes:
>> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>>> The problem, in words of one syllable, is that we are not sure we want
>>> it.  Do you see a user community clamoring for SEPostgres, or a hacker
>>> community that is willing or able to maintain it?
> 
>> No, it doesn't have as large a user base as the Windows port or
>> integrated text search.  On the other hand, there *are* users out there,
>> and hackers, who are willing and interested in it for PostgreSQL because
>> it would give them an alternative to the de-facto standards.
> 
> Then why has *nobody* stepped up to review the design, much less the
> whole patch?  The plain truth is that no one appears to care enough to
> expend any real effort.  But this patch is far too large and invasive
> to accept on the basis that only one guy understands it and will/might
> continue to maintain it.

The matter we're currently faced can be called as like a disconnection
between OSS communities.
At least, as several folks introduced in this thread, security focused
people are strongly waiting for SE-PostgreSQL feature upstreamed.
However, we have a wall to be overed, if they join to review the patches,
because most of security experts are not database experts (familiar to
its internal architectures).

In addition, I have hesitated to involve security experts due to the
discussion will need deep knowledge about its internal architectures.
But I think Bruce's suggestion is whorthwhile. At least, it is a case
we need cross-community discussion.

> I'll risk being rude to make my point: those who want SEPostgres in core
> need to put up or shut up.  Now, not at some future time.  We need
> people to sign off that this patch implements the features they want
> (not "sounds roughly like some vague future need I might have") and does
> so correctly.  An incorrect security feature is considerably worse than
> useless.  And once it's in core we aren't going to have a whole lot of
> elbow room to change the definition later.

At least, the security design of SE-PostgreSQL has been accepted for
two years in SELinux community. An evidence is its upstreamed security
policy (reference policy) contains rules for SE-PostgreSQL.
 http://oss.tresys.com/repos/refpolicy/trunk/policy/modules/services/postgresql.te

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Jaime Casanova wrote:
>> SE-Linux:  this patch has effectively been in development for 2 years
>> ourside the core process before putting it in; the forked SEPostgres is in
>> use in production. KaiGai has been available for 20 hours a week (or more)
>> to troubleshoot issues and change APIs.  I really don't see what the problem
>> is with committing it.
>>
> 
> it hasn't been testing by ours in different platforms (ie: ubuntu has
> selinux and i want to give it a try, badly enough i have never used
> selinux so this is new to me)...
> 
> nor we have any evidence that it doesn't affect to users that doesn't
> have any variant of selinux (ie: windows)... the real problem here is
> the base of users we enough knowledge of the tool to make some usefull
> tests

It is obvious, if you see the code.
When SELinux is disabled by platform or GUC option, security hooks
works nothing.

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Well, I've been trying to get Red Hat to interest their NSA contacts in
> it, and Bruce and Josh B have been making efforts via EDB and Sun that
> I don't have details about, but nothing much has come of any of that.

It's certainly frustrating to hear that Red Hat, who have been pretty
big proponents of SELinux, not showing interest in having it supported
in PostgreSQL.. :/

> Maybe waving a red flag in front of the selinux list will have some
> effect.

I certainly hope so.  I'm planning to review the patch and documentation
and set up an environment and run it through its paces this week.
Thanks,
    Stephen

Joshua Brindle wrote:
> While we haven't been able to analyze the patches directly to
> determine whether the security goals are indeed being met we
> have had much discussion and eventually community agreement on
> the security model being implemented. This happened years ago
> and has since been merged into the SELinux reference policy that
> practically all SELinux users use (distributions start with the
> reference policy and add rules/modules suitable for them).
> 
> So the security model has been looked at, though not the
> implementation and we do have a community of developers, users
> and customers interested in this work.

That is good to know.

-- Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Tom Lane wrote:
> Ron Mayer <rm_pg@cheapcomplexdevices.com> writes:
>> Tom Lane wrote:
>>> The second problem is that we're not sure it's really the right thing,
>>> because we have no one who is competent to review the design from a
>>> security standpoint.
> 
>> Are we underestimating Kaigai Kohei?
> 
> Perhaps he walks on water, but still I'd like to have more than one
> person who has confidence that this design and implementation are correct.
> 
>> and it seems his patches there related to postgresql were pretty widely
>> discussed on the SELinux lists:
>>   http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163
> 
> Well, a quick look through that thread shows a lot of discussion of the
> selinux policy code that's in the patch, which is good as far as it goes
> because for sure there's no one in *this* list who understands a line of
> that stuff.  But to be blunt there's no evidence there that anyone in
> that discussion has heard of a foreign key, much less understands why
> it might be an issue for this patch.  I see a lot of reasoning by
> analogy to X servers, and little if any database-specific knowledge.

I believe they understand the issues related to covert channels (via PK/FK)
and polyinstantiation, though it is not talked on the thread.
(NOTE: The origin of poluinstantiation is previous security research!)
Yes, there is indeed no evidence. If so, it is a good idea to ask
their opinion with SELinux folks (expect for me?).

As I noted before several times, there are several classes in security
requirements. Historically, TSCEC (Trusted Computer System Evaluation
Criteria, Orange-Book) is a representative evaluation criteria for
security features in IT producets. Now it is inherited to ISO/IEC15408
called as CC (Common Criteria).
We can also consider CC as a set of security functional requirements,
and it defined various kind of requirements.

An interaction between foreign keys and invisible rows is a kind of
covert channel issue. Indeed, it has a possibility users to infer
existance of invisible tuples.
In ISO/IEC15408, FDP_IFF (Information Flow Function, Functionality
of Data Protection) section describes about related requirements.
It defines various classes of requirements. One highest stuff requires
to eliminate all the information-flows via covert-channel, but others
does not require to eliminate at all, or does not mention about it.
Here, it is important what requirement are choosen depends on a
sort of evaluated product, environment, estimated-threat and so on.

NOTE: Oracle Label Security does not care about covert-channels,     because it may also compound for FK/PK issues and
row-level    security.     http://www.commoncriteriaportal.org/files/epfiles/20080306_0402b.pdf
 

In the previous discussion, we (including me) decided that SE-PostgreSQL
also does not care about the FK/PK issues, because polyinstantiation feature
needs unacceptable widespread reworks on PostgreSQL, so I added an explicit
text in the documentation to notice its restriction for end-users.

At least, earlier commercial database (Oracle) does not care, and it can
follow ISO/IEC15408 manner, so I think it is fair enough approach.
It is a summary of previous discussion.

Joshua, Chad, please comment anything, if my understanding was incorrect.

BTW, I have not walked on water yet.

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

On Tue, 2009-01-27 at 12:30 +0900, KaiGai Kohei wrote:

> 
> BTW, I have not walked on water yet.
> 

I have but I always end up wet. :)

Joshua D. Drake

> Thanks,
-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

> -----Original Message-----
> From: pgsql-hackers-owner@postgresql.org [mailto:pgsql-hackers-
> owner@postgresql.org] On Behalf Of Joshua D. Drake
> Sent: Monday, January 26, 2009 7:42 PM
> To: KaiGai Kohei
> Cc: Tom Lane; Ron Mayer; Josh Berkus; Robert Haas; Merlin Moncure;
> Jonah H. Harris; Gregory Stark; Simon Riggs; Bruce Momjian; Bernd
> Helmle; Peter Eisentraut; pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] 8.4 release planning
>
> On Tue, 2009-01-27 at 12:30 +0900, KaiGai Kohei wrote:
>
> >
> > BTW, I have not walked on water yet.
> >
>
> I have but I always end up wet. :)

I find that it helps to freeze the water first.
;-)

Dann Corbit wrote:
>> -----Original Message-----
>> From: pgsql-hackers-owner@postgresql.org [mailto:pgsql-hackers-
>> owner@postgresql.org] On Behalf Of Joshua D. Drake
>> Sent: Monday, January 26, 2009 7:42 PM
>> To: KaiGai Kohei
>> Cc: Tom Lane; Ron Mayer; Josh Berkus; Robert Haas; Merlin Moncure;
>> Jonah H. Harris; Gregory Stark; Simon Riggs; Bruce Momjian; Bernd
>> Helmle; Peter Eisentraut; pgsql-hackers@postgresql.org
>> Subject: Re: [HACKERS] 8.4 release planning
>>
>> On Tue, 2009-01-27 at 12:30 +0900, KaiGai Kohei wrote:
>>
>>> BTW, I have not walked on water yet.
>>>
>> I have but I always end up wet. :)
> 
> I find that it helps to freeze the water first.
> ;-)

I'm sorry for the improper humor.
Shall we return to the discussion?
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Joshua Brindle <method@manicmethod.com> writes:
> http://marc.info/?l=selinux&m=115762285013528&w=2
> Is the original discussion thread for the security model used in the 
> sepostgresql work. Hopefully you'll see some of the evidence you speak of there.

Thanks for the link.  I took a look through that thread and saw a lot of
discussion about issues like how to relate the database-side and
client-side permissions, which is all good stuff but mostly outside my
purview as a database geek.  I didn't find anything about the stuff that
is really bothering me, which I think can be broken down into two main
categories:

1. Silently filtering out rows according to an arbitrary security policy
can break a bunch of fundamental SQL semantics, the most obvious being
foreign key constraints --- an application might be able to see a
dependent row that apparently has no referenced row, or might get an
update or delete failure for a row that is unreferenced as far as it can
see.  Things get worse if an application can insert, update or delete
rows that it can't select.  The only answer I've been able to get about
what SEPostgres will do about that is "we really don't care that we're
breaking SQL semantics".  I don't find that to be a satisfactory answer.
The security-geek reason why not is that it represents a huge
information leak.  The database-geek reason why not is that this will
permanently destroy our ability to do a lot of important optimizations,
eg join removal on the basis of foreign key constraints.  (There are
probably other good reasons, but that one will do for starters.)
Perhaps this is fixable by constraining what a security policy is
allowed to do, or in some other way that I don't know about, but I've
seen no serious discussion about that.

2. I don't understand where to draw the dividing line between database
system accesses (which can't be security constrained, at least not
without breaking things entirely --- eg it will do you little good to
imagine that you can hide rows in pg_security from the
security-enforcement code ;-)) and user accesses that should be
security-constrained.  I am certain that the line is muddied beyond
usability in the current system; there are a lot of user-exposed
functions that are making use of the same infrastructure that core
system accesses do.  As an example, some of the people who think they
want this feature would like to use it to hide the bodies of their
user-defined functions from other people whom they don't wish to see
their code.  But pg_get_functiondef() uses the same catcache
infrastructure as the code that would be called on to actually execute
their function, so there is no reasonable place to prevent the function
body from being exposed through that inquiry function or others of its
ilk.  This problem gets rapidly worse when you consider that Postgres is
designed to be a very extensible system.  It's not clear how to classify
add-on code, and it is clear that any of it could unintentionally
introduce a security hole.  The only solution I can see is "we stop
guaranteeing that SEPostgres is good for anything the moment you load
even one extension module", and that isn't a very satisfactory answer
either.  Even accepting such a restriction, there's too much code in
core Postgres to let anyone feel very good about keeping the core free
of security leaks.

There are some other problems, like the rather frightening thought that
a database dump might silently omit critical data (remember pg_dump is
just another client).  But I think the two categories above cover the
issues that are making me seriously leery of this patch.
        regards, tom lane

OK, time for me to chime in.

I think the outstanding commit-fest items can be broken down into four
sections:
o  Log streamingo  Hot standbyo  SE-PostgreSQLo  Others

I think we all agree that log streaming is not ready for 8.4, and that
delaying for this feature would lead to an indeterminate delay.  I have
already talked to the NTT folks, and as painful as it is, I think they
have accepted this --- many thanks to them.

Hot standby seems to be having a lot of code churn.  I am not sure if it
is because the patch is being polished for completion or if lots of new
problems are being discovered and fixed.  Tom and I have both given up
trying to track this patch.  Fortunately, Heikki has been following it
closely, so I think Heikki is going to make the final decision if hot
standby is ready for 8.4 (because he is going to have to stand behind it
as a committer).

SE-PostgreSQL has been in steady development for a year so this is the
time to decide about it.  My feeling is if we don't accept it now, we
are never going to have SE-Linux or row-level security.  The next week
should show us the right direction when we start discussion on
Wednesday, noon GMT.

The other patches are 1-2 weeks work and will be dealt with like patches
from previous commit-fests.

FYI, I think delaying a major release to get feature X always going to
be counter-productive because it requires restarting development with no
known completion time, and they typical behavior is "just 2 more weeks",
"just two more weeks", over and over again, so no one knows how much
time they have to develop stuff and things stall badly.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

All,

FWIW, I'll comment that what we're seeing here is nothing new.  We have:

--One invasive patch which everybody (myself included) procrastinated on 
reviewing even though we got it early, and:

--One "must have" big complex patch which arrived very late in the 
development cycle.

These are our two reasons for every release delay I can think of.  In 
prior releases, HOT, PITR, Windows, etc., all presented the same issues.

--Josh Berkus

Josh Berkus <josh@agliodbs.com> writes:
> FWIW, I'll comment that what we're seeing here is nothing new.

Certainly the Hot Standby situation is the same old song, different verse.

(I'm personally of the opinion that the project has usually been better
served when we decided not to postpone a release to wait for a specific
feature.)

SEPostgres seems qualitatively different to me, though.  I think PG
people have avoided reviewing it because (a) they weren't interested in
it and (b) they knew they were unqualified to review it.

Meanwhile it's emerging that the selinux people don't feel qualified to
review it either.  I'm not quite sure what to do about that.  But "throw
it in there on faith" doesn't sound like an appealing answer, and I've
got no idea how long it will take to work out a non-faith-based answer.
        regards, tom lane

On Mon, 2009-01-26 at 23:39 -0500, Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
> > FWIW, I'll comment that what we're seeing here is nothing new.

> Meanwhile it's emerging that the selinux people don't feel qualified to
> review it either.  I'm not quite sure what to do about that.  But "throw
> it in there on faith" doesn't sound like an appealing answer, and I've
> got no idea how long it will take to work out a non-faith-based answer.
> 

O.k. maybe it is time to consider something non-traditional. What about
two 8.4 releases?

8.4-stable
8.4-experimental

stable is everything that stable is. PostgreSQL at its best.

experimental is the day after we branch. Same catalog version but
contains say SEPostgres and Hot Standby etc...

8.4-experimental gives people a stable same version compatible version
to test at their leisure. We support it until 8.5 comes out. At 8.5,
those features are merged (or ripped out) into 8.5-stable.

Yes I know this is non-standard for our community but maybe it is time
to crank it up a notch?

Sincerely,

Joshua D. Drake



>             regards, tom lane
> 
-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

> SEPostgres seems qualitatively different to me, though.  I think PG
> people have avoided reviewing it because (a) they weren't interested in
> it and (b) they knew they were unqualified to review it.

I think that you are off-base here.  As I've pointed out previously,
nobody was ever ASSIGNED to review SE-PostgreSQL.

http://archives.postgresql.org/message-id/603c8f070901111132m2a595ec0g677762c1fff587c0@mail.gmail.com

The reviewing that happened during this CommitFest did not happen on
the basis of who was interested in which patches.  There was a bit of
that, but for the most part people reviewed the patches that they were
asked to review.  I assumed (am I the only one?) that the REASON why
we were not asked to review SE-PostgreSQL or Hot Standby is because
the committers were planning to do that themselves due to the
complexity of the patches.

Now, apparently, that assumption was totally wrong.  But this doesn't
seem complicated to me.  If we bump SE-PostgreSQL to the next
CommitFest and assign reviewers, they will review it.  Maybe their
review will not be 100% perfect, but that is why we have committers.
If we continue to NOT assign reviewers, reviewers are unlikely to
crawl out of the woodwork, just as they (mostly) didn't crawl out of
the woodwork for any other patches (HS/SR being a notable exception).

...Robert

Robert,

> The reviewing that happened during this CommitFest did not happen on
> the basis of who was interested in which patches.  There was a bit of
> that, but for the most part people reviewed the patches that they were
> asked to review.  I assumed (am I the only one?) that the REASON why
> we were not asked to review SE-PostgreSQL or Hot Standby is because
> the committers were planning to do that themselves due to the
> complexity of the patches.

Actually, I did assign someone to do a build and specification review. 
But yes, I expected that the code review would *have* to be done by a 
long-term committer.  I pretty much assume that of anything over 300 lines.

The idea behind having new reviewers take on all the small patches, was, 
of course, to give the main committers more time with patches like 
SEPostgres.  It worked with other stuff (like Windowing and CTE).

--Josh

>> The reviewing that happened during this CommitFest did not happen on
>> the basis of who was interested in which patches.  There was a bit of
>> that, but for the most part people reviewed the patches that they were
>> asked to review.  I assumed (am I the only one?) that the REASON why
>> we were not asked to review SE-PostgreSQL or Hot Standby is because
>> the committers were planning to do that themselves due to the
>> complexity of the patches.
>
> Actually, I did assign someone to do a build and specification review. But
> yes, I expected that the code review would *have* to be done by a long-term
> committer.  I pretty much assume that of anything over 300 lines.
>
> The idea behind having new reviewers take on all the small patches, was, of
> course, to give the main committers more time with patches like SEPostgres.
>  It worked with other stuff (like Windowing and CTE).

Right, so, then I'm not sure why Tom is taking the lack of review as a
sign of lack of interest.

...Robert

2009/1/27 Joshua D. Drake <jd@commandprompt.com>:
> On Mon, 2009-01-26 at 23:39 -0500, Tom Lane wrote:
>> Josh Berkus <josh@agliodbs.com> writes:
>> > FWIW, I'll comment that what we're seeing here is nothing new.
>
>> Meanwhile it's emerging that the selinux people don't feel qualified to
>> review it either.  I'm not quite sure what to do about that.  But "throw
>> it in there on faith" doesn't sound like an appealing answer, and I've
>> got no idea how long it will take to work out a non-faith-based answer.
>>
>
> O.k. maybe it is time to consider something non-traditional. What about
> two 8.4 releases?
>
> 8.4-stable
> 8.4-experimental
>
> stable is everything that stable is. PostgreSQL at its best.
>

I dislike this idea - it's same like short processed 8.5 - that is
more simple. Actually current base 8.4 has lot of features, enough for
people, so it could be released. 8.5 should be implemented in shorted
cycle - only one commitfest, that is enough (+3 month) for well
completing SE and replication patches. In czech, big users started
testing and using 8.3, and propably skip 8.4. For small projects
should be 8.4 available early.

regards
Pavel Stehule

> experimental is the day after we branch. Same catalog version but
> contains say SEPostgres and Hot Standby etc...
>
> 8.4-experimental gives people a stable same version compatible version
> to test at their leisure. We support it until 8.5 comes out. At 8.5,
> those features are merged (or ripped out) into 8.5-stable.
>
> Yes I know this is non-standard for our community but maybe it is time
> to crank it up a notch?
>
> Sincerely,
>
> Joshua D. Drake
>
>
>
>>                       regards, tom lane
>>
> --
> PostgreSQL - XMPP: jdrake@jabber.postgresql.org
>   Consulting, Development, Support, Training
>   503-667-4564 - http://www.commandprompt.com/
>   The PostgreSQL Company, serving since 1997
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

On Mon, Jan 26, 2009 at 6:07 PM, Gregory Stark <stark@enterprisedb.com> wrote:
>> I realize in the current system (emailed patches), this would be a horrible
>> pain to maintain such a branch; but perhaps some of the burden could be
>> pushed down to the patch submitters (asking them to merge their own changes
>> into this merged branch).
>
> I've considered maintaining such a repository a few times and dismissed it
> when I realized how much work it would be to maintain.

I don't think it would be too bad if everyone were using git.  And in
fact, if people weren't using git, but we had some kind of a system
that could (1) return a list of all of the current patches and (2)
return for any given patch the message-ids of the messages wherein the
various versions of the patch were submitted, it would be harder, but
possibly managable.  You might have trouble with really big patches
creating lots of merge conflicts, but even if you just merged all the
smaller patches into one tree and push it out for people to test
against, that might still have some real value if a decent number of
people did testing against that tree.

I think that it would probably be pretty easy to write a webapp to
replace the CommitFest web page that basically did the same thing but
with a bit more structure around it - with database tables like
"commitfest", "patch", "patch_version", "patch_comment", and
"patch_review".  I think I might even be willing to write such a
webapp if someone would be willing to provide the infrastructure.  The
CommitFest web page was really useful this time around, but it's not
conducive to any kind of automated pull.

...Robert

On Tue, Jan 27, 2009 at 12:40 AM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
> so it could be released. 8.5 should be implemented in shorted
> cycle - only one commitfest, that is enough (+3 month) for well
> completing SE and replication patches.

we tried this before (8.2 to 8.3 i think), the idea was that the next
release should be in 6 months... we release at least 6 months later...

ATM that a new release cycle starts new patch will arrive and there
will be no way to get the shorted release in time...

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

2009/1/27 Jaime Casanova <jcasanov@systemguards.com.ec>:
> On Tue, Jan 27, 2009 at 12:40 AM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
>> so it could be released. 8.5 should be implemented in shorted
>> cycle - only one commitfest, that is enough (+3 month) for well
>> completing SE and replication patches.
>
> we tried this before (8.2 to 8.3 i think), the idea was that the next
> release should be in 6 months... we release at least 6 months later...
>
> ATM that a new release cycle starts new patch will arrive and there
> will be no way to get the shorted release in time...
>

I remember it. Solution is - don't accept new patches for next commitfest.

Pavel

> --
> Atentamente,
> Jaime Casanova
> Soporte y capacitación de PostgreSQL
> Asesoría y desarrollo de sistemas
> Guayaquil - Ecuador
> Cel. +59387171157
>

Pavel Stehule wrote:
> 2009/1/27 Jaime Casanova <jcasanov@systemguards.com.ec>:
>> On Tue, Jan 27, 2009 at 12:40 AM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
>>> so it could be released. 8.5 should be implemented in shorted
>>> cycle - only one commitfest, that is enough (+3 month) for well
>>> completing SE and replication patches.
>> we tried this before (8.2 to 8.3 i think), the idea was that the next
>> release should be in 6 months... we release at least 6 months later...
>>
>> ATM that a new release cycle starts new patch will arrive and there
>> will be no way to get the shorted release in time...
> 
> I remember it. Solution is - don't accept new patches for next commitfest.

I don't think that'll work. People will still keep writing patches. Or 
if they don't, that's even worse! Either people will work on patches 
that they're interested in, or they'll go away and do something else. 
Only very few will drop their pet projects for the common good and help 
with the review instead.

We can adjust the length of the release cycle by adjusting the number of 
commitfests. I think the normal ~ 1 year cycle is quite optimal, though.

--   Heikki Linnakangas  EnterpriseDB   http://www.enterprisedb.com

Sorry for long description.

Tom Lane wrote:
> Joshua Brindle <method@manicmethod.com> writes:
>> http://marc.info/?l=selinux&m=115762285013528&w=2
>> Is the original discussion thread for the security model used in the 
>> sepostgresql work. Hopefully you'll see some of the evidence you speak of there.
> 
> Thanks for the link.  I took a look through that thread and saw a lot of
> discussion about issues like how to relate the database-side and
> client-side permissions, which is all good stuff but mostly outside my
> purview as a database geek.  I didn't find anything about the stuff that
> is really bothering me, which I think can be broken down into two main
> categories:
> 
> 1. Silently filtering out rows according to an arbitrary security policy
> can break a bunch of fundamental SQL semantics, the most obvious being
> foreign key constraints --- an application might be able to see a
> dependent row that apparently has no referenced row, or might get an
> update or delete failure for a row that is unreferenced as far as it can
> see.  Things get worse if an application can insert, update or delete
> rows that it can't select.  The only answer I've been able to get about
> what SEPostgres will do about that is "we really don't care that we're
> breaking SQL semantics".  I don't find that to be a satisfactory answer.

As I repeated it several times, it is a "covert channel" issue.
An important thing is what the limilation is well documented and
allows end-users to choose the option.
As I summarized in another message, it is not a must-requirement
for security evaluation criteria (ISO/IEC15408, CC).
The criteria allows to include a feature to eliminate covert channels,
but it also allows not to include ones to eliminate them. http://wiki.postgresql.org/wiki/SEPostgreSQL#Covert_channels

For example, Oracle Label Security which is certified as EAL4+ class
and also provides row-level mandatory access controls, but it does not
care about covert channels. This fact is documented as "Security Target",
so end-users can know that elimination of covert channel is over spec
for the product.
They can choose the product with their own responsibility.

At first, we should understand is individual security features have
its own targets, coverages and so on.
I assumes the target of SE-PostgreSQL is enterprise class users
who are interested in web application security like a cloud-computing
platform, similar to Oracle's one.

> The security-geek reason why not is that it represents a huge
> information leak.  The database-geek reason why not is that this will
> permanently destroy our ability to do a lot of important optimizations,
> eg join removal on the basis of foreign key constraints.  (There are
> probably other good reasons, but that one will do for starters.)
> Perhaps this is fixable by constraining what a security policy is
> allowed to do, or in some other way that I don't know about, but I've
> seen no serious discussion about that.

IIRC, we had discussions refering to ISO/IEC15408 a few times at the
past. I believe it was serious discussions.
Polyinstantiation technology might help the situation, but we decided
not to port it on PostgreSQL due to widespread unacceptable changes.

I make clear it again.
It is an over spec for SE-PostgreSQL to eliminate all the covert channels,
however, we documented the limitation for end-users, and they can choose
SE-PostgreSQL with their own responsibility.


> 2. I don't understand where to draw the dividing line between database
> system accesses (which can't be security constrained, at least not
> without breaking things entirely --- eg it will do you little good to
> imagine that you can hide rows in pg_security from the
> security-enforcement code ;-)) and user accesses that should be
> security-constrained.

Please consider the symmetrical architecture between OS and DBMS.
A process accesses resources managed by OS via system calls, and
SELinux acquire the system calls and applies its decision making.
In other hand, a client accesses database object managed by DBMS
via SQL queries, and SE-PostgreSQL applies its decision come from
security policy of SELinux.

Please note that SELinux cannot apply its security policy on
accesses come from in-kernel code in principle, although it
enables to control kernel-loadable-modules.
In same manner, SE-PostgreSQL cannot apply its access controls
on internal database system accesses.

It come from characteristics of a reference monitor which applies
its security policy for all the required accesses, but internal
steps are exceptions.

For example, if SELinux allows a malicious one to load a kernel
loadable module which hijacks filesystems, any other users have
a possibility to invoke the malicious code indirectly which
can bypass SELinux's checks.

> I am certain that the line is muddied beyond
> usability in the current system; there are a lot of user-exposed
> functions that are making use of the same infrastructure that core
> system accesses do.  As an example, some of the people who think they
> want this feature would like to use it to hide the bodies of their
> user-defined functions from other people whom they don't wish to see
> their code.  But pg_get_functiondef() uses the same catcache
> infrastructure as the code that would be called on to actually execute
> their function, so there is no reasonable place to prevent the function
> body from being exposed through that inquiry function or others of its
> ilk.  This problem gets rapidly worse when you consider that Postgres is
> designed to be a very extensible system.  It's not clear how to classify
> add-on code, and it is clear that any of it could unintentionally
> introduce a security hole.

In this case, security administrator should not allow unprivileged
users to invoke pg_get_functiondef(). User invokes pg_get_functiondef()
via SQL queries, but pg_get_functiondef() refers system cache via
its internal process.

For add-on module, SE-PostgreSQL checks whether the client has
db_database:{install_module} privilege on the target database
and installed module (on the filesystem).

I would like to make clear I have never said it is possible to
prevent malicious accesses from system internal entities.

As you mentioned before, it is not fundamentally possible.
Do you remember a previous discussion you suggested to remove
the pgaceAllowOverridePlannerHook() hook and I agreed soon?

> The only solution I can see is "we stop
> guaranteeing that SEPostgres is good for anything the moment you load
> even one extension module", and that isn't a very satisfactory answer
> either. Even accepting such a restriction, there's too much code in
> core Postgres to let anyone feel very good about keeping the core free
> of security leaks.

Forgive me, it seems to me you concern about SE-PostgreSQL is not
a magic-ballet of security. However, we never guarantee it enables
to protect malicious anything. Any security features have its merits
and limitation, needless to say. SELinux is not also perfect.
(At least, who can prove it is perfect?)

Information-Security has a long history more than 2,000 years since
the period of Caesar-cipher, but mankind has not see perfect security
yet. Thus, we need to improve it step by step.

I would like you to understand not negligible number of users are
waiting for SE-PostgreSQL feature, even if these limitations.


> There are some other problems, like the rather frightening thought that
> a database dump might silently omit critical data (remember pg_dump is
> just another client).  But I think the two categories above cover the
> issues that are making me seriously leery of this patch.

As I documented, a client runs pg_dump/pg_restore should have enough
privileges on whole of the databases.

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Bruce Momjian wrote:
> OK, time for me to chime in.
> 
> I think the outstanding commit-fest items can be broken down into four
> sections:
> 
>     o  Log streaming
>     o  Hot standby
>     o  SE-PostgreSQL
>     o  Others
 - snip -

> SE-PostgreSQL has been in steady development for a year so this is the
> time to decide about it.  My feeling is if we don't accept it now, we
> are never going to have SE-Linux or row-level security.  The next week
> should show us the right direction when we start discussion on
> Wednesday, noon GMT.

Hmm...?
This conditional punishment of death seems to me not a reasonable one.
If we can found a matter as a result of discussion, which is impossible
to fix within reasonable term, I'll agree it being postponed to v8.5.
However, why is the punishment of death necessary here?

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Josh Berkus <josh@agliodbs.com> writes:
> The idea behind having new reviewers take on all the small patches, was, 
> of course, to give the main committers more time with patches like 
> SEPostgres.  It worked with other stuff (like Windowing and CTE).

Huh?  There were certainly non-committer reviewers for the window
functions and CTE patches.

But the larger point here is that SEPostgres has been around for nearly
two years, and has been submitted into every 8.4 commit fest not just
this last one, and has drawn no noticeable amount of enthusiasm from the
pghacker community.  It's not just a matter of "we haven't gotten to it
yet"; AFAICS no one has wanted to get to it.
        regards, tom lane

On Mon, Jan 26, 2009 at 8:12 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Josh Berkus <josh@agliodbs.com> writes:
>> So, some feedback to make this decision more difficult:
>
>> Users: care about HS more than anything else in the world.
>
> I don't think this is correct.  There are certainly a lot of users who
> would like an in-core replication solution, but HS by itself is not that
> --- you also need (near) real-time log shipping, which we have already
> decided to punt to 8.5.  That being the case, I think the argument
> that HS is a must-have feature for 8.4 is actually rather weak.

I don't buy that. Sure, sync-rep would be the icing on the cake, but
HS with a small archive_timeout (even of the order of 10 or 15
minutes) would have been extremely useful on a number of systems I
used to run.


-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote:

> Then why has *nobody* stepped up to review the design, much less the
> whole patch?  The plain truth is that no one appears to care enough to
> expend any real effort.

I've spent some time looking at it and have made all the comments I
wished to make. The design seems clear and fit for purpose, having read
KaiGai's excellent Wiki description of how it all fits together and also
read some PDF links Bruce sent out.

But I've not had time to look at the whole patch and my contacts have
not had sufficient time to do anything meaningful with it either.

If we can minimise the impact on normal running and it doesn't have any
implications for robustness, it should be OK. Surely we should give it a
quick review to see if it has any gotchas. If not, and KaiGai is willing
to commit to supporting it, then should be good to go. KaiGai isn't a
home hacker, he's a lead developer for a major multinational, so we
should be able to take his word if he says he will continue to
contribute fixes if problems are found. If we don't commit to him and
his company then they won't commit to us either.

The process works like this: software gets developed, then it gets
certified. If its not certified, then Undercover Elephant will not be
used by the secret people. We can't answer the "will it be certified?"
question objectively yet. If we have someone willing to write the
software and put it forward for certification then we should trust that
it probably will pass certification and if it doesn't we will see
further patches to allow that to happen.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

Dave Page wrote:
> On Mon, Jan 26, 2009 at 8:12 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>   
>> Josh Berkus <josh@agliodbs.com> writes:
>>     
>>> So, some feedback to make this decision more difficult:
>>>       
>>> Users: care about HS more than anything else in the world.
>>>       
>> I don't think this is correct.  There are certainly a lot of users who
>> would like an in-core replication solution, but HS by itself is not that
>> --- you also need (near) real-time log shipping, which we have already
>> decided to punt to 8.5.  That being the case, I think the argument
>> that HS is a must-have feature for 8.4 is actually rather weak.
>>     
>
> I don't buy that. Sure, sync-rep would be the icing on the cake, but
> HS with a small archive_timeout (even of the order of 10 or 15
> minutes) would have been extremely useful on a number of systems I
> used to run.
>
>
>   
+1

I have customers who want exactly this - a simple to administer, 
query-able slave that does DDL transparently and is up to date within a 
controllable time frame. Bluntly, it looks like a killer feature.

regards

Mark

On Jan 27, 2009, at 2:41 AM, Mark Kirkwood wrote:

> Dave Page wrote:
>> On Mon, Jan 26, 2009 at 8:12 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>
>>> Josh Berkus <josh@agliodbs.com> writes:
>>>
>>>> So, some feedback to make this decision more difficult:
>>>>      Users: care about HS more than anything else in the world.
>>>>
>>> I don't think this is correct.  There are certainly a lot of users  
>>> who
>>> would like an in-core replication solution, but HS by itself is  
>>> not that
>>> --- you also need (near) real-time log shipping, which we have  
>>> already
>>> decided to punt to 8.5.  That being the case, I think the argument
>>> that HS is a must-have feature for 8.4 is actually rather weak.
>>>
>>
>> I don't buy that. Sure, sync-rep would be the icing on the cake, but
>> HS with a small archive_timeout (even of the order of 10 or 15
>> minutes) would have been extremely useful on a number of systems I
>> used to run.
>>
>>
>>
> +1
>
> I have customers who want exactly this - a simple to administer,  
> query-able slave that does DDL transparently and is up to date  
> within a controllable time frame. Bluntly, it looks like a killer  
> feature.
>
> regards

+1

So, I am just a lurker here.  I mostly follow hackers to find out if  
any new features are coming out that will make it worth upgrading, and  
to keep up on any backwards compatibly changes that I should be aware  
of.  I am on 8.1 and it performs well and no features added since then  
have seemed worth downing the whole system to do the upgrade for.   
However, a simple to administer, query-able slave that does DDL  
transparently and is up to date within a controllable time frame is  
something that would undoubtably make it worth the upgrade.  Whatever  
version this feature makes it into will probably be the one I will  
upgrade to.

Of course this is just one developer giving you anecdotal evidence and  
there are obviously many concerns other than just how in demand it is,  
but I just wanted to register my vote that this is a very sought after  
feature and it is hard for me to imagine a situation (especially for a  
24x7 web application) where having an easy to admin hot standby server  
wouldn't help your local DBA sleep better at night.

Rick

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> SEPostgres seems qualitatively different to me, though.  I think PG
> people have avoided reviewing it because (a) they weren't interested in
> it and (b) they knew they were unqualified to review it.
>
> Meanwhile it's emerging that the selinux people don't feel qualified to
> review it either.  I'm not quite sure what to do about that.  But "throw
> it in there on faith" doesn't sound like an appealing answer, and I've
> got no idea how long it will take to work out a non-faith-based answer.

Erm, I have to say here that this strikes me as rather unfair.  Perhaps
I'm wrong, but I suspect KaiGai feels pretty good about the patch and
his qualifications in both the PG realm and the SELinux realm.  He's
asking the PG folks to review it because that's the process that the PG
community (through the CommitFest, etc) has laid out for getting a patch
included upstream.  I'm confident KaiGai isn't going to just disappear
into the ether if the patch is committed.

Sure, it'd be nice if 4 or 5 other SELinux developers got in and
understood the PG code well enough to implement such a patch, but I
think the combination of KaiGai (overall), a seperate SELinux hacker
(for the security design and SELinux side of it), and a PG committer
(for where the hooks are placed and how), reviewing the patch and being
comfortable with it is quite sufficient for a high quality result.
Thanks,
    Stephen

Simon Riggs wrote:
> On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote:
> 
>> Then why has *nobody* stepped up to review the design, much less the
>> whole patch?  The plain truth is that no one appears to care enough to
>> expend any real effort.
> 
> I've spent some time looking at it and have made all the comments I
> wished to make. The design seems clear and fit for purpose, having read
> KaiGai's excellent Wiki description of how it all fits together and also
> read some PDF links Bruce sent out.

Thanks for your comment, although you also have a tough work.

> But I've not had time to look at the whole patch and my contacts have
> not had sufficient time to do anything meaningful with it either.
> 
> If we can minimise the impact on normal running and it doesn't have any
> implications for robustness, it should be OK. Surely we should give it a
> quick review to see if it has any gotchas. If not, and KaiGai is willing
> to commit to supporting it, then should be good to go. KaiGai isn't a
> home hacker, he's a lead developer for a major multinational, so we
> should be able to take his word if he says he will continue to
> contribute fixes if problems are found. If we don't commit to him and
> his company then they won't commit to us either.

Needless to say, I will continue to support the feature.
I cannot understand why is it necessary to disappear from here.

At least, a binary with "--enable-selinux" passes all regression
test with/without "pgace_feature=selinux".
The benchmark results I have is a bit legacy, so it is necessary
to record it again, but I don't think it gives significant
implications on normal running (pgace_feature=none).
(Yes, it indeed gives us performance loss with selinux-enabled,but we assume performance is not the first priority in
thiscase.)
 

> The process works like this: software gets developed, then it gets
> certified. If its not certified, then Undercover Elephant will not be
> used by the secret people. We can't answer the "will it be certified?"
> question objectively yet. If we have someone willing to write the
> software and put it forward for certification then we should trust that
> it probably will pass certification and if it doesn't we will see
> further patches to allow that to happen.

-- 
KaiGai Kohei <kaigai@kaigai.gr.jp>

On Tuesday 27 January 2009 00:42:32 Ron Mayer wrote:
> If it were just as easy for us to pull from a
>   "all 'pending-patches' for-commit-fest-nov that pass regression tests"
> branch, I'd happily pull from that instead.

Considering that most patches don't come with regression tests, this would 
accomplish very little.  And even those patches that did come with regression 
tests (e.g., updatable views) need a design analysis much more than running 
an automated test suite.  Ultimately, it does come down to human work.

Robert Haas escribió:

> I think that it would probably be pretty easy to write a webapp to
> replace the CommitFest web page that basically did the same thing but
> with a bit more structure around it - with database tables like
> "commitfest", "patch", "patch_version", "patch_comment", and
> "patch_review".  I think I might even be willing to write such a
> webapp if someone would be willing to provide the infrastructure.  The
> CommitFest web page was really useful this time around, but it's not
> conducive to any kind of automated pull.

Hey, if you're willing to do it, we're certainly accepting proposals.
The current wiki-based CommitFest is supposed to be just a stop-gap.  It
was started not only to support 8.4 development, but also as a test of
the Commitfest idea itself.  This has proven so successful that it's
clear we should be going somewhere with it.

As for somewhere to host it, we certainly have some servers; not tons,
but probably enough.  Some of them even have Postgres running on it.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Tue, Jan 27, 2009 at 1:42 PM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:

> As for somewhere to host it, we certainly have some servers; not tons,
> but probably enough.  Some of them even have Postgres running on it.

We can certainly host an app under postgresql.org. The bigger issue
will be speccing it to meet the requirements of the community without
getting bogged down in bike shedding.


-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

Dave Page wrote:
> On Tue, Jan 27, 2009 at 1:42 PM, Alvaro Herrera
> <alvherre@commandprompt.com> wrote:
> 
>> As for somewhere to host it, we certainly have some servers; not tons,
>> but probably enough.  Some of them even have Postgres running on it.
> 
> We can certainly host an app under postgresql.org. The bigger issue
> will be speccing it to meet the requirements of the community without
> getting bogged down in bike shedding.

I have started some very trivial work around this a while ago with the
intent to get something simple up and working before too much bike
shedding is done. I'll contact Robert off-list to discuss that. If
somebody else - who actively works with what we have now!! - is
interested in that discussion, let me know.

Will obviously take it on-list before any decisions are made. So far I'm
just talking about discussing a prototype.

//Magnus

On 1/27/09, Alvaro Herrera <alvherre@commandprompt.com> wrote:
> Robert Haas escribió:
>  > I think that it would probably be pretty easy to write a webapp to
>  > replace the CommitFest web page that basically did the same thing but
>  > with a bit more structure around it - with database tables like
>  > "commitfest", "patch", "patch_version", "patch_comment", and
>  > "patch_review".  I think I might even be willing to write such a
>  > webapp if someone would be willing to provide the infrastructure.  The
>  > CommitFest web page was really useful this time around, but it's not
>  > conducive to any kind of automated pull.
>
>  Hey, if you're willing to do it, we're certainly accepting proposals.
>  The current wiki-based CommitFest is supposed to be just a stop-gap.  It
>  was started not only to support 8.4 development, but also as a test of
>  the Commitfest idea itself.  This has proven so successful that it's
>  clear we should be going somewhere with it.
>
>  As for somewhere to host it, we certainly have some servers; not tons,
>  but probably enough.  Some of them even have Postgres running on it.

Such app already exists:
 http://ozlabs.org/~jk/projects/patchwork/

So it's a matter of just setting it up.

--
marko

On Sunday 25 January 2009 19:06:50 Tom Lane wrote:
> Particularly with regard to hot standby, which by any sane reading was
> not close to being committable on 1 November (a fortiori from the fact
> that it's *still* not committable despite large amounts of later work).
> I'm also feeling that we are not in a position to commit SE-Postgres in
> a timely fashion; which is not KaiGai-san's fault, rather that of the
> community which has taken nearly zero interest in that patch.
>
> If we want to ensure that 8.5 development opens soon, what we have to do
> is reject those two patches, revert updatable views, and finish up the
> other stuff (which is all small and could likely be dealt with in a week
> or two).

Updatable views is reverted.  I agree that we should reject the rest and 
prepare a release.

Bruce Momjian píše v po 26. 01. 2009 v 23:02 -0500:
> OK, time for me to chime in.
> 
> I think the outstanding commit-fest items can be broken down into four
> sections:
> 
>     o  Log streaming
>     o  Hot standby
>     o  SE-PostgreSQL
>     o  Others

You omit pg_upgrade. Does it mean that this project is already killed
for 8.4?
Thanks Zdenek

On Tuesday 27 January 2009 02:21:41 Tom Lane wrote:
> Then why has *nobody* stepped up to review the design, much less the
> whole patch?  The plain truth is that no one appears to care enough to
> expend any real effort.  But this patch is far too large and invasive
> to accept on the basis that only one guy understands it and will/might
> continue to maintain it.

As one of the earlier reviewers, I think the design is OK, but the way the
implementation is presented was not acceptable, and very little has been
accomplished in terms of reacting to our comments.  For example, where is the
SQL row security feature, which should have been designed, implemented, and
committed separately, in the opinion of most commentaries.

On Tue, Jan 27, 2009 at 2:01 PM, Peter Eisentraut <peter_e@gmx.net> wrote:

> Updatable views is reverted.  I agree that we should reject the rest and
> prepare a release.

That will send a fine message to those companies that have sponsored
development work - that we will arbitrarily reject large patches that
have been worked on following the procedures that we require.

We must at least have the solid belief (of a committer that that has
done a proper review) that a patch cannot be polished in an
appropriate timeframe, or another justifiable reason for rejecting
rather than vague handwaving, guesswork and estimates based on email
traffic.

If we do not, we will rapidly find that no company wants to sponsor
features for PostgreSQL in the future for fear that their money will
be wasted even if they jump through all the right hoops.

-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

Simon Riggs wrote:
> The process works like this: software gets developed, then it gets
> certified. If its not certified, then Undercover Elephant will not be
> used by the secret people. We can't answer the "will it be certified?"
> question objectively yet. If we have someone willing to write the
> software and put it forward for certification then we should trust that
> it probably will pass certification and if it doesn't we will see
> further patches to allow that to happen.

For what it's worth, we can see that there are indeed
Postgres forks on the Common Criteria certified list.
http://www.commoncriteriaportal.org/products_DB.html   PostgreSQL Certified Version V8.1.5 for Linux   Manufacturer
Assurancelevel     Certification date   NTT DATA CORPORATION     EAL1     22-MAR-07   Certification report
c0089_ecvr.pdf  http://www.commoncriteriaportal.org/files/epfiles/c0089_ecvr.pdf
 

though at EAL1 they're quite far from the EAL4+ that DB2,
Oracle, etc get.

That someone went through the effort suggests that there's at least
some interest in getting security certifications for postgres.

It'd be interesting to hear from whomever at NTT was involved with
that certification, if SEPostgreSQL would have either made that
process easier or help postgres achieve a higher level.

Dave Page wrote:
> On Mon, Jan 26, 2009 at 11:35 AM, Heikki Linnakangas
> <heikki.linnakangas@enterprisedb.com> wrote:
> 
>> I'm sure it depends on the user. Users that are more interested in the
>> features we already have in the bag like window functions and WITH-clause,
>> will obviously prefer to release earlier without hot standby. And users that
>> want hot standby (or SE-postgresql) will prefer to delay the release and
>> have those features included.
> 
> At LinuxLive (UK) the overwhelming majority of people I spoke to over
> three days wanted hot standby and replication (preferably
> multi-master, but thats another story). Window functions, recursive
> queries, SE PostgreSQL, updatable views and other new features were
> barely mentioned.

I'd say the FSM rework is the largest feature in 8.4 that most of my
customers would have immediate use for. With visibility map not far
behind. It's not a "sexy" feature, it's removing a piece of annoyance.
So it's not something people would think of when you say "what feature
are you looking for in the next version". But it will help pretty much
all users, and that's certainly more than hot standby for example. And
the longer we push that back for some other feature, the more these
users suffer.

That said, I've certainly got a fair number of places where hot standby
would be very popular. More than most others on your list above. And I'm
not making a comment as to how ready hot standby is - I haven't kept up
enough to comment on that.


> As I've pointed out before, we're not a commercial company working for
> our shareholders, we're a FOSS project working for our end users. If
> we can include an important and popular feature like this at the
> expense of a few weeks extra wait for the release, it seems to me that
> we'll be serving our users far better overall than making a fair
> percentage of them wait another 12 months for work that is more or
> less complete.

Just playing the devils advocate here, but you can turn that argument
around easily. We're not a commercial company who need to release a
feature just because marketing said it'd be there...

//Magnus

> I have started some very trivial work around this a while ago with the
> intent to get something simple up and working before too much bike
> shedding is done. I'll contact Robert off-list to discuss that. If
> somebody else - who actively works with what we have now!! - is
> interested in that discussion, let me know.
>
> Will obviously take it on-list before any decisions are made. So far I'm
> just talking about discussing a prototype.

Sounds good.  I think we will have the best chance of success if we
keep it real simple.  I don't want this to turn into a propaganda war
about using everyone's favorite tool.  I just want to write down a
database schema that mimics the organization of the existing wiki
page, put a thin web interface around it, and call it a day.  It will
take longer to analyze whether some other tool is sufficiently close
to that than it will to write a tool that is exactly that.

...Robert

Peter Eisentraut wrote:
> On Tuesday 27 January 2009 00:42:32 Ron Mayer wrote:
>> If it were just as easy for us to pull from a
>>   "all 'pending-patches' for-commit-fest-nov that pass regression tests"
>> branch, I'd happily pull from that instead.
> 
> Considering that most patches don't come with regression tests, this would 
> accomplish very little.  And even those patches that did come with regression 
> tests (e.g., updatable views) need a design analysis much more than running 
> an automated test suite.  Ultimately, it does come down to human work.

So long as the patch passes the pre-existing regression tests, it's likely
to be stable enough to run on some of our development instances.

I certainly don't suggest that this is a substitute for reviews.  Just that
more testing of patches might happen incidentally (by people who currently
test their own software against CVS head) if all the pending patches
for a commit fest were as easy to pull as CVS head.

Peter,

* Peter Eisentraut (peter_e@gmx.net) wrote:
> As one of the earlier reviewers, I think the design is OK, but the way the
> implementation is presented was not acceptable, and very little has been
> accomplished in terms of reacting to our comments.  For example, where is the
> SQL row security feature, which should have been designed, implemented, and
> committed separately, in the opinion of most commentaries.

Eh?  Are you thinking of column-level privileges, which was committed
last week?  The SQL spec doesn't define row-level security, and coming
up with something willy-nilly on our own doesn't really strike me as the
best approach.  Oracle, SQL Server, etc, also use the security labels
concept that the SE-PostgreSQL patch implements.
Thanks,
    Stephen

Dave Page wrote:
> On Tue, Jan 27, 2009 at 2:01 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> 
>> Updatable views is reverted.  I agree that we should reject the rest and
>> prepare a release.
> 
> That will send a fine message to those companies that have sponsored
> development work - that we will arbitrarily reject large patches that
> have been worked on following the procedures that we require.

To some extent that seems more an issue of linguistics and tone.

If Peter had written  "we should defer the rest and try to help resolve
specific issues identified in the reviews during commitfest 2009-First",
sponsors might be happy rather than upset.


I think one can make a strong argument that the features should
be moved to the next commit-fest, just so the other patches in that
commit fest ( http://wiki.postgresql.org/wiki/CommitFest_2009-First )
don't bit-rot too badly.

Whether the community wants to release an 8.4 between
commitfest 2008-11 and 2009-First seems to me a largely
orthogonal question that would be based more on what demand
there is for an 8.4 release and how distracting it would be
to do a release between commitfests.

On Tue, Jan 27, 2009 at 06:20:41AM -0800, Ron Mayer wrote:
> For what it's worth, we can see that there are indeed
> Postgres forks on the Common Criteria certified list.
> 
>  http://www.commoncriteriaportal.org/products_DB.html
>     PostgreSQL Certified Version V8.1.5 for Linux
>     Manufacturer     Assurance level     Certification date
>     NTT DATA CORPORATION     EAL1     22-MAR-07
>     Certification report
>     c0089_ecvr.pdf
>     http://www.commoncriteriaportal.org/files/epfiles/c0089_ecvr.pdf
> 
> though at EAL1 they're quite far from the EAL4+ that DB2,
> Oracle, etc get.

As far as I understand, the different levels are about assuring a
set of code/features to some assurance level.  The Wikipedia page[1]
gives a reasonable overview of the levels, but basically EAL1 says
that a limited amount of effort (in practical terms, several person
months/years of time for something like PG) was put in, EAL4 is the
highest level before things start getting formal (i.e. you actually have
to start doing some mathematical proofs about the design) and EAL7 has
barely started, but says that the design is formally verified but the
code isn't (as far as I understand).  Research groups are suggesting
that there should also be levels above EAL7 as we are *starting* to know
how to verify code well enough that the code, as well as the design, can
now be formally verified (e.g. [2]).

Equally important as the assurance level are the actual feature set
(there are technical names for this that I know very little about) that
was actually tested for.  For example, it would be comparatively easy
to get PG certified saying that it loads and could be killed, but much
harder to get it certified as complying with the complete SQL spec.

--  Sam  http://samason.me.uk/
[1] http://en.wikipedia.org/wiki/Evaluation_Assurance_Level[2] http://ertos.nicta.com.au/research/l4.verified/

Tom Lane wrote:
> Joshua Brindle <method@manicmethod.com> writes:
>> http://marc.info/?l=selinux&m=115762285013528&w=2
>> Is the original discussion thread for the security model used in the 
>> sepostgresql work. Hopefully you'll see some of the evidence you speak of there.
> 
> Thanks for the link.  I took a look through that thread and saw a lot of
> discussion about issues like how to relate the database-side and
> client-side permissions, which is all good stuff but mostly outside my
> purview as a database geek.  I didn't find anything about the stuff that
> is really bothering me, which I think can be broken down into two main
> categories:
> 
> 1. Silently filtering out rows according to an arbitrary security policy
> can break a bunch of fundamental SQL semantics, the most obvious being
> foreign key constraints --- an application might be able to see a

This is correct. Strange error conditions can happen when you are using 
mandatory access controls. The same thing happened in linux when selinux was 
introduced. There was plenty of code out there that assumed if it was running as 
root it could do anything. Lots of it didn't even check for error conditions. 
The existence of poorly written applications should never be an argument against 
adding security.

> dependent row that apparently has no referenced row, or might get an
> update or delete failure for a row that is unreferenced as far as it can
> see.  Things get worse if an application can insert, update or delete
> rows that it can't select.  The only answer I've been able to get about

Because type enforcement (the primary mechanism behind selinux) is very flexible 
it is true that policy writers have plenty of rope to hang themselves with. We 
can only attempt to educate and document these issues, blocking out security is 
not a satisfactory answer.

> what SEPostgres will do about that is "we really don't care that we're
> breaking SQL semantics".  I don't find that to be a satisfactory answer.

Plenty of people feel the same way about SELinux (or any mandatory access 
controls). That is why there are options, if you want this security and don't 
care if your applications puke then enable it, else disable it. Noone is going 
to force you to use this, right?

> The security-geek reason why not is that it represents a huge
> information leak.  The database-geek reason why not is that this will

The great thing about security is that, by itself, it actually doesn't mean 
anything. Security is where you are willing to balance between stopping people 
from getting something done and letting them get something done. In this case, 
removing all covert channels would not only be impossible but it would make an 
unusable database system. In SELinux we didn't worry about covert flows, 
(actually we ignored/documented plenty of overt flows as well). With something 
as complex as the Linux kernel or an enterprise rdbms it is nearly impossible to 
eliminate such things.

People who need absolute separation of information already have options, 
multiple server processes, polyinstanciated views, etc. For people that don't 
care if someone can see that you've inserted a couple rows since the primary key 
got larger, or can tell that an associated row that isn't visible exists in 
another table a more flexible, yet still mandatory system like sepostgresql is 
the answer.

The great thing about this work is that, in many cases, a well designed system 
(that is, well designed for the security policy it is going to be constrained 
under) should not be impacted greatly by these issues. If a client needs 
information where they can't see all of the associated rows you can have trusted 
stored procedures (which run in a different selinux context, as defined by a 
type transition from the client context) that do the work and return the 
appropriate results. I know you can't use stored procedures for everything but 
they'd go a long way in binding queries we trust to the data they expose (just 
like in SELinux we bind binary code on the filesystem to domains that code can 
be used to enter)

> permanently destroy our ability to do a lot of important optimizations,
> eg join removal on the basis of foreign key constraints.  (There are
> probably other good reasons, but that one will do for starters.)
> Perhaps this is fixable by constraining what a security policy is
> allowed to do, or in some other way that I don't know about, but I've
> seen no serious discussion about that.
> 
> 2. I don't understand where to draw the dividing line between database
> system accesses (which can't be security constrained, at least not
> without breaking things entirely --- eg it will do you little good to
> imagine that you can hide rows in pg_security from the
> security-enforcement code ;-)) and user accesses that should be
> security-constrained.  I am certain that the line is muddied beyond
> usability in the current system; there are a lot of user-exposed
> functions that are making use of the same infrastructure that core
> system accesses do.  As an example, some of the people who think they
> want this feature would like to use it to hide the bodies of their
> user-defined functions from other people whom they don't wish to see
> their code.  But pg_get_functiondef() uses the same catcache
> infrastructure as the code that would be called on to actually execute
> their function, so there is no reasonable place to prevent the function
> body from being exposed through that inquiry function or others of its
> ilk.  This problem gets rapidly worse when you consider that Postgres is
> designed to be a very extensible system.  It's not clear how to classify
> add-on code, and it is clear that any of it could unintentionally
> introduce a security hole.  The only solution I can see is "we stop
> guaranteeing that SEPostgres is good for anything the moment you load
> even one extension module", and that isn't a very satisfactory answer
> either.  Even accepting such a restriction, there's too much code in
> core Postgres to let anyone feel very good about keeping the core free
> of security leaks.
> 

KaiGai already talked about the permissions he's added to prevent user functions 
from being inserted. This is in the same class as in SELinux where we can't stop 
you from taking over the machine and disabling SELinux if you can insert kernel 
modules so we'll control the insertion of modules and if a policy writer allows 
it then so be it.

> There are some other problems, like the rather frightening thought that
> a database dump might silently omit critical data (remember pg_dump is
> just another client).  But I think the two categories above cover the
> issues that are making me seriously leery of this patch.
> 

pg_dump, just like a backup agent on an SELinux machine would have to have read 
access. Backup applications are generally considered trusted applications.

Stephen Frost wrote:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> SEPostgres seems qualitatively different to me, though.  I think PG
>> people have avoided reviewing it because (a) they weren't interested in
>> it and (b) they knew they were unqualified to review it.
>>
>> Meanwhile it's emerging that the selinux people don't feel qualified to
>> review it either.  I'm not quite sure what to do about that.  But "throw
>> it in there on faith" doesn't sound like an appealing answer, and I've
>> got no idea how long it will take to work out a non-faith-based answer.
> 
> Erm, I have to say here that this strikes me as rather unfair.  Perhaps
> I'm wrong, but I suspect KaiGai feels pretty good about the patch and
> his qualifications in both the PG realm and the SELinux realm.  He's

Not only that but he's had many discussions with us about sepostgres, from the 
security model to his reimplementation of the access vector cache. Just because 
we haven't been on this list doesn't mean we haven't been watching the work.

> asking the PG folks to review it because that's the process that the PG
> community (through the CommitFest, etc) has laid out for getting a patch
> included upstream.  I'm confident KaiGai isn't going to just disappear
> into the ether if the patch is committed.
> 

He hasn't disappeared yet, that is probably a good sign :)

> Sure, it'd be nice if 4 or 5 other SELinux developers got in and
> understood the PG code well enough to implement such a patch, but I

We aren't a huge community and because of the nature of SELinux we have people 
spread out over many different projects (X, dbus, NFS, distributions, 
ipsec/networking, solaris fmac), etc. I'm probably more familiar with databases 
than the others so I'm here to help (though my time is also spread over many 
other things).

> think the combination of KaiGai (overall), a seperate SELinux hacker
> (for the security design and SELinux side of it), and a PG committer
> (for where the hooks are placed and how), reviewing the patch and being
> comfortable with it is quite sufficient for a high quality result.

That is all I asked for. No matter how familiar I become with the pgsql code 
I'll never be as qualified as you guys for identifying security hook call sites 
that are missing/misplaced. Assuming I think the security backend is correct 
then it shouldn't be hard for you guys to look at the docs, see that permissions 
x, y and z are required for operation foo, and know where the possible codepaths 
for operation foo are and check that the hooks for x, y and z are called.

On Wed, Jan 28, 2009 at 1:35 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> I have started some very trivial work around this a while ago with the
>> intent to get something simple up and working before too much bike
>> shedding is done. I'll contact Robert off-list to discuss that. If
>> somebody else - who actively works with what we have now!! - is
>> interested in that discussion, let me know.

I'm very interested in that discussion.  I don't know whether I am
"actively working" with what we have now, but that's because since I
wrote the original template structure, it hasn't changed a whole lot.
Most of the tweaking has had to do with presentation, and massaging
mediawiki to do what we wanted.

As Alvaro points out, the wiki approach was intended to provide a
stop-gap solution to patch tracking, and also to help us identify what
we actually needed from a patch tracker, so that we could make a
sensible decision about which tool to use when we did eventually move
forward.

>>
>> Will obviously take it on-list before any decisions are made. So far I'm
>> just talking about discussing a prototype.
>
> Sounds good.  I think we will have the best chance of success if we
> keep it real simple.  I don't want this to turn into a propaganda war
> about using everyone's favorite tool.  I just want to write down a
> database schema that mimics the organization of the existing wiki
> page, put a thin web interface around it, and call it a day.  It will
> take longer to analyze whether some other tool is sufficiently close
> to that than it will to write a tool that is exactly that.
>

I can understand the desire to avoid a propaganda war.  These
discussions have borne little fruit previously, in part because we
haven't had a clear idea of what was actually required from the tool.

I think the picture has started to become more clear during the 8.4
dev cycle.  Most importantly, there was much ado made about the need
for powerful email integration features in previous discussions.  This
severely restricted our choices (possibly to zero?).  I feel that the
commitfest wiki has demonstrated that no such integration is required.Everyone wants to keep on using the mailing list
fordiscussion, but
 
we need somewhere else to keep track of patches and their status.

To my knowledge, authors have been happy to add patches to the wiki
and reviewers have been happy to update their status with no email
integration whatsoever.  We've continued to discuss things on the
lists, while updating the wiki as required.

If we forget about trying to integrate with email, the field opens
right up and we can use pretty much any just-install-the-package
tracking software out there and it will get the job done.  For the
sake of not advocating my "favourite tool", I won't name any
particular software, but I can think of several off the top of my head
that could mirror the structure we currently have on the wiki without
stretching.

I think it's possible to skip the "roll our own" step in all of this
and just move on to using a ready-made solution.  In reality our
requirements are very simple.  Writing a low-fi version of the wiki
would be pretty easy, but just dropping the patch data we already have
into a patch tracker would be even easier.

Cheers,
BJ

> I think it's possible to skip the "roll our own" step in all of this
> and just move on to using a ready-made solution.  In reality our
> requirements are very simple.  Writing a low-fi version of the wiki
> would be pretty easy, but just dropping the patch data we already have
> into a patch tracker would be even easier.

Well, if you're volunteering to set something up... great.  We'll take
a look at it when you have it working.

That's not what I'm volunteering to do, though.

...Robert

On Tue, 2009-01-27 at 06:40 +0100, Pavel Stehule wrote:

> > 8.4-stable
> > 8.4-experimental
> >
> > stable is everything that stable is. PostgreSQL at its best.
> >
> 
> I dislike this idea - it's same like short processed 8.5 - 

Actually it isn't because we wouldn't accept features into
8.4-experimental. The only thing we would accept into 8.4-experimental
would be bug fixes that would automatically be ported up to 8.5 (or
perhaps the other way around). We would still continue to build 8.5 as
normal. 

> that is
> more simple.

We have tried the short release cycle before, it was called 8.2. It
fails, remarkably.

> 
> regards
> Pavel Stehule
> 

Well like I said, its just an idea :)

Joshua D. Drake

-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

On Tue, 2009-01-27 at 00:58 -0500, Jaime Casanova wrote:
> On Tue, Jan 27, 2009 at 12:40 AM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
> > so it could be released. 8.5 should be implemented in shorted
> > cycle - only one commitfest, that is enough (+3 month) for well
> > completing SE and replication patches.
> 
> we tried this before (8.2 to 8.3 i think), the idea was that the next
> release should be in 6 months... we release at least 6 months later...

8.2 was a short cycle release that lasted just as long as a normal
release cycle :)

Joshua D. Drake

-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
> > So, some feedback to make this decision more difficult:
> 
> > Users: care about HS more than anything else in the world.
> 
> I don't think this is correct.

I do.

People literally grab my shoulder and ask when we'll have it.  I've
never seen anything like the interest in this for any database
feature, including the fairies-and-unicorns multi-master replication
people imagine will scale linearly in every dimension by plugging in
nodes.

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Dave Page <dpage@pgadmin.org> writes:
> On Tue, Jan 27, 2009 at 2:01 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>> Updatable views is reverted.  I agree that we should reject the rest and
>> prepare a release.

> That will send a fine message to those companies that have sponsored
> development work - that we will arbitrarily reject large patches that
> have been worked on following the procedures that we require.

"defer to 8.5" would have been better phraseology than "reject",
no doubt.

> We must at least have the solid belief (of a committer that that has
> done a proper review) that a patch cannot be polished in an
> appropriate timeframe,

I already pointed out some pretty serious problems with the updatable
views patch.  Are you claiming they are trivial to fix?
        regards, tom lane

On Tue, Jan 27, 2009 at 3:40 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Dave Page <dpage@pgadmin.org> writes:
>
>> We must at least have the solid belief (of a committer that that has
>> done a proper review) that a patch cannot be polished in an
>> appropriate timeframe,
>
> I already pointed out some pretty serious problems with the updatable
> views patch.  Are you claiming they are trivial to fix?

Not at all. I think the deferral of that particular patch is the
correct thing to do because there are confirmed, real problems with it
that are not realistic to fix in an appropriate timeframe for the
release. The primary case that I'm objecting to is HS which you've
been saying will take 10 - 12 months to complete having by your own
admission not looked at the code or followed the discussion
particularly closely.


-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

On Tue, 2009-01-27 at 14:10 +0000, Dave Page wrote:
> On Tue, Jan 27, 2009 at 2:01 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> 
> > Updatable views is reverted.  I agree that we should reject the rest and
> > prepare a release.
> 
> That will send a fine message to those companies that have sponsored
> development work - that we will arbitrarily reject large patches that
> have been worked on following the procedures that we require.

We are not subject to the whims of company sponsorship. We are not a
company with shareholders... Where have I heard that before?

> 
> We must at least have the solid belief (of a committer that that has
> done a proper review) that a patch cannot be polished in an
> appropriate timeframe, or another justifiable reason for rejecting
> rather than vague handwaving, guesswork and estimates based on email
> traffic.
> 

If this is actually what happen then I agree. I am not sure that it is
though.


> If we do not, we will rapidly find that no company wants to sponsor
> features for PostgreSQL in the future for fear that their money will
> be wasted even if they jump through all the right hoops.
> 

I sincerely doubt this. The sponsoring company if properly educated
about the process understands this is a possibility. It is a risk. If
they don't have a contract in place that allows for things like this
including responsibility on the developer end, then that is there
problem. Not to mention the developer could offer to support their patch
for a time until it is mature enough to be committed.

Sincerely,

Joshua D. Drake

-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

On Tue, Jan 27, 2009 at 3:48 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
> On Tue, 2009-01-27 at 14:10 +0000, Dave Page wrote:
>> On Tue, Jan 27, 2009 at 2:01 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>>
>> > Updatable views is reverted.  I agree that we should reject the rest and
>> > prepare a release.
>>
>> That will send a fine message to those companies that have sponsored
>> development work - that we will arbitrarily reject large patches that
>> have been worked on following the procedures that we require.
>
> We are not subject to the whims of company sponsorship. We are not a
> company with shareholders... Where have I heard that before?

Not basing our release schedule on our commitments to shareholders is
an entirely different thing to treating sponsors of major features
like crap by arbitrarily bouncing the patches they've paid to have
properly developed within the community process with no good reason.

-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

Dave Page <dpage@pgadmin.org> writes:
> On Mon, Jan 26, 2009 at 8:12 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Josh Berkus <josh@agliodbs.com> writes:
>>> Users: care about HS more than anything else in the world.
>> 
>> I don't think this is correct.  There are certainly a lot of users who
>> would like an in-core replication solution, but HS by itself is not that
>> --- you also need (near) real-time log shipping, which we have already
>> decided to punt to 8.5.  That being the case, I think the argument
>> that HS is a must-have feature for 8.4 is actually rather weak.

> I don't buy that. Sure, sync-rep would be the icing on the cake, but
> HS with a small archive_timeout (even of the order of 10 or 15
> minutes) would have been extremely useful on a number of systems I
> used to run.

Sure, I don't deny that HS by itself would have significant use cases.
But what those zillions of users want is easy-to-set-up replication
(think mysql).  Without an integrated and fairly high-performance log
shipping capability, they are not going to find HS very compelling.
Claiming otherwise is just wishful thinking.

My own feeling about it is that once we have both HS and log shipping
integrated and reasonably well polished, we'd have something that
deserved the fabled 9.0 version number.  But that's probably a year
away, and we are not doing anyone a favor by not putting out 8.4
in the meantime.
        regards, tom lane

On Mon, 2009-01-26 at 22:55 -0500, Tom Lane wrote:

> Silently filtering out rows according to an arbitrary security policy
> can break a bunch of fundamental SQL semantics, the most obvious being
> foreign key constraints

That was exactly my reaction when I read the way it worked and I was
ready to reject the patch as a result. Bruce and KaiGai provided
documents that discuss the problem and it's a clearly a known issue in
the security community. Specifically, it hasn't prevented Oracle from
gaining security Certification and it shouldn't prevent us either. In
the end it's the certification that matters here, rather than a general
review of what database security is, or could be.

I've seen enough to be happy that KaiGai has done a thorough job on
*attempting* to address the needs of the security people. Passing
security audit is the real test and I won't be beating him up if we do
miss slightly. We have to try, otherwise we'll never know. 

My concerns are all about what it does to our code and the impacts of
that. These are things we know how to check.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

On Tue, 2009-01-27 at 15:51 +0000, Dave Page wrote:
> On Tue, Jan 27, 2009 at 3:48 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
> > On Tue, 2009-01-27 at 14:10 +0000, Dave Page wrote:
> >> On Tue, Jan 27, 2009 at 2:01 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> >>
> >> > Updatable views is reverted.  I agree that we should reject the rest and
> >> > prepare a release.
> >>
> >> That will send a fine message to those companies that have sponsored
> >> development work - that we will arbitrarily reject large patches that
> >> have been worked on following the procedures that we require.
> >
> > We are not subject to the whims of company sponsorship. We are not a
> > company with shareholders... Where have I heard that before?
> 
> Not basing our release schedule on our commitments to shareholders is
> an entirely different thing to treating sponsors of major features
> like crap by arbitrarily bouncing the patches they've paid to have
> properly developed within the community process with no good reason.

Certainly but I haven't seen a suggestion to that. Updateable views has
as I have seen in threads, issues that can not be fixed in the
appropriately time line. If they can be fixed for 8.5 great.

Sincerely,

Joshua D. Drake

-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

On Tue, 2009-01-27 at 10:51 -0500, Tom Lane wrote:
> Without an integrated and fairly high-performance log
> shipping capability, they are not going to find HS very compelling.
> Claiming otherwise is just wishful thinking.

What HS will give us is same or better than the equivalent feature in
latest release of Oracle. Oracle requires you to manually
freeze/unfreeze the standby and so the data is never even close to being
current.

For us, it might be better if it was streamed, but its not a critically
important issue for most users. An idle server is.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

Dave Page <dpage@pgadmin.org> writes:
> On Tue, Jan 27, 2009 at 3:40 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I already pointed out some pretty serious problems with the updatable
>> views patch.  Are you claiming they are trivial to fix?

> Not at all. I think the deferral of that particular patch is the
> correct thing to do because there are confirmed, real problems with it
> that are not realistic to fix in an appropriate timeframe for the
> release. The primary case that I'm objecting to is HS which you've
> been saying will take 10 - 12 months to complete having by your own
> admission not looked at the code or followed the discussion
> particularly closely.

Well, perhaps I'm being pessimistic, or perhaps you're being
optimistic.  What is undeniable fact is that HS will not be committable
this week, which will make it three months since feature freeze.
As for when it *will* be committable --- Heikki is saying two weeks
if no new problems crop up, but given the rate at which new problems
have been found so far, what are the odds of that?  We've seen this
movie before.

Since it's going to take us two weeks to clean up the other loose ends
anyway, there's no harm in letting Simon and Heikki try to complete the
patch by then.  But I'll happily lay a side bet with you about what the
situation will be two weeks from now.

Even if the improbable happens and there's an apparently bug-free patch
ready in two weeks, I think it would be far better project management to
have it go in at the beginning of the 8.5 cycle than at the tail end of
8.4.  This is a major patch that has very real possibilities for
breaking plain old crash recovery, even for users not using HS.  The
idea of shipping it with only a minimal amount of testing should scare
the pants off you.
        regards, tom lane

Simon Riggs <simon@2ndQuadrant.com> writes:
> On Mon, 2009-01-26 at 22:55 -0500, Tom Lane wrote:
>> Silently filtering out rows according to an arbitrary security policy
>> can break a bunch of fundamental SQL semantics, the most obvious being
>> foreign key constraints

> That was exactly my reaction when I read the way it worked and I was
> ready to reject the patch as a result. Bruce and KaiGai provided
> documents that discuss the problem and it's a clearly a known issue in
> the security community. Specifically, it hasn't prevented Oracle from
> gaining security Certification and it shouldn't prevent us either. In
> the end it's the certification that matters here, rather than a general
> review of what database security is, or could be.

Yeah, people like certification, but they also like products that work.
Did you stop reading before getting to my non-security-based complaints?
        regards, tom lane

On Tue, Jan 27, 2009 at 4:23 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Since it's going to take us two weeks to clean up the other loose ends
> anyway, there's no harm in letting Simon and Heikki try to complete the
> patch by then.  But I'll happily lay a side bet with you about what the
> situation will be two weeks from now.

If Heikki comes back and says it's not feasible, then so be it. I'll
be happy because that decision was made following a proper review and
some effort to get the work done.

> Even if the improbable happens and there's an apparently bug-free patch
> ready in two weeks, I think it would be far better project management to
> have it go in at the beginning of the 8.5 cycle than at the tail end of
> 8.4.  This is a major patch that has very real possibilities for
> breaking plain old crash recovery, even for users not using HS.  The
> idea of shipping it with only a minimal amount of testing should scare
> the pants off you.

It scares me for sure, but I'm reassured because I know and trust the
guys at 2ndQuadrant who have been testing extensively, as well as
other people such as Merlin and Heikki. Most patches get far less
testing I'd wager.

-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

> Yeah, people like certification, but they also like products that work.
> Did you stop reading before getting to my non-security-based complaints?

I read them, but I suspect they are issues that can be addressed.  How
would any of this affect join removal, anyway?  At most it would
affect join removal WHEN USING SE-PostgreSQL, but I don't even see why
it would affect that.  We've already decided we're not overly
concerned with covert channels, and the user being able to discern
that a join got removed is surely no more than that.

Furthermore, as covert channels go, it seems unlikely to be the one
that breaks the bank.

...Robert

David Fetter <david@fetter.org> writes:
> On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote:
>> I don't think this is correct.

> I do.

> People literally grab my shoulder and ask when we'll have it.

Do these people understand the difference between HS and a complete
replication solution?  Are they still as excited after you explain
the difference?
        regards, tom lane

On Tue, 2009-01-27 at 11:23 -0500, Tom Lane wrote:
> Dave Page <dpage@pgadmin.org> writes:
> > On Tue, Jan 27, 2009 at 3:40 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> I already pointed out some pretty serious problems with the updatable
> >> views patch.  Are you claiming they are trivial to fix?
> 

> Even if the improbable happens and there's an apparently bug-free patch
> ready in two weeks, I think it would be far better project management to
> have it go in at the beginning of the 8.5 cycle than at the tail end of
> 8.4.  This is a major patch that has very real possibilities for
> breaking plain old crash recovery, even for users not using HS.  The
> idea of shipping it with only a minimal amount of testing should scare
> the pants off you.

This is my take as well. This is very real, very scary things that are
being worked on. HS should only ship after a very, very long non change
cycle (meaning no significant bugs (or refactoring) found in HS patch
for X period of time)... say after a full 8.5 dev cycle. I do not want
to commit this patch and then have to yank it out 3 months from now.

Lastly, the last time a developer told me two weeks it was 3 months.
Unless we get a written development plan that describes specifically
what, when, why and how long I am severely suspect that Heikki or Simon
have a clue on an actual deliverable time line (no offense guys).
Developers are generally overly optimistic about their delivery time
lines. 

Oh and before someone says, "Why would I write that. It is a waste of
time when I could be coding..." That was the second thing they said to
be before it took 3 months.

Joshua D. Drake



-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

On Tue, 2009-01-27 at 11:36 -0500, Tom Lane wrote:
> David Fetter <david@fetter.org> writes:
> > On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote:
> >> I don't think this is correct.
> 
> > I do.
> 
> > People literally grab my shoulder and ask when we'll have it.
> 
> Do these people understand the difference between HS and a complete
> replication solution?  Are they still as excited after you explain
> the difference?

No. Because everyone I see rambling about it don't realize there is a
difference.

Joshua D. Drake


> 
>             regards, tom lane
> 
-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

On Tuesday 27 January 2009 17:47:52 Dave Page wrote:
> The primary case that I'm objecting to is HS which you've
> been saying will take 10 - 12 months to complete having by your own
> admission not looked at the code or followed the discussion
> particularly closely.

Is there another committer or expert with a different ETA estimate?

On Tue, Jan 27, 2009 at 4:42 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On Tuesday 27 January 2009 17:47:52 Dave Page wrote:
>> The primary case that I'm objecting to is HS which you've
>> been saying will take 10 - 12 months to complete having by your own
>> admission not looked at the code or followed the discussion
>> particularly closely.
>
> Is there another committer or expert with a different ETA estimate?
>

Heikki, who has been in-depth reviewing the patch for some time,
quoted a figure of 2 weeks for the known issues.

AFAIK, he has not revised that estimate since.

-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

Joshua D. Drake wrote:
> Lastly, the last time a developer told me two weeks it was 3 months.
> Unless we get a written development plan that describes specifically
> what, when, why and how long I am severely suspect that Heikki or Simon
> have a clue on an actual deliverable time line (no offense guys).
> Developers are generally overly optimistic about their delivery time
> lines. 

I'm totally aware of that. My two weeks estimate really was with the 
assumption that no new major issues crop up. I totally expect that new 
major issues will crop up. But I'm going to give it a shot. I'm planning 
to review the code anyway, whether it goes into 8.4 or 8.5, so I might 
as well keep going and hope for the best.

--   Heikki Linnakangas  EnterpriseDB   http://www.enterprisedb.com

On Tue, 2009-01-27 at 11:26 -0500, Tom Lane wrote:

> Yeah, people like certification, but they also like products that work.
> Did you stop reading before getting to my non-security-based complaints?

Yes, I'm sorry, I did. Will read on.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

Robert Haas <robertmhaas@gmail.com> writes:
>> Yeah, people like certification, but they also like products that work.
>> Did you stop reading before getting to my non-security-based complaints?

> I read them, but I suspect they are issues that can be addressed.  How
> would any of this affect join removal, anyway?

It would prevent us from making optimizations that assume foreign key
constraints hold; which is a performance issue not a covert-channel
issue.
        regards, tom lane

On Tue, 2009-01-27 at 18:46 +0200, Heikki Linnakangas wrote:
> Joshua D. Drake wrote:
> > Lastly, the last time a developer told me two weeks it was 3 months.
> > Unless we get a written development plan that describes specifically
> > what, when, why and how long I am severely suspect that Heikki or Simon
> > have a clue on an actual deliverable time line (no offense guys).
> > Developers are generally overly optimistic about their delivery time
> > lines. 
> 
> I'm totally aware of that. My two weeks estimate really was with the 
> assumption that no new major issues crop up. I totally expect that new 
> major issues will crop up. But I'm going to give it a shot. I'm planning 
> to review the code anyway, whether it goes into 8.4 or 8.5, so I might 
> as well keep going and hope for the best.

Well I certainly can't argue with that :)

Sincerely,

Joshua D. Drake

-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

On Tue, 2009-01-27 at 11:36 -0500, Tom Lane wrote:
> David Fetter <david@fetter.org> writes:
> > On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote:
> >> I don't think this is correct.
> 
> > I do.
> 
> > People literally grab my shoulder and ask when we'll have it.
> 
> Do these people understand the difference between HS and a complete
> replication solution?  Are they still as excited after you explain
> the difference?

Yes, I think they do.

http://www.postgresql.org/community/survey.55
These people seem to understand also.

Sync rep *is* important, but it opens up new classes of applications for
us. As does SEP. Both of those are more speculative and harder to
measure, but we've seen big impact before from this type of new feature.

HS appeals to current users. Current users aren't so worried about new
applications, they look forward to being able to run queries on their
currently idle standby servers.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

"Joshua D. Drake" <jd@commandprompt.com> writes:

> This is my take as well. This is very real, very scary things that are
> being worked on. HS should only ship after a very, very long non change
> cycle (meaning no significant bugs (or refactoring) found in HS patch
> for X period of time)... say after a full 8.5 dev cycle. I do not want
> to commit this patch and then have to yank it out 3 months from now.

In general I'm for planning large features with the potential to break
existing functionality going in the beginning of cycles. I don't think that's
the same as "no changes" though. The reason we make changes is because they're
believed to be for the better. 

The point in my mind is to get more people playing with the new feature in
contexts that *aren't* expected by the developers. Developers are notoriously
bad at testing their work no matter how diligent they are they just don't
think of things they didn't anticipate when they're coding. (Which is only
logical -- surely they would have just written it right the first time if they
anticipated the problems...)

> Lastly, the last time a developer told me two weeks it was 3 months.
> Unless we get a written development plan that describes specifically
> what, when, why and how long I am severely suspect that Heikki or Simon
> have a clue on an actual deliverable time line (no offense guys).

Well, Simon's been pretty impressively bang-on with his estimates for his
*development* projects going back at least to async-commit.

The *review* process, however, is inherently hard to estimate though. I doubt
anyone will give Tom better than even odds on his side bet, even if that's our
best estimate.

Simon has been refactoring and recoding based on Heikki's suggestions as fast
as he's been proposing them though. It seems the question isn't how fast Simon
will get the work done so much as how many items we'll want to change before
committing it.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production
Tuning

On Tue, Jan 27, 2009 at 11:36:02AM -0500, Tom Lane wrote:
> David Fetter <david@fetter.org> writes:
> > On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote:
> >> I don't think this is correct.
> 
> > I do.
> 
> > People literally grab my shoulder and ask when we'll have it.
> 
> Do these people understand the difference between HS and a complete
> replication solution?

Yes, and those who don't catch on quickly.  The difference between
warm standby and hot is the difference between an idle resource which
only consumes money to mitigate risk and a revenue-generating one.
The former is for fat organizations with money to throw around, and
the latter is for anybody who needs to scale reads.

> Are they still as excited after you explain the difference?

Yes.

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Brendan Jurd <direvus@gmail.com> writes:
> I think the picture has started to become more clear during the 8.4
> dev cycle.  Most importantly, there was much ado made about the need
> for powerful email integration features in previous discussions.  This
> severely restricted our choices (possibly to zero?).  I feel that the
> commitfest wiki has demonstrated that no such integration is required.

Hardly --- one of the most critical usability fixes for the wiki was to
make it relatively painless to insert links to the mail list archives
(even for messages that hadn't made it there yet!).  We're still gonna
need that.  I agree that we found out that we don't need to be able to
send mail directly to the patch tracker, although perhaps cc'ing it
would be a nice way to get such links installed.

The other thing that is commonly thought of as "email integration"
is the ability to generate notification email, which AFAIK the wiki
does have (I haven't felt a need for it, but other people might be
using that).
        regards, tom lane

On Mon, 2009-01-26 at 22:55 -0500, Tom Lane wrote:
> Even accepting such a restriction, there's too much code in
> core Postgres to let anyone feel very good about keeping the core free
> of security leaks

I see what you're saying, but we're trying to pass certification, not
provide security in all cases.

The security policy & its implementation is part of the wall, so its
straightforward to say "don't do those things". Since both backups and
plugins are not typically managed by unprivileged users, that seems
reasonable. (And anyway, they should be using PITR :-).

I'd rather see it go in now. It needs to be audited, and it might fail.
If we put it in 8.5 and it still fails, we'll be in 8.6, which is far,
far away and we shouldn't expect NEC to fund such a long range mission.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

Tom Lane <tgl@sss.pgh.pa.us> writes:

> Robert Haas <robertmhaas@gmail.com> writes:
>>> Yeah, people like certification, but they also like products that work.
>>> Did you stop reading before getting to my non-security-based complaints?
>
>> I read them, but I suspect they are issues that can be addressed.  How
>> would any of this affect join removal, anyway?
>
> It would prevent us from making optimizations that assume foreign key
> constraints hold; which is a performance issue not a covert-channel
> issue.

It does seem weird to simply omit records rather than throw an error and
require the user to use a where clause, even if it's something like WHERE
pg_accessible(tab).

I wonder if we need a special kind of relational integrity trigger which
requires that the privileges on a source row be a superset of the privileges
on the target row. 

Can you even test "superset" on these privileges? Or are they too general for
that? And would you have trouble adjusting the privileges later because giving
someone access to a label would require checking every row to see if they have
access to every referenced row too?

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support!

On Tue, 2009-01-27 at 11:49 -0500, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> >> Yeah, people like certification, but they also like products that work.
> >> Did you stop reading before getting to my non-security-based complaints?
> 
> > I read them, but I suspect they are issues that can be addressed.  How
> > would any of this affect join removal, anyway?
> 
> It would prevent us from making optimizations that assume foreign key
> constraints hold; which is a performance issue not a covert-channel
> issue.

Well, only when sepostgresql = on.

I would imagine Oracle has exactly the same issue.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

Dave Page <dpage@pgadmin.org> writes:
> Not basing our release schedule on our commitments to shareholders is
> an entirely different thing to treating sponsors of major features
> like crap by arbitrarily bouncing the patches they've paid to have
> properly developed within the community process with no good reason.

Nobody has suggested bouncing HS; there is only a debate about how soon
it's likely to be appliable.  Any company who imagined they had a
guarantee about it getting into 8.4 is simply misguided.

As for SEPostgres, I think that bouncing it entirely is quite a possible
outcome, but that's because there does not appear to be adequate
interest to justify taking on a major maintenance burden (and anyone who
thinks it won't be a major burden is equally misguided --- at the very
least it will be an endless source of bug reports that we'll be forced
to classify as security issues, with all the hoop-leaping that that
entails).  We are not bound to accept features that are only wanted by a
small number of users, no matter how badly those users want them.
        regards, tom lane

On Tue, Jan 27, 2009 at 5:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Dave Page <dpage@pgadmin.org> writes:
>> Not basing our release schedule on our commitments to shareholders is
>> an entirely different thing to treating sponsors of major features
>> like crap by arbitrarily bouncing the patches they've paid to have
>> properly developed within the community process with no good reason.
>
> Nobody has suggested bouncing HS; there is only a debate about how soon
> it's likely to be appliable.  Any company who imagined they had a
> guarantee about it getting into 8.4 is simply misguided.

I was complaining about it being bounced to 8.5 without proper review
as I've said elsewhere in one of these threads.

> As for SEPostgres, I think that bouncing it entirely is quite a possible
> outcome, but that's because there does not appear to be adequate
> interest to justify taking on a major maintenance burden (and anyone who
> thinks it won't be a major burden is equally misguided --- at the very
> least it will be an endless source of bug reports that we'll be forced
> to classify as security issues, with all the hoop-leaping that that
> entails).  We are not bound to accept features that are only wanted by a
> small number of users, no matter how badly those users want them.

That wouldn't be 'arbitrarily bounced' though, it would be with good
reason. And I have no problem with that.

-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

* Gregory Stark (stark@enterprisedb.com) wrote:
> It does seem weird to simply omit records rather than throw an error and
> require the user to use a where clause, even if it's something like WHERE
> pg_accessible(tab).

It is weird from an SQL perspective, I agree with you there.  On the
other hand, it's what the security community is looking for, and is
what's implemented by other databases (Oracle, SQL Server...) that
do row-level security and security labels.  Requiring a where clause
or you throw an error would certainly make porting applications that
depend on that mechanism somewhat difficult, and doesn't really seem
like it'd gain you all that much...
Stephen

On Tue, Jan 27, 2009 at 11:49 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>>> Yeah, people like certification, but they also like products that work.
>>> Did you stop reading before getting to my non-security-based complaints?
>
>> I read them, but I suspect they are issues that can be addressed.  How
>> would any of this affect join removal, anyway?
>
> It would prevent us from making optimizations that assume foreign key
> constraints hold; which is a performance issue not a covert-channel
> issue.

Oh, I see now.  That problem is going to be common to row-level DAC
and SE-PostgreSQL proper.  It would not surprise me if any sort of
row-level access control turns out to be bad for performance, but
mainly because the overhead of checking permissions on every tuple is
bound to cost something.  If some day we have join removal and it has
to be disabled when row-level access control is turned on, those users
will be no worse off than they are today: no join removal.

...Robert

Dave Page <dpage@pgadmin.org> writes:
> On Tue, Jan 27, 2009 at 5:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Nobody has suggested bouncing HS; there is only a debate about how soon
>> it's likely to be appliable.  Any company who imagined they had a
>> guarantee about it getting into 8.4 is simply misguided.

> I was complaining about it being bounced to 8.5 without proper review
> as I've said elsewhere in one of these threads.

Huh?  It's been getting "proper review" for three months now.  I don't
think I need to have studied the code in detail before I'm entitled to
point out that it's horribly late and is still getting revised like mad.
If it weren't something that people desperately want, it would have been
punted to 8.5 quite some time ago.
        regards, tom lane

Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Jan 27, 2009 at 11:49 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> It would prevent us from making optimizations that assume foreign key
>> constraints hold; which is a performance issue not a covert-channel
>> issue.

> Oh, I see now.  That problem is going to be common to row-level DAC
> and SE-PostgreSQL proper.  It would not surprise me if any sort of
> row-level access control turns out to be bad for performance, but
> mainly because the overhead of checking permissions on every tuple is
> bound to cost something.

Right, but you expect that to be a small and predictable cost, say in
the single-digits-percentage range.  Plan optimizations that
suddenly stop happening can cost you multiple orders of magnitude.
And you won't soothe people by telling them that obsolete versions of
Postgres would have been that slow all the time.
        regards, tom lane

On Tue, Jan 27, 2009 at 12:41:36PM -0500, Stephen Frost wrote:
> * Gregory Stark (stark@enterprisedb.com) wrote:
> > It does seem weird to simply omit records rather than throw an error and
> > require the user to use a where clause, even if it's something like WHERE
> > pg_accessible(tab).

[…]

> do row-level security and security labels.  Requiring a where clause
> or you throw an error would certainly make porting applications that
> depend on that mechanism somewhat difficult, and doesn't really seem
> like it'd gain you all that much...

Throwing an error would entail a side-channel leak that would not be
acceptable to the security community, I bet.  That said, I have
reservations, along the lines of Peter E's, that the early
design-level objections to the approach were never answered.  I
certainly never got any real answer to questions I asked, for what
it's worth.  

I will note that I tried to have a look at the literature on this
topic.  As I started to read, it became obvious that it was copious,
but pretty well-determined.  What bothered me most about the answers I
got was that there never seemed to be an answer to "please outline the
design principles" except for "it's what SE-Linux does".  The OS-level
control rules seemed to me to be totally foreign to the database
world, precisely because ACID is a problem in databases in a way it
isn't for filesystems under the traditional UNIX model.  I formed the
impression -- only an impression, mind, that there was a poor fit
between SE-Linux and database systems, and that the proponents had
decided that enough caulk (in the form of "don't do that") would seal
the gap.

I haven't (obviously) been paying much attention to this topic since,
but I detect something of the same sort of response in the recent
discussion.  Maybe the goal isn't explicit enough.  If the goal is
compliance with some set of well-defined tests, what are they?  If the
goal is buzzword compliance, what are the tests of that (to the extent
there ever are some)?  If the goal is just "security enhancement", I
confess that I am still unable to understand the definitions of
"security" and "enhancement" such that I think we have some
operationalization of what the patch is supposed to provide.

I know there are people who think this is cool.  I guess, if I were
running the circus, I'd want to know what's cool about it, and why.
Then maybe the project would be in a position to understand whether
that kind of cool is the way it wants to be.  But without a clear
problem statement, and a roadmap of how the patches solve the problem,
I'm at a loss.  And last I checked (which was, admittedly, not today),
the project pages didn't have that information.

A

-- 
Andrew Sullivan
ajs@crankycanuck.ca

Stephen Frost wrote:
> * Gregory Stark (stark@enterprisedb.com) wrote:
>> It does seem weird to simply omit records rather than throw an error and
>> require the user to use a where clause, even if it's something like WHERE
>> pg_accessible(tab).

The idea is for the level of informations security we're talking about, 
someone with limited permissions not only isn't allowed to know certain 
data, they're not allowed to know certain data *exists*.  Within the 
SELinux framework, this is accomplished by hiding files you don't have 
permission to see, not merely denying access to them.

The presumption is that if you know the data exists but can't access it 
directly, you'll use indirect methods to derive what it is.  But if you 
don't even know it exists, then you won't look for it.

There's a level above that which I don't think SEPostgres implements, 
which is data substitution, in which you see different data according to 
what security level you are.  While this may seem insane for a business 
application, for military-support applications it makes some sense.

--Josh

Josh Berkus <josh@agliodbs.com> writes:
> Stephen Frost wrote:
> It does seem weird to simply omit records rather than throw an error

> The presumption is that if you know the data exists but can't access it 
> directly, you'll use indirect methods to derive what it is.  But if you 
> don't even know it exists, then you won't look for it.

Right, which is why it's bad for something like a foreign key constraint
to expose the fact that the row does exist after all.

> There's a level above that which I don't think SEPostgres implements, 
> which is data substitution, in which you see different data according to 
> what security level you are.  While this may seem insane for a business 
> application, for military-support applications it makes some sense.

I think it might be possible to build such a thing using views, but I
agree that the patch doesn't give it to you for free.
        regards, tom lane

Tom,

> The other thing that is commonly thought of as "email integration"
> is the ability to generate notification email, which AFAIK the wiki
> does have

Um, no.  It doesn't, and really can't.

Notifying everyone who's updated the page isn't terribly useful.

--Josh

Josh Berkus <josh@agliodbs.com> writes:
> Tom,
>> The other thing that is commonly thought of as "email integration"
>> is the ability to generate notification email, which AFAIK the wiki
>> does have

> Um, no.  It doesn't, and really can't.

Oh.  What's that "watch this page" option do, then?
        regards, tom lane

Josh Berkus wrote:
> Stephen Frost wrote:
>> * Gregory Stark (stark@enterprisedb.com) wrote:
>>> It does seem weird to simply omit records rather than throw an error and
>>> require the user to use a where clause, even if it's something like 
>>> WHERE
>>> pg_accessible(tab).
> 
> The idea is for the level of informations security we're talking about, 
> someone with limited permissions not only isn't allowed to know certain 
> data, they're not allowed to know certain data *exists*.  Within the 
> SELinux framework, this is accomplished by hiding files you don't have 
> permission to see, not merely denying access to them.
> 

SELinux does not (and can not) hide the existence of files, as file names are 
part of the directory data and not part of the file data.

We can restrict read on the directory so you can't see any of the file names but 
this is not a complete solution as anyone with search/create perms on the 
directory can enumerate the file name namespace to discover existence of files.

We do not consider that a short coming, anyone who needs to hide existence of 
files needs to set up their directory structure to disallow read/search/create 
on the directories they aren't allowed to discover filenames in. 
Polyinstanciation can also address this issue.

> The presumption is that if you know the data exists but can't access it 
> directly, you'll use indirect methods to derive what it is.  But if you 
> don't even know it exists, then you won't look for it.
> 

There are ways to mitigate this in the current security model of sepostgres, 
namely using trusted stored procedures to fetch data you are unwilling to let 
someone query against directly (or indirectly). Just as with SELinux anyone who 
needs this kind of capability would have to correctly design their tables and 
stored procedures to address their security goals.

> There's a level above that which I don't think SEPostgres implements, 
> which is data substitution, in which you see different data according to 
> what security level you are.  While this may seem insane for a business 
> application, for military-support applications it makes some sense.
> 

Trusted stored procedures can handle the above requirement, either by fuzzing 
data (consider coordinates x,y are in a table but below top secret you can't 
have access to the exact coordinates, the trusted stored procedure can fetch and 
fuzz them sufficiently before delivering them to the client).

Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
>> Stephen Frost wrote:
>> It does seem weird to simply omit records rather than throw an error
> 
>> The presumption is that if you know the data exists but can't access it 
>> directly, you'll use indirect methods to derive what it is.  But if you 
>> don't even know it exists, then you won't look for it.
> 
> Right, which is why it's bad for something like a foreign key constraint
> to expose the fact that the row does exist after all.
> 

Once again, this is not an issue for us. We would much rather have a database 
that allows you to hide data from unauthorized clients using a mandatory policy 
than one that does nothing because you couldn't close some covert channels.

I'll repeat what I said in an earlier email, SELinux doesn't (and can't) address 
all covert channels in Linux, and that is fine as long as it is understood and 
documented (which is part of the evaluation process).


>> There's a level above that which I don't think SEPostgres implements, 
>> which is data substitution, in which you see different data according to 
>> what security level you are.  While this may seem insane for a business 
>> application, for military-support applications it makes some sense.
> 
> I think it might be possible to build such a thing using views, but I
> agree that the patch doesn't give it to you for free.
> 

Tom,

> Oh.  What's that "watch this page" option do, then?

Notifies you when anyone makes any change of any kind to *any* patch (or 
piece of text, for that matter) in the commitfest.  Including something 
like changing the number of patches assigned to an RRR.

My inability to systematically send reminder e-mails to submitters and 
reviewers -- or for that matter, even track when they were assigned or 
last updated -- has been a significant drag on the effectiveness of the 
commitfests.  Some patches stalled, and I missed them.

--Josh

Josh,

> We do not consider that a short coming, anyone who needs to hide 
> existence of files needs to set up their directory structure to disallow 
> read/search/create on the directories they aren't allowed to discover 
> filenames in. Polyinstanciation can also address this issue.

Hmmm.  Why try to hide individual rows in tables then?  That would seem 
not in keeping with the filesystem policies.

--Josh

Josh Berkus wrote:
> Josh,
> 
>> We do not consider that a short coming, anyone who needs to hide 
>> existence of files needs to set up their directory structure to 
>> disallow read/search/create on the directories they aren't allowed to 
>> discover filenames in. Polyinstanciation can also address this issue.
> 
> Hmmm.  Why try to hide individual rows in tables then?  That would seem 
> not in keeping with the filesystem policies.
> 

Because rows have data in them. It is analogous to not allowing the contents of 
the file to be visible. However, the primary key is still known to exist through 
various means, which is more analogous to the filename.

On Tue, 2009-01-27 at 16:01 +0200, Peter Eisentraut wrote:
> Updatable views is reverted.  I agree that we should reject the rest and 
> prepare a release.

BTree-GIN has been ready for committer review for quite some time. It
has been mostly-ready for much longer: the only real code change since
submission was a workaround to support numeric that I requested. It's
also a small patch, easy to review, and offers a nice benefit.

Kenneth Marshall's updated hash functions (separated mix()/final())
haven't had any code changes since Nov. 4, the code is tiny, and the
only thing we've been waiting on is proof that it doesn't hurt
randomness. Ken has put in the effort to show that. At least look at his
analysis, and see if you agree.

I think both of these deserve at least a glance by a committer before
bouncing them. There are more committers who can look at these features
than, say, Hot Standby or Sync Rep. Details on commitfest wiki for those
interested.

GIN Fast Insert is also ready for a committer review. I happen to like
this feature, and I think it is compelling for anyone using GIN (Teodor
posted some good performance results today). However, it has gone
through more code changes than the two patches above and it's more
complex, so if you think this deserves to be put off, that's
understandable. I think it would be worth a look for any committer not
on the critical path for release though.

Regards,Jeff Davis

Josh Berkus <josh@agliodbs.com> writes:
> Tom,
>> Oh.  What's that "watch this page" option do, then?

> Notifies you when anyone makes any change of any kind to *any* patch (or 
> piece of text, for that matter) in the commitfest.  Including something 
> like changing the number of patches assigned to an RRR.

Okay, so it does have notification ability and you do want that, you
just want it on a different granularity level.

> My inability to systematically send reminder e-mails to submitters and 
> reviewers -- or for that matter, even track when they were assigned or 
> last updated -- has been a significant drag on the effectiveness of the 
> commitfests.  Some patches stalled, and I missed them.

Agreed, that would be helpful, and the wiki doesn't help you with it.
So we want a patch tracker that can do that.
        regards, tom lane

On Tue, Jan 27, 2009 at 12:52 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Tue, Jan 27, 2009 at 11:49 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> It would prevent us from making optimizations that assume foreign key
>>> constraints hold; which is a performance issue not a covert-channel
>>> issue.
>
>> Oh, I see now.  That problem is going to be common to row-level DAC
>> and SE-PostgreSQL proper.  It would not surprise me if any sort of
>> row-level access control turns out to be bad for performance, but
>> mainly because the overhead of checking permissions on every tuple is
>> bound to cost something.
>
> Right, but you expect that to be a small and predictable cost, say in
> the single-digits-percentage range.  Plan optimizations that
> suddenly stop happening can cost you multiple orders of magnitude.
> And you won't soothe people by telling them that obsolete versions of
> Postgres would have been that slow all the time.

Well, look at it another way.  If we don't accept row-level security
into PostgreSQL, then people will have to implement it themselves.  In
fact, I currently have a real application that does exactly this.  The
row-filtering is done, in essence, by having the web application add
certain conditions to the WHERE clause of certain queries depending on
which user is making the request.  And if those WHERE clauses happen
to mention columns from table X, then table X won't be a candidate for
join removal.  The only difference is that the logic is in my app
rather than in the database itself.

To put that another way, row-level permissions are just another
attribute of a table that could potentially affect the query result,
and the impact of referring to that attribute will be exactly the same
as the impact of referring to any other attribute in that table.

...Robert

Andrew,

* Andrew Sullivan (ajs@crankycanuck.ca) wrote:
> Throwing an error would entail a side-channel leak that would not be
> acceptable to the security community, I bet.  That said, I have
> reservations, along the lines of Peter E's, that the early
> design-level objections to the approach were never answered.  I
> certainly never got any real answer to questions I asked, for what
> it's worth.

I've added Joshua Brindle (hope that's alright, Josh), since I'm not sure
if he's trying to follow this whole 'release planning' thread and your
concerns are regarding SE-PostgreSQL.  I hope he can help to address
them (below).

> I will note that I tried to have a look at the literature on this
> topic.  As I started to read, it became obvious that it was copious,
> but pretty well-determined.  What bothered me most about the answers I
> got was that there never seemed to be an answer to "please outline the
> design principles" except for "it's what SE-Linux does".  The OS-level
> control rules seemed to me to be totally foreign to the database
> world, precisely because ACID is a problem in databases in a way it
> isn't for filesystems under the traditional UNIX model.  I formed the
> impression -- only an impression, mind, that there was a poor fit
> between SE-Linux and database systems, and that the proponents had
> decided that enough caulk (in the form of "don't do that") would seal
> the gap.
>
> I haven't (obviously) been paying much attention to this topic since,
> but I detect something of the same sort of response in the recent
> discussion.  Maybe the goal isn't explicit enough.  If the goal is
> compliance with some set of well-defined tests, what are they?  If the
> goal is buzzword compliance, what are the tests of that (to the extent
> there ever are some)?  If the goal is just "security enhancement", I
> confess that I am still unable to understand the definitions of
> "security" and "enhancement" such that I think we have some
> operationalization of what the patch is supposed to provide.
>
> I know there are people who think this is cool.  I guess, if I were
> running the circus, I'd want to know what's cool about it, and why.
> Then maybe the project would be in a position to understand whether
> that kind of cool is the way it wants to be.  But without a clear
> problem statement, and a roadmap of how the patches solve the problem,
> I'm at a loss.  And last I checked (which was, admittedly, not today),
> the project pages didn't have that information.
Thanks,
    Stephen

Joshua Brindle <method@manicmethod.com> writes:
> Tom Lane wrote:
>> Right, which is why it's bad for something like a foreign key constraint
>> to expose the fact that the row does exist after all.

> Once again, this is not an issue for us.

Yes it is an issue; claiming it isn't doesn't make it so.  You just
stated, in effect, that you don't implement data hiding in the
filesystem because it would break standard Unix filesystem semantics.
How is that consistent with your opinion that it's okay to break
standard SQL semantics in order to implement data hiding in a database?

The question of whether there is a covert channel is only a small part
of my complaint here.  If it's the judgement of security experts that
that's an acceptable thing, that's fine, it's their turf.  But SQL
behavior is my turf, and I'm not happy with discarding fundamental
semantic properties.

Quoting from your other message:

> We do not consider that a short coming, anyone who needs to hide
> existence of files needs to set up their directory structure to
> disallow read/search/create on the directories they aren't allowed to
> discover filenames in.

This seems to me to be exactly parallel to deciding that SELinux should
control only table/column permissions within SQL; an approach that would
be enormously less controversial, less expensive, and more reliable than
what SEPostgres tries to do.
        regards, tom lane

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> This seems to me to be exactly parallel to deciding that SELinux should
> control only table/column permissions within SQL; an approach that would
> be enormously less controversial, less expensive, and more reliable than
> what SEPostgres tries to do.

While also ignoring a feature that is available, and used by these same
security communities, in other enterprise RDBMSs...

http://www.securityfocus.com/infocus/1743

http://www.microsoft.com/technet/prodtechnol/sql/2005/multisec.mspx

It's not codified in the SQL spec (yet..) that I saw, and maybe we could
seperate out the SE bits from the row-level bits, but I'm really not
sure I see the value in doing that..
Stephen

Stephen Frost wrote:
> * Gregory Stark (stark@enterprisedb.com) wrote:
>> It does seem weird to simply omit records rather than throw an error and
>> require the user to use a where clause, even if it's something like WHERE
>> pg_accessible(tab).
> 
> It is weird from an SQL perspective, I agree with you there.  On the
> other hand, it's what the security community is looking for, and is
> what's implemented by other databases (Oracle, SQL Server...) that
> do row-level security and security labels.  Requiring a where clause
> or you throw an error would certainly make porting applications that
> depend on that mechanism somewhat difficult, and doesn't really seem
> like it'd gain you all that much...


It seems to me that there are two different standards to which this feature
might be held.

Is the goal
 a) SEPostgres can provide useful rules to add security to some    specific applications so long as you're careful to
avoidcrafting    policies that produce bizarre behaviors (like avoiding restricing    access to foreign key data you
mightneed).   On the other hand it    gives you enough rope to hang yourself and produce weird results    that don't
makesense from a SQL standard point of view if you    aren't careful matching the SEPostgres rules with your apps.
 

or
 b) SEPostgreSQL should only give enough rope that you can not    craft rules that produce unexpected behavior from a
SQLpoint    of view; and that it would be bad if one can produce SEPostgres    policies that produce unexpected SQL
behavior.


It seems to me many of the security-enhanced products seem to
do the former; while it seems some of the objections to this
patch are more of the latter.

Tom Lane wrote:
> Joshua Brindle <method@manicmethod.com> writes:
>> Tom Lane wrote:
>>> Right, which is why it's bad for something like a foreign key constraint
>>> to expose the fact that the row does exist after all.
> 
>> Once again, this is not an issue for us.
> 
> Yes it is an issue; claiming it isn't doesn't make it so.  You just
> stated, in effect, that you don't implement data hiding in the
> filesystem because it would break standard Unix filesystem semantics.

We break plenty of standard semantics. Applications generally expect to be able 
to unlink files they can write to, to truncate or append. They expect 
read()/write() calls to succeed after a successful open(). SELinux is more fine 
grained than POSIX/UNIX is so some of these fundamental assumptions got broken.

In the end this created better/safer software, after some growing pains of course.

> How is that consistent with your opinion that it's okay to break
> standard SQL semantics in order to implement data hiding in a database?
> 

Please step me through how the SQL semantics are being broken.
From my perspective (I've used databases, written applications for them, etc 
but never hacked on a database server directly) this seems pretty straightforward.

There already exists the possibility that you won't be able to read/write one or 
more of the tables for which a foreign key constraint exists, how do you handle 
it now? Why isn't it appropriate to handle it the same way?

> The question of whether there is a covert channel is only a small part
> of my complaint here.  If it's the judgement of security experts that
> that's an acceptable thing, that's fine, it's their turf.  But SQL
> behavior is my turf, and I'm not happy with discarding fundamental
> semantic properties.
> 
> Quoting from your other message:
> 
>> We do not consider that a short coming, anyone who needs to hide
>> existence of files needs to set up their directory structure to
>> disallow read/search/create on the directories they aren't allowed to
>> discover filenames in.
> 
> This seems to me to be exactly parallel to deciding that SELinux should
> control only table/column permissions within SQL; an approach that would
> be enormously less controversial, less expensive, and more reliable than
> what SEPostgres tries to do.
> 

It isn't parallel though, in applications using the filesystem specifying 
directories where files go is fairly common. In database applications you expect 
all the data to be in the same table, having multiple tables with the same 
schema to try and separate out data that should be visible to some and not 
others isn't acceptable to most people and certainly wouldn't be compatible with 
many existing applications.

Further, we have the ability to use per-process namespaces/polyinstantiation on 
the filesystem. From what I've heard that is somewhere you are unwilling to go 
as a community for postgresql so the options are somewhat more limited.

I'm not trying to pretend that SELinux and sepostgresql are the same thing and 
always have to be treated the same. The trusted X work uses the same 
infrastructure and security policy language as sepostgres and selinux itself but 
noone is comparing windows and pointers to tables and rows.

Type enforcement is a flexible system. You identify the objects that are 
important to you in the object manager (postgresql in this case) and assign 
types to it and write policies around those types. The SELinux community came to 
a consensus quite a while back about what objects were important to us. I had 
assumed (perhaps foolishly) that this had been communicated and also agreed upon 
in the postgresql community.

If that wasn't the case I hope we can sit down and talk about the model to 
determine what objects are important to everyone and talk about the 
implementation details afterward.

On Tue, 2009-01-27 at 13:57 -0500, Joshua Brindle wrote:
> Josh Berkus wrote:
> > Josh,
> > 
> >> We do not consider that a short coming, anyone who needs to hide 
> >> existence of files needs to set up their directory structure to 
> >> disallow read/search/create on the directories they aren't allowed to 
> >> discover filenames in. Polyinstanciation can also address this issue.
> > 
> > Hmmm.  Why try to hide individual rows in tables then?  That would seem 
> > not in keeping with the filesystem policies.
> > 
> 
> Because rows have data in them. It is analogous to not allowing the contents of 
> the file to be visible. However, the primary key is still known to exist through 
> various means, which is more analogous to the filename.

Since most keys are likely to be non-meaningful IDs, its not going to
help you much.

And besides, all you have to do is reserve key ranges for different
security levels so there would never be any overlap.

So its not really even a difficult problem to get around.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

On Tue, Jan 27, 2009 at 1:50 PM, Jeff Davis <pgsql@j-davis.com> wrote:
> On Tue, 2009-01-27 at 16:01 +0200, Peter Eisentraut wrote:
>> Updatable views is reverted.  I agree that we should reject the rest and
>> prepare a release.
>
> BTree-GIN has been ready for committer review for quite some time. It
[...]
> Kenneth Marshall's updated hash functions (separated mix()/final())
[....]
> I think both of these deserve at least a glance by a committer before
> bouncing them.

While we're at it, I think the Ramon Lawrence/Bryce Cutt patch to
"Improve Performance of Multi-Batch Hash Join for Skewed Data Sets"
also deserves a look.

...Robert

Stephen Frost wrote:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> This seems to me to be exactly parallel to deciding that SELinux should
>> control only table/column permissions within SQL; an approach that would
>> be enormously less controversial, less expensive, and more reliable than
>> what SEPostgres tries to do.
> 
> While also ignoring a feature that is available, and used by these same
> security communities, in other enterprise RDBMSs...  
> 
> http://www.securityfocus.com/infocus/1743
> 
> http://www.microsoft.com/technet/prodtechnol/sql/2005/multisec.mspx
> 
> It's not codified in the SQL spec (yet..) that I saw, and maybe we could
> seperate out the SE bits from the row-level bits, but I'm really not
> sure I see the value in doing that..

They are separate. If you look at the patches you'll see a pgace part, this is 
where the core interfaces to the security backends, and you'll see a rowacl 
backend and an sepgsql backend.

Personally I'd like to see all of the access control moved out to use pgace, 
including the standard DAC permissions but I doubt that would never happen.

Tom Lane escribió:
> Josh Berkus <josh@agliodbs.com> writes:
> > Tom,
> >> The other thing that is commonly thought of as "email integration"
> >> is the ability to generate notification email, which AFAIK the wiki
> >> does have
> 
> > Um, no.  It doesn't, and really can't.
> 
> Oh.  What's that "watch this page" option do, then?

It allows you to click on a special "watched pages" link, which then
lists the last few revisions for each of them.  So it has two problems
-- one is the wrong granularity level, and the other is that it's "pull"
rather than "push".

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Ron Mayer wrote:
> Stephen Frost wrote:
>> * Gregory Stark (stark@enterprisedb.com) wrote:
>>> It does seem weird to simply omit records rather than throw an error and
>>> require the user to use a where clause, even if it's something like WHERE
>>> pg_accessible(tab).
>> It is weird from an SQL perspective, I agree with you there.  On the
>> other hand, it's what the security community is looking for, and is
>> what's implemented by other databases (Oracle, SQL Server...) that
>> do row-level security and security labels.  Requiring a where clause
>> or you throw an error would certainly make porting applications that
>> depend on that mechanism somewhat difficult, and doesn't really seem
>> like it'd gain you all that much...
> 
> 
> It seems to me that there are two different standards to which this feature
> might be held.
> 
> Is the goal
> 
>   a) SEPostgres can provide useful rules to add security to some
>      specific applications so long as you're careful to avoid crafting
>      policies that produce bizarre behaviors (like avoiding restricing
>      access to foreign key data you might need).   On the other hand it
>      gives you enough rope to hang yourself and produce weird results
>      that don't make sense from a SQL standard point of view if you
>      aren't careful matching the SEPostgres rules with your apps.
> 

This is what we like to call a flexible mandatory access control system, such as 
type enforcement. Give everything a type, write rules around the types. If you 
mess up your system may not boot (or your database queries might be strange).

In talking to KaiGai the rules should not allow you to create inconsistencies, 
however. If you delete a row with a foreign key constraint and you cannot delete 
the associated row in another table the whole operation should fail. We want to 
fail on the side of maintaining data integrity and consistency while still 
allowing you enough rope to have a fine grained, flexible access control system.

> or
> 
>   b) SEPostgreSQL should only give enough rope that you can not
>      craft rules that produce unexpected behavior from a SQL point
>      of view; and that it would be bad if one can produce SEPostgres
>      policies that produce unexpected SQL behavior.

SELinux (and associated security aware applications) tend to go the way of very 
fine grained and let the policy author decide how to use it. I've seen SELinux 
policies with as few as 3 types, to as many as 3000. It is up to the policy 
author to choose the granularity appropriate for their purpose. I'd hate to see 
arbitrary limits put on the policy because of applications that might now 
operate correctly if they get an unexpected error (the applications should be 
fixed, not the security policy crippled)

> 
> It seems to me many of the security-enhanced products seem to
> do the former; while it seems some of the objections to this
> patch are more of the latter.
> 

Historically trusted systems were fairly course grained, with only Bell and 
LaPadula policies[1]. SELinux is probably the first fine grained mandatory 
access control system in the mainstream.


1 http://en.wikipedia.org/wiki/Bell-LaPadula_model

On Tue, Jan 27, 2009 at 2:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Joshua Brindle <method@manicmethod.com> writes:
>> Tom Lane wrote:
>>> Right, which is why it's bad for something like a foreign key constraint
>>> to expose the fact that the row does exist after all.
>
>> Once again, this is not an issue for us.
>
> Yes it is an issue; claiming it isn't doesn't make it so.  You just
> stated, in effect, that you don't implement data hiding in the
> filesystem because it would break standard Unix filesystem semantics.
> How is that consistent with your opinion that it's okay to break
> standard SQL semantics in order to implement data hiding in a database?

I think you're being pedantic.  There are different levels of breakage
and it is a matter of finding one that produces an acceptable
cost-benefit trade-off.

ISTM that we have plenty of evidence on these threads that other
databases do things in a way that is similar to what SE-PostgreSQL is
proposing to do.  If people are using the feature in Oracle and
getting value out of it, why should it suddenly become useless when
ported to PostgreSQL?

BTW, Oracle also has join removal, which proves that it isn't
impossible for a high-quality database product to support both
features.

...Robert

On Tue, 2009-01-27 at 14:18 -0500, Tom Lane wrote:
> Joshua Brindle <method@manicmethod.com> writes:
> > Tom Lane wrote:
> >> Right, which is why it's bad for something like a foreign key constraint
> >> to expose the fact that the row does exist after all.
> 
> > Once again, this is not an issue for us.
> 
> Yes it is an issue

> The question of whether there is a covert channel is only a small part
> of my complaint here.  If it's the judgement of security experts that
> that's an acceptable thing, that's fine, it's their turf.  But SQL
> behavior is my turf, and I'm not happy with discarding fundamental
> semantic properties.

Why did we bother to invite Joshua here if we aren't going to listen to
him?

Thanks for coming to help Joshua, much appreciated.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Jan 27, 2009 at 12:52 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Right, but you expect that to be a small and predictable cost, say in
>> the single-digits-percentage range.  Plan optimizations that
>> suddenly stop happening can cost you multiple orders of magnitude.

> Well, look at it another way.  If we don't accept row-level security
> into PostgreSQL, then people will have to implement it themselves.  In
> fact, I currently have a real application that does exactly this.  The
> row-filtering is done, in essence, by having the web application add
> certain conditions to the WHERE clause of certain queries depending on
> which user is making the request.  And if those WHERE clauses happen
> to mention columns from table X, then table X won't be a candidate for
> join removal.  The only difference is that the logic is in my app
> rather than in the database itself.

> To put that another way, row-level permissions are just another
> attribute of a table that could potentially affect the query result,
> and the impact of referring to that attribute will be exactly the same
> as the impact of referring to any other attribute in that table.

The flaw in that argument is that as you are doing it, the
de-optimization only happens on queries that actually need the behavior.
As the SEPostgres patch is constructed, the planner could *never* trust
an FK for optimization since it would have no way to know whether row
level permissions might be present (perhaps only for some rows) at
execution time.  You could only get back the optimization in builds with
SEPostgres compiled out.  That's pretty nasty, especially for packagers
who have to decide which build setting will displease fewer users.
        regards, tom lane

Tom Lane wrote:
>> We do not consider that a short coming, anyone who needs to hide
>> existence of files needs to set up their directory structure to
>> disallow read/search/create on the directories they aren't allowed to
>> discover filenames in.
> 
> This seems to me to be exactly parallel to deciding that SELinux should
> control only table/column permissions within SQL; an approach that would
> be enormously less controversial, less expensive, and more reliable than
> what SEPostgres tries to do.

With the table/column approach, could users who needed some row-level
capabilities work around this easily by setting table-level access
control on partitions?

In some ways that seems like it'd be easier to manage as well.

* Joshua Brindle (method@manicmethod.com) wrote:
> They are separate. If you look at the patches you'll see a pgace part,
> this is where the core interfaces to the security backends, and you'll
> see a rowacl backend and an sepgsql backend.

Right, guess it wasn't clear to me that the PGACE bits for row-level
access control could be used independently of SELinux (and maybe even on
systems that don't have SELinux..?).

> Personally I'd like to see all of the access control moved out to use
> pgace, including the standard DAC permissions but I doubt that would
> never happen.

Yeah...  That's a whole 'nother discussion.
Stephen

Stephen Frost wrote:
> * Joshua Brindle (method@manicmethod.com) wrote:
>> They are separate. If you look at the patches you'll see a pgace part, 
>> this is where the core interfaces to the security backends, and you'll 
>> see a rowacl backend and an sepgsql backend.
> 
> Right, guess it wasn't clear to me that the PGACE bits for row-level
> access control could be used independently of SELinux (and maybe even on
> systems that don't have SELinux..?).
> 

Sure, if you look at pgaceHooks.c you'll see:

bool
pgaceExecScan(Scan *scan, Relation rel, TupleTableSlot *slot)
{        /* Hardwired DAC checks */        if (!rowaclExecScan(scan, rel, slot))                return false;
        switch (pgace_feature)        {
#ifdef HAVE_SELINUX        case PGACE_FEATURE_SELINUX:                if (sepgsqlIsEnabled())
returnsepgsqlExecScan(scan, rel, slot);                break;
 
#endif        default:                break;        }        return true;
}

Notice the rowacl call outside of the HAVE_SELINUX ifdefs

Ron Mayer <rm_pg@cheapcomplexdevices.com> writes:
> It seems to me that there are two different standards to which this feature
> might be held.

> Is the goal
>   a) SEPostgres can provide useful rules to add security to some
>      specific applications so long as you're careful to avoid crafting
>      policies that produce bizarre behaviors (like avoiding restricing
>      access to foreign key data you might need).   On the other hand it
>      gives you enough rope to hang yourself and produce weird results
>      that don't make sense from a SQL standard point of view if you
>      aren't careful matching the SEPostgres rules with your apps.

> or
>   b) SEPostgreSQL should only give enough rope that you can not
>      craft rules that produce unexpected behavior from a SQL point
>      of view; and that it would be bad if one can produce SEPostgres
>      policies that produce unexpected SQL behavior.

With my other hat on (the red one) what I'm concerned about is whether
this patch will ever produce a feature that I could turn on in the
standard Red Hat/Fedora build of Postgres.  Right at the moment it seems
that the potential performance hit, for users who are *not using*
SEPostgres but merely have to use a build in which it is present,
might be bad enough to guarantee that that will never happen.
        regards, tom lane

> The flaw in that argument is that as you are doing it, the
> de-optimization only happens on queries that actually need the behavior.
> As the SEPostgres patch is constructed, the planner could *never* trust
> an FK for optimization since it would have no way to know whether row
> level permissions might be present (perhaps only for some rows) at
> execution time.  You could only get back the optimization in builds with
> SEPostgres compiled out.  That's pretty nasty, especially for packagers
> who have to decide which build setting will displease fewer users.

OK, I think I am starting to understand your concern now.

My understanding of how the world works is SE-PostgreSQL would always
be compiled in but could be turned off at run-time with a GUC.  I know
that the original design called for a compile-time switch, but
everyone hated it and I am pretty sure KaiGai changed it.  If he
hasn't, he will.  :-)

There was also talk of having a table-level option to include/exclude
the security ID (I'm not sure if it's currently implemented that way).Obviously that wouldn't be relevant for row-level
MAC(because
 
presumably you would need/want that turned on for all tables) but it
would be very relevant for row-level DAC (because it's easy, at least
for me, to imagine that you would only turn this on for a subset,
possibly quite a small subset, of your tables where you knew that it
was really needed).

If, by default, we make sepostgresql disabled, MAC security IDs on
newly created tables off, and DAC security IDs on newly created tables
off, then the pain will be confined to people who explicitly request
sepostgresql or row-level DAC.

...Robert

Simon Riggs <simon@2ndQuadrant.com> writes:
> On Tue, 2009-01-27 at 13:57 -0500, Joshua Brindle wrote:
>> Josh Berkus wrote:
>>> Hmmm.  Why try to hide individual rows in tables then?  That would seem 
>>> not in keeping with the filesystem policies.
>> 
>> Because rows have data in them. It is analogous to not allowing the contents of 
>> the file to be visible. However, the primary key is still known to exist through 
>> various means, which is more analogous to the filename.

> Since most keys are likely to be non-meaningful IDs, its not going to
> help you much.

Even more to the point: if the expectation is that you can hide a row's
data payload but not its primary key, you can accomplish that with
column-level permissions, without having to get into any non-standard
or even faintly surprising SQL behavior,
        regards, tom lane

Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Jan 27, 2009 at 1:50 PM, Jeff Davis <pgsql@j-davis.com> wrote:
>> I think both of these deserve at least a glance by a committer before
>> bouncing them.

> While we're at it, I think the Ramon Lawrence/Bryce Cutt patch to
> "Improve Performance of Multi-Batch Hash Join for Skewed Data Sets"
> also deserves a look.

There is no proposal or intention to bounce any of the remaining
commitfest items without consideration.  SEPostgres and Hot Standby
are the ones under debate.
        regards, tom lane

Tom Lane wrote:
> Simon Riggs <simon@2ndQuadrant.com> writes:
>> On Tue, 2009-01-27 at 13:57 -0500, Joshua Brindle wrote:
>>> Josh Berkus wrote:
>>>> Hmmm.  Why try to hide individual rows in tables then?  That would seem 
>>>> not in keeping with the filesystem policies.
>>> Because rows have data in them. It is analogous to not allowing the contents of 
>>> the file to be visible. However, the primary key is still known to exist through 
>>> various means, which is more analogous to the filename.
> 
>> Since most keys are likely to be non-meaningful IDs, its not going to
>> help you much.
> 
> Even more to the point: if the expectation is that you can hide a row's
> data payload but not its primary key, you can accomplish that with
> column-level permissions, without having to get into any non-standard
> or even faintly surprising SQL behavior,
> 

We aren't saying we want to hide the payload of the data in an entire column, 
just the data in some of the rows. For example, if you have top secret and 
secret data in the same table a secret user would be able to see the entire row 
for secret rows but maybe only some of the data on the top secret rows (or maybe 
not see the rows at all).

Further, the top secret rows may have some fields that are inaccessible but are 
accessible through a trusted stored procedure that does fuzzing on the data 
(back to the coordinates example I used earlier)

On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote:
> Such app already exists:
>
>   http://ozlabs.org/~jk/projects/patchwork/
>
> So it's a matter of just setting it up.

I was in fact in the process of setting that up just now. :-)

On Tuesday 27 January 2009 16:48:21 Sam Mason wrote:
> > though at EAL1 they're quite far from the EAL4+ that DB2,
> > Oracle, etc get.
>
> As far as I understand, the different levels are about assuring a
> set of code/features to some assurance level.  The Wikipedia page[1]
> gives a reasonable overview of the levels, but basically EAL1 says
> that a limited amount of effort (in practical terms, several person
> months/years of time for something like PG) was put in,

EAL1 pretty much means, a test suite exists and passes.  So that was easy. :-)

On Tuesday 27 January 2009 16:36:50 Stephen Frost wrote:
> * Peter Eisentraut (peter_e@gmx.net) wrote:
> > As one of the earlier reviewers, I think the design is OK, but the way
> > the implementation is presented was not acceptable, and very little has
> > been accomplished in terms of reacting to our comments.  For example,
> > where is the SQL row security feature, which should have been designed,
> > implemented, and committed separately, in the opinion of most
> > commentaries.
>
> Eh?  Are you thinking of column-level privileges, which was committed
> last week?

No.

> The SQL spec doesn't define row-level security, and coming 
> up with something willy-nilly on our own doesn't really strike me as the
> best approach.

Exactly.  But there is plenty of discussion on that elsewhere.

Ron Mayer <rm_pg@cheapcomplexdevices.com> writes:
> Tom Lane wrote:
>> This seems to me to be exactly parallel to deciding that SELinux should
>> control only table/column permissions within SQL; an approach that would
>> be enormously less controversial, less expensive, and more reliable than
>> what SEPostgres tries to do.

> With the table/column approach, could users who needed some row-level
> capabilities work around this easily by setting table-level access
> control on partitions?

Yeah, the same thing had just occurred to me.  We currently throw an
error if a user doesn't have permissions on every partition (child
table), but perhaps that behavior could be adjusted.  Ignoring
unreadable children would provide behavior pretty similar to that
proposed by SEPostgres.

To some extent that just postpones the semantic pain until the day when
we try to do unique and FK constraints that span partitions.  However,
I think (after only minimal thought) that that will only re-introduce
the covert-channel issue, which Joshua has already stated to be
acceptable.
        regards, tom lane

Tom Lane wrote:
> Ron Mayer <rm_pg@cheapcomplexdevices.com> writes:
>> It seems to me that there are two different standards to which this feature
>> might be held.
> 
>> Is the goal
>>   a) SEPostgres can provide useful rules to add security to some
>>      specific applications so long as you're careful to avoid crafting
>>      policies that produce bizarre behaviors (like avoiding restricing
>>      access to foreign key data you might need).   On the other hand it
>>      gives you enough rope to hang yourself and produce weird results
>>      that don't make sense from a SQL standard point of view if you
>>      aren't careful matching the SEPostgres rules with your apps.
> 
>> or
>>   b) SEPostgreSQL should only give enough rope that you can not
>>      craft rules that produce unexpected behavior from a SQL point
>>      of view; and that it would be bad if one can produce SEPostgres
>>      policies that produce unexpected SQL behavior.
> 
> With my other hat on (the red one) what I'm concerned about is whether
> this patch will ever produce a feature that I could turn on in the
> standard Red Hat/Fedora build of Postgres.  Right at the moment it seems
> that the potential performance hit, for users who are *not using*
> SEPostgres but merely have to use a build in which it is present,
> might be bad enough to guarantee that that will never happen.
> 

According to the comments in security/sepgsql/core.c:


/* * sepgsqlIsEnabled * * This function returns the state of SE-PostgreSQL when PGACE hooks * are invoked, to prevent
tocall sepgsqlXXXX() functions when * SE-PostgreSQL is disabled. * * We can config the state of SE-PostgreSQL in
$PGDATA/postgresql.conf.* The GUC option "sepostgresql" can have the following four parameter. * * - default    : It
alwaysfollows the in-kernel SELinux state. When it *                works in Enforcing mode, SE-PostgreSQL also works
in*                Enforcing mode. Changes of in-kernel state are delivered *                to userspace SE-PostgreSQL
soon,and SELinux state *                monitoring process updates it rapidly. * - enforcing  : It always works in
Enforcingmode. In-kernel SELinux *                has to be enabled. * - permissive : It always works in Permissive
mode.In-kernel SELinux *                has to be enabled. * - disabled   : It disables SE-PostgreSQL feature. It works
asif *                original PostgreSQL */
 


and in the hooks there is a pgace_feature that turns off the checks:

void
pgaceGramAlterRelation(Relation rel, HeapTuple tuple, DefElem *defel)
{        switch (pgace_feature)        {
#ifdef HAVE_SELINUX        case PGACE_FEATURE_SELINUX:                if (sepgsqlIsEnabled())                {
             sepgsqlGramAlterRelation(rel, tuple, defel);                        return;                }
break;
 
#endif        default:                break;        }
        if (defel)                ereport(ERROR,                                (errcode(ERRCODE_PGACE_ERROR),
                      errmsg("unable to set security attribute of 
 
table "                                                "via ALTER TABLE")));
}


So the pgace_feature turns off the backend call, there is an extra function 
call, and a branch but that shouldn't cause the kind of performance degradation 
you are talking about.

On Tuesday 27 January 2009 19:10:40 Gregory Stark wrote:
> It does seem weird to simply omit records rather than throw an error and
> require the user to use a where clause, even if it's something like WHERE
> pg_accessible(tab).

Not to pick on you personally, but this is the kind of review that should have 
happened six months ago, not during a "why is our development process 
inadequate" discussion on the eve of beta.

Peter Eisentraut <peter_e@gmx.net> writes:
> Not to pick on you personally, but this is the kind of review that should have 
> happened six months ago, not during a "why is our development process 
> inadequate" discussion on the eve of beta.

Right now, today, in this thread, is the first time that we've had any
opportunity to debate the design of SEPostgres with knowledgeable people
other than KaiGai-san.  It would likely be better if we started a new
thread with a more appropriate title, but I see nothing wrong with
asking pretty fundamental questions.
        regards, tom lane

* Peter Eisentraut (peter_e@gmx.net) wrote:
> > The SQL spec doesn't define row-level security, and coming
> > up with something willy-nilly on our own doesn't really strike me as the
> > best approach.
>
> Exactly.  But there is plenty of discussion on that elsewhere.

That's the nice thing about the SE-PostgreSQL patch..  It's at least
following established row-level security setups in other enterprise
RDBMSs...  The folks coming out now and saying we should require using a
WHERE clause or similar would cause a serious deviation from what's
being done today, without any real change or advantage that I see..
Thanks,
    Stephen

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > Not to pick on you personally, but this is the kind of review that should have
> > happened six months ago, not during a "why is our development process
> > inadequate" discussion on the eve of beta.
>
> Right now, today, in this thread, is the first time that we've had any
> opportunity to debate the design of SEPostgres with knowledgeable people
> other than KaiGai-san.  It would likely be better if we started a new
> thread with a more appropriate title, but I see nothing wrong with
> asking pretty fundamental questions.

I agree with asking the questions, but I don't like the immediate
assumption that we're going to have to kick the patch because someone
asked a question or suggested an alternative design unless we actively
decide that's the approach we want to go and it requires a serious
rework of the patch.

Personally, I think it'd be terrible to implement the suggestion that
started this sub-thread since it breaks with what is currently done
elsewhere and what the users of this feature would expect.
Thanks,
    Stephen

Peter Eisentraut <peter_e@gmx.net> writes:
> On Tuesday 27 January 2009 16:36:50 Stephen Frost wrote:
>> The SQL spec doesn't define row-level security, and coming 
>> up with something willy-nilly on our own doesn't really strike me as the
>> best approach.

> Exactly.  But there is plenty of discussion on that elsewhere.

BTW, whilst we are being beat about the head and shoulders with how
Oracle et al already have features like this, it is entirely appropriate
to wonder how come it's not in the standard.  Those companies surely
pretty much control the standards committee, and they have managed to
push a ton of rather dubious things into the last couple of SQL updates.
If row-level security is such a mess that they couldn't standardize it,
that tells me something.
        regards, tom lane

Stephen Frost <sfrost@snowman.net> writes:
> Personally, I think it'd be terrible to implement the suggestion that
> started this sub-thread since it breaks with what is currently done
> elsewhere and what the users of this feature would expect.

Upthread we were being told that this patch breaks new ground and will
offer capability available nowhere else.  Now I'm hearing that it's just
a "me too" patch to catch up with capability already available from N
commercial vendors.  Which is it?
        regards, tom lane

On 1/27/09, Peter Eisentraut <peter_e@gmx.net> wrote:
> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote:
>  > Such app already exists:
>  >
>  >   http://ozlabs.org/~jk/projects/patchwork/
>  >
>  > So it's a matter of just setting it up.
>
>
> I was in fact in the process of setting that up just now. :-)

Nice to know. :)   I feel that even if we decide to do our own
solution it would be good to try existing solution first.

-- 
marko

On Tue, 2009-01-27 at 15:09 -0500, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Tue, Jan 27, 2009 at 1:50 PM, Jeff Davis <pgsql@j-davis.com> wrote:
> >> I think both of these deserve at least a glance by a committer before
> >> bouncing them.
> 
> > While we're at it, I think the Ramon Lawrence/Bryce Cutt patch to
> > "Improve Performance of Multi-Batch Hash Join for Skewed Data Sets"
> > also deserves a look.
> 
> There is no proposal or intention to bounce any of the remaining
> commitfest items without consideration.  SEPostgres and Hot Standby
> are the ones under debate.
> 

I must have misunderstood Peter's comment.

Regards,Jeff Davis

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> BTW, whilst we are being beat about the head and shoulders with how
> Oracle et al already have features like this, it is entirely appropriate
> to wonder how come it's not in the standard.  Those companies surely
> pretty much control the standards committee, and they have managed to
> push a ton of rather dubious things into the last couple of SQL updates.
> If row-level security is such a mess that they couldn't standardize it,
> that tells me something.

For my 2c, for whatever it's worth, it's a combination of a specialized
user base and the fact that this kind of security used to only be in a
seperate product (eg: Trusted Oracle).  Perhaps it will be in the
standard some day, I don't think it's unreasonable to think that.
Certainly it'd be nice if it was already there.
Thanks,
    Stephen

Tom Lane wrote:
> Stephen Frost <sfrost@snowman.net> writes:
>> Personally, I think it'd be terrible to implement the suggestion that
>> started this sub-thread since it breaks with what is currently done
>> elsewhere and what the users of this feature would expect.
> 
> Upthread we were being told that this patch breaks new ground and will
> offer capability available nowhere else.  Now I'm hearing that it's just
> a "me too" patch to catch up with capability already available from N
> commercial vendors.  Which is it?
> 

It is like the difference between Trusted Solaris (really all the old trusted 
OS's) and SELinux. They both implement mandatory access control and both 
implement Bell and LaPadula as needed by the government/military but SELinux, 
via type enforcement, goes further to provide a completely flexible mandatory 
access control system.

SELinux is useful to meet all sorts of security goals, from system and 
application integrity to data pipelining and confidentiality. The SELinux 
community believes this sort of access control is important to not only the 
military but commercial and even small scale systems.

Further, because sepostgresql integrates well with SELinux the same system wide 
access controls flow seamlessly into the database. Are you able to access secret 
data on the filesystem? If so you'll be able to access secret data in the 
database. Are you able to update accounting information in the filesystem? Then 
you'll be able to update accounting information in the database.

This also integrates with KaiGai's other work to SELinux-ize apache so that an 
apache server can run a user script from a users home directory and a type 
transition occurs to run the script in the appropriate domain for that user, 
then when that script accesses the database they'll have only the access that 
users script should have.

This kind of end-to-end integration with mandatory access control is certainly 
ground breaking and isn't just the same ol' same ol' that other database vendors 
are doing (and have been doing for quite some time).

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > Personally, I think it'd be terrible to implement the suggestion that
> > started this sub-thread since it breaks with what is currently done
> > elsewhere and what the users of this feature would expect.
>
> Upthread we were being told that this patch breaks new ground and will
> offer capability available nowhere else.  Now I'm hearing that it's just
> a "me too" patch to catch up with capability already available from N
> commercial vendors.  Which is it?

argh, it's a combination, in the end.  Oracle and SQL Server offer row
level security, that's something we don't have today and is provided
through PGACE and is a big piece of the security labels/context part of
the high security RDBMS world.  Neither of them (far as I know..)
interoperate with a OS-level policy system to provide that additional
integration with the rest of the system as a whole (the SE-Linux bits).

I wasn't sure how easy they were to seperate and to use seperately.  It
looks like they can be used independently, which is great, and means you
could implement row level security on a BSD platform, but you wouldn't
get the integration with the OS policy unless you hooked in with the
Trusted BSD system (which I think actually can be done through an
SE-Linux userland port.. but I've never played with it).
Thanks,
    Stephen

On Tue, 2009-01-27 at 15:03 -0500, Tom Lane wrote:
>
> With my other hat on (the red one) what I'm concerned about is whether
> this patch will ever produce a feature that I could turn on in the
> standard Red Hat/Fedora build of Postgres.

FWIW, as you know, sepostgresql is already included in Fedora. You can
continue shipping it as a seperate RPM set.
--
Devrim GÜNDÜZ, RHCE
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr                  http://www.gunduz.org

Devrim GÜNDÜZ wrote:
> On Tue, 2009-01-27 at 15:03 -0500, Tom Lane wrote:
>> With my other hat on (the red one) what I'm concerned about is whether
>> this patch will ever produce a feature that I could turn on in the
>> standard Red Hat/Fedora build of Postgres. 
> 
> FWIW, as you know, sepostgresql is already included in Fedora. You can
> continue shipping it as a seperate RPM set.

That is non-ideal. Getting the capability in to the standard database shipped 
with RHEL is very important to me and my customers.

Since you can turn this off with GUC I don't see why it makes sense to ship 2 
databases (nevermind the maintenance issues)

* Devrim GÜNDÜZ (devrim@gunduz.org) wrote:
> On Tue, 2009-01-27 at 15:03 -0500, Tom Lane wrote:
> > With my other hat on (the red one) what I'm concerned about is whether
> > this patch will ever produce a feature that I could turn on in the
> > standard Red Hat/Fedora build of Postgres.
>
> FWIW, as you know, sepostgresql is already included in Fedora. You can
> continue shipping it as a seperate RPM set.

hrmpf, I wonder if that's why RH hasn't been as interested in pushing
for inclusion upstream...  Maybe the feel that's a workable solution?  I
wouldn't agree...  but I don't run RH.
Thanks,
    Stephen

On Tue, 2009-01-27 at 15:46 -0500, Tom Lane wrote:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > Not to pick on you personally, but this is the kind of review that should have 
> > happened six months ago, not during a "why is our development process 
> > inadequate" discussion on the eve of beta.
> 
> Right now, today, in this thread, is the first time that we've had any
> opportunity to debate the design of SEPostgres with knowledgeable people
> other than KaiGai-san.  It would likely be better if we started a new
> thread with a more appropriate title, but I see nothing wrong with
> asking pretty fundamental questions.

Except that Bruce and I already checked detailed documentation
references on this very topic months ago. Check with Bruce; he was
careful to point those things out to me, so I'm sure he'll do the same
for you. I'm satisfied, on this point.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

On Mon, Jan 26, 2009 at 08:28:32PM -0500, Tom Lane wrote:
> Hmm, you think selinux people read pgsql-announce?

Maybe not, but it got it onto LWN, which is a lot more people.

Have a nice day,
--
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Please line up in a tree and maintain the heap invariant while
> boarding. Thank you for flying nlogn airlines.

Joshua Brindle wrote:
>> FWIW, as you know, sepostgresql is already included in Fedora. You can
>> continue shipping it as a seperate RPM set.
> 
> That is non-ideal. Getting the capability in to the standard database
> shipped with RHEL is very important to me and my customers.

Could you speak - even in general terms - about who your customers are
and what kinds of needs (is row-level acls the most important to them?
mandantory access control at the table level?  both?) they have?

I'm guessing a better understanding of how real-world users would
use this feature would be enlightening.

> Since you can turn this off with GUC I don't see why it makes sense to
> ship 2 databases (nevermind the maintenance issues)

On Monday 26 January 2009 15:13:56 dpage@pgadmin.org wrote:
> On 1/26/09, Josh Berkus <josh@agliodbs.com> wrote:
> > All,
> >
> >>> 1) having the last CF on Nov. 1 was a mistake.  That put us square in
> >>> the path of the US & Christian holidays during the critical integration
> >>> phase ..
> >>> which means we haven't really had 3 months of integration, we've had
> >>> *two*.
> >
> > Actually, I'm thinking about this again, and made a mistake about the
> > mistake.  The *original plan* was that we were not going to accept any
> > new patches for Nov-CF.  Just revised patches from eariler Fests.  We
> > didn't stick to that, which is mostly why we are still reviewing now.
>
> I don't recall us discussing that, but it sounds like it might help next
> cycle.
>

What would be the significance of opening up the tree to future development 
between the last commitfest and last commitfest -1, if no new patches could 
be introduced?

Essentially this is where we are at now... November commit fest finished, 
December we "re-opened" for development, and we're in the January commitfest 
where no new features have been accepted.  

The problem is what to do when we get to the end of the commit fests, and we 
have a few reamining (invariably large/complex) patches that people don't 
want to push. 

I had been leaning toward the idea of pushing the 8.4 release back six months, 
but reopening development for 2-3 more development/commitfest cycles, but I 
am starting to think this is moving things in the wrong direction. 

Now I am starting to think that we cannot prevent large patches from showing 
up at the end of a cycle no matter what, and the only way to really "solve" 
that problem is to lesson the pain of getting bumped from a release. Ie. 
instead of being bump meaning you must wait 12-14 months till next release, 
we move toward more of a 6 month cycle of development.  I'm not sure it's 
feasible to boil down beta/rc phase to two months, tough it seems possible if 
we were strict about bumping features that aren't ready, and if our 
development cycles only went 4 months.  For end users who are concerned about 
continuous upgrading, we might think about putting restrictions on every 
other release (ie. require binary or data file level compatability with 8.4 
for the 8.5 release, and remove that restriction for 8.6) to lesson the 
upgrade path. (alternativly, a working IPU plan could make that less of an 
issue)

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

Marko Kreen wrote:
> On 1/27/09, Peter Eisentraut <peter_e@gmx.net> wrote:
>> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote:
>>  > Such app already exists:
>>  >
>>  >   http://ozlabs.org/~jk/projects/patchwork/
>>  >
>>  > So it's a matter of just setting it up.
>>
>>
>> I was in fact in the process of setting that up just now. :-)
> 
> Nice to know. :)   I feel that even if we decide to do our own
> solution it would be good to try existing solution first.

IIRC, we already installed and tried this a while ago. I don't remember
exactly what it failed on, but there was something pretty clear. But
maybe it's been fixed by now.

//Magnus

On Tuesday 27 January 2009 11:56:51 Simon Riggs wrote:
> On Tue, 2009-01-27 at 11:36 -0500, Tom Lane wrote:
> > David Fetter <david@fetter.org> writes:
> > > On Mon, Jan 26, 2009 at 03:12:02PM -0500, Tom Lane wrote:
> > >> I don't think this is correct.
> > >
> > > I do.
> > >
> > > People literally grab my shoulder and ask when we'll have it.
> >
> > Do these people understand the difference between HS and a complete
> > replication solution?  Are they still as excited after you explain
> > the difference?
>
> Yes, I think they do.
>
> http://www.postgresql.org/community/survey.55
> These people seem to understand also.
>
> Sync rep *is* important, but it opens up new classes of applications for
> us. As does SEP. Both of those are more speculative and harder to
> measure, but we've seen big impact before from this type of new feature.
>
> HS appeals to current users. Current users aren't so worried about new
> applications, they look forward to being able to run queries on their
> currently idle standby servers.
>

That's modest. I've talked to several oracle and db2 shops that want a standby 
for reporting that has relatively easy setup/maintenance (handling ddl is a 
big part of this) and the HS feature your working on will give them something 
as good as what they are getting now. So yeah, HS appeals to future users as 
well.  

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

> That's modest. I've talked to several oracle and db2 shops that want a standby 
> for reporting that has relatively easy setup/maintenance (handling ddl is a 
> big part of this) and the HS feature your working on will give them something 
> as good as what they are getting now. So yeah, HS appeals to future users as 
> well.  

I've talked to some of my clients, and while they *want* synch or 
near-synch HS, even slow HS is useful to them *now*.

One client is planning on deploying a rather complex FS cloning 
infrastructure just to have a bunch of reporting, testing and read-only 
search databases they need.  They'd be thrilled with an HS feature which 
produced DBs which were an hour out of date (or even 6 hours out of 
date), but ran read-only queries.

--Josh.

On 1/27/09 4:40 PM, "Ron Mayer" <rm_pg@cheapcomplexdevices.com> wrote:

> Joshua Brindle wrote:
>>> FWIW, as you know, sepostgresql is already included in Fedora. You can
>>> continue shipping it as a seperate RPM set.
>> 
>> That is non-ideal. Getting the capability in to the standard database
>> shipped with RHEL is very important to me and my customers.
> 
> Could you speak - even in general terms - about who your customers are
> and what kinds of needs (is row-level acls the most important to them?
> mandantory access control at the table level?  both?) they have?
> 
I'll speak to this a bit (Josh is also a Tresys employee). I can't say who
my customers are, but I can speak to their needs. They really need row-level
mandatory access controls (so both). I'll give a few examples that will
hopefully help here.

1) One customer had several networks, each with a different classification.
When a user needed to find anything, it was very painful as they would have
to log into each network one at a time to search for anything. What they
wanted to do was have a trusted database with a combined index of all the
networks. They wanted to be able to write a flexible policy that could grant
access to individual entries in the database based on what network the
request came from in addition to the security clearance of the user
requesting data.

2) Another example that we've been asked for repeatedly but had to turn away
as the capability did not yet exist is a trusted LAMP/LAPP stack (Note that
KaiGai is also working on the apache portion of this to provide a complete
trusted LAPP stack). Under this model, individual web scripts can be run
with a specific type. Combined with an SE-PostgreSQL policy, this gives the
ability to restrict what a specific script can access. Additionally, as
SE-PostgreSQL ties back in to the operating system mandatory access control
mechanism, we can tie the type of the script back to the type of the actual
user making the request. Coupled with labeled IPSec, this means we can
control access to data in the database based on the clearance or role (or
anything else you want to represent in their type) of the user on their end
system.

3) A customer wanted to implement an approval process that required several
steps. Without SE-PostgreSQL, we were forced to implement this by making
lots of copies. Stage 1 would approve the package and copy it into Stage 2's
inbox. We would grant each stage write access to the next stage's inbox in
order to enforce information flow. This was very expensive, and didn't scale
well. With SE-PostgreSQL, we could leave the data where it was and simply
relabel the row(s). Each stage would be granted the ability to relabel from
its type to the type of the next stage. No copies are necessary.

I hope those help. I realize that many of you may not be used to dealing
with customers who have such stringent security requirements, but if
SE-PostgreSQL gets merged, that could change.

Thanks,
Chad Sellers

On Tuesday 27 January 2009 10:34:59 Joshua D. Drake wrote:
> On Tue, 2009-01-27 at 06:40 +0100, Pavel Stehule wrote:
> > > 8.4-stable
> > > 8.4-experimental
> > >
> > > stable is everything that stable is. PostgreSQL at its best.
> >
> > I dislike this idea - it's same like short processed 8.5 -
>
> Actually it isn't because we wouldn't accept features into
> 8.4-experimental. The only thing we would accept into 8.4-experimental
> would be bug fixes that would automatically be ported up to 8.5 (or
> perhaps the other way around). We would still continue to build 8.5 as
> normal.
>
> > that is
> > more simple.
>
> We have tried the short release cycle before, it was called 8.2. It
> fails, remarkably.
>

I think this is a bit of revisionsit history. While I'd agree it didn't work, 
the cycle was shorter... or to put it another way, would you say that the 
commitfest model failed miserably? should we scrap that for next cycle?

I wonder... 

Feb 1 - all remaining patches bump / beta starts
March 15th - beta ends / rc starts(note, giving 3 months for beta/rc, since we had a longer dev round)  
May 1st - release 8.4, open 8.5 dev
June 1st - first 8.5 commitfest  (we don't worry about short May because we 
have built up patche queue)
July - second round of dev
August - second/final commitfest
Sept  - beta opens 
October - rc opens

November - release 8.5, open 8.6
December - first commitfest for 8.6  
Jan 2010 - second dev cycle
Feb    - final commitfest 
March - 8.6 beta
April - 8.6 rc

May 2010 - release 8.6
<rinse, repeat>

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

Robert Treat <xzilla@users.sourceforge.net> writes:
> Now I am starting to think that we cannot prevent large patches from showing 
> up at the end of a cycle no matter what, and the only way to really "solve" 
> that problem is to lesson the pain of getting bumped from a release. Ie. 
> instead of being bump meaning you must wait 12-14 months till next release, 
> we move toward more of a 6 month cycle of development.

I really can't see going in that direction.  In the first place, no one
wants to spend a third or more of the time in beta mode.  In the
second place, if the problem is big patches that take a long time to
develop, halving the length of the development cycle is no solution.
(If it did work, our plea to break large patches into segments landing
in different commitfests would have had more results.)  In the third
place, unless we get an upgrade-in-place process that works all the
time, we would be looking at maintaining twice as many back branches
in order to provide the same kind of release lifespan we have today.
We are at the limit of what we can realistically do in back-branch
maintenance already :-(
        regards, tom lane

On Tue, 27 Jan 2009, Chad Sellers wrote:

> I'll speak to this a bit (Josh is also a Tresys employee). I can't say who
> my customers are, but I can speak to their needs. They really need row-level
> mandatory access controls (so both).

From the perspective of what this would buy as far as this feature being a 
PostgreSQL advocacy point, one of the questions Tom asked about a bit 
upthread is still a bit hazy here.  There are commercial database 
offerings selling into the "trusted" space already.  While the use-cases 
you describe make perfect sense, I don't think it's clear to everyone yet 
if there's a unique draw to a PostgreSQL + selinux solution that the class 
of customers you're talking about would prefer it to purchasing one of 
those products.  Is the cost savings the main driver here, or is there 
something else about a secure LAPP stack that makes it particularly 
compelling?

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Robert Treat <xzilla@users.sourceforge.net> writes:
> On Tuesday 27 January 2009 10:34:59 Joshua D. Drake wrote:
>> We have tried the short release cycle before, it was called 8.2. It
>> fails, remarkably.

> I think this is a bit of revisionsit history.

JD got the release number wrong, it was 8.3, but otherwise there's no
revisionism involved:
http://archives.postgresql.org/pgsql-hackers/2006-12/msg00786.php

The theme that our release cycles are too long is not exactly new,
of course, eg
http://archives.postgresql.org/pgsql-hackers/2000-05/msg00574.php
http://archives.postgresql.org/pgsql-hackers/2001-06/msg00766.php
http://archives.postgresql.org/pgsql-hackers/2003-11/msg00889.php
but by now I think we've learned to stop banging our heads against
that particular rock.  One-year major cycles work for this project,
shorter ones are wishful thinking.
        regards, tom lane

On Tuesday 27 January 2009 18:51:01 Tom Lane wrote:
> Robert Treat <xzilla@users.sourceforge.net> writes:
> > Now I am starting to think that we cannot prevent large patches from
> > showing up at the end of a cycle no matter what, and the only way to
> > really "solve" that problem is to lesson the pain of getting bumped from
> > a release. Ie. instead of being bump meaning you must wait 12-14 months
> > till next release, we move toward more of a 6 month cycle of development.
>
> I really can't see going in that direction.  In the first place, no one
> wants to spend a third or more of the time in beta mode. 

Yeah, I was thinking that, but the truth is we do that now. We released last 
Febuary right? and we're looking at releasing (optimisttically) May 1st, 
right? So thats 15months, of which November - May (6 months) will have been 
feature freeze / beta / rc phase of development.   

> In the 
> second place, if the problem is big patches that take a long time to
> develop, halving the length of the development cycle is no solution.
> (If it did work, our plea to break large patches into segments landing
> in different commitfests would have had more results.) 

I think this is a mis-assesment of our problem. The problem is not that big 
patches take a long time; not so much that they don't, just that is not a 
problem we can solve... "Hey Simon, code faster!" is not going to work ;-) 

The problem is that the pain point is extremely high for missing a given 
release cycle. If you don't make a specific release, you have a 12-14 month 
wait for feature arrival. Given that choice, someone who deperately need (aka 
wants) HS/SEPostgres/Win32/HOT/IPU/etc... will likely be willing to push a 
release 3-6 months for that one feature. 

Incidentally, this is probably why people didnt worry about making a given 
commitfest. The pain point was low, so there was no percieved need to rework 
a patch for a specific commit, since there was another one just a couple 
months away. However, we still see a rush of patches at the final freeze 
because people know that "there is no tommorrow" at that point.  

> In the third 
> place, unless we get an upgrade-in-place process that works all the
> time, we would be looking at maintaining twice as many back branches
> in order to provide the same kind of release lifespan we have today.
> We are at the limit of what we can realistically do in back-branch
> maintenance already :-(
>

Yeah, I can't argue with that. I'm not sure it's an unsolvable problem though; 
if odd/even release maintenance doesn't sound good, we could do something 
like ubuntus LTS, where we pick 1 release every 3 years to make a long-term 
support commitment (I think 5 years is our current max), and keep other 
releases on a shorter lifespan (1 or 2 years). Certainly having IPU (or is 
that UIP?) would make that an easier decision.  
-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

Here is morning now, so I started to follow the discussion now...

Stephen Frost wrote:
> * Gregory Stark (stark@enterprisedb.com) wrote:
>> It does seem weird to simply omit records rather than throw an error and
>> require the user to use a where clause, even if it's something like WHERE
>> pg_accessible(tab).

It was an implementation of very earlier version of SE-PostgreSQL.
(Maybe, its revision number was still less than 500.)
It rewrites WHERE clause of given queries, but Tom suggested such
a query rewrite easily makes a bug and hard to maintain in the
future, so I removed the code and put a hook in ExecScan(), which
featch a tuple from relation and checks condition.
(I think it was a good suggestion. It also enables to reduce the
scale of SE-PostgreSQL patches.)

Indeed, it requires additional checks and disables a few kind of
optimization, when these enhanced-security features are activated.
However, I made clear some times that we assume security focused
users don't give their first priority on performance.

I can understand performance is a significant factor for database
management system, so the default of these features are *disabled*
unless user explicitly activate them.

> It is weird from an SQL perspective, I agree with you there.  On the
> other hand, it's what the security community is looking for, and is
> what's implemented by other databases (Oracle, SQL Server...) that
> do row-level security and security labels.  Requiring a where clause
> or you throw an error would certainly make porting applications that
> depend on that mechanism somewhat difficult, and doesn't really seem
> like it'd gain you all that much...

-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

On Tuesday 27 January 2009 19:04:49 Tom Lane wrote:
> Robert Treat <xzilla@users.sourceforge.net> writes:
> > On Tuesday 27 January 2009 10:34:59 Joshua D. Drake wrote:
> >> We have tried the short release cycle before, it was called 8.2. It
> >> fails, remarkably.
> >
> > I think this is a bit of revisionsit history.
>
> JD got the release number wrong, it was 8.3, but otherwise there's no
> revisionism involved:
> http://archives.postgresql.org/pgsql-hackers/2006-12/msg00786.php
>

The revisionism was that of "remarkable failure".  That was our shortest 
release cycle in the modern era. And it didn't have the advantage of the 
commitfest process. 

But I think what is important here is to recognize why it didn't work. Once 
again we ended up with large, complex features (HOT, tsearch) that people 
didn't want to wait 14 months to see if they missed the 8.3 release. And yes, 
most of these same arguements were raised then... "full text search is killer 
feature", "whole applications are waiting for in-core full text search", "hot 
will give allow existing customers to use postgres on a whole new 
level", "not fair to push back patches so long when developers followed the 
rules", "sponsors wont want to pay for features they wont see for 
years", "developers dont want to wait so long to see features committed", and 
on and on...  

The more I think about it, the more I feel that where we failed for 8.3 was 
not having a short 8.4 cycle lined up, which would give more freedom to bump 
patches to the next release. 

> The theme that our release cycles are too long is not exactly new,
> of course, eg
> http://archives.postgresql.org/pgsql-hackers/2000-05/msg00574.php
> http://archives.postgresql.org/pgsql-hackers/2001-06/msg00766.php
> http://archives.postgresql.org/pgsql-hackers/2003-11/msg00889.php


Yeah, I remember all that, and I think you'll find that I mostly was on the 
other side of this issue :-)

But many of the arguments from back then don't apply any more. Remember when 
you couldn't depend on pg_dump giving you dumps in the right order? Now that 
was a recipe for painful upgrades. With things like Slony, it's now possible 
to upgrade a majority of the Postgres installations with extremely minimal 
downtime. (And yes, I happen to have one that wont work with slony, but 
hey...)

> but by now I think we've learned to stop banging our heads against
> that particular rock.  One-year major cycles work for this project,
> shorter ones are wishful thinking.
>

Do they? 1 year cycles certainly don't solve the problem of being left with 
big/complex left over patches. They don't decrease the amount of time (as a 
percentage) we spend in freeze/beta/release mode. And we don't even get them 
out in 1 year; this 1 year cycle looks like at least 15 months. 

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

Joshua Brindle wrote:
> Tom Lane wrote:
>> Ron Mayer <rm_pg@cheapcomplexdevices.com> writes:
>>> It seems to me that there are two different standards to which this 
>>> feature
>>> might be held.
>>
>>> Is the goal
>>>   a) SEPostgres can provide useful rules to add security to some
>>>      specific applications so long as you're careful to avoid crafting
>>>      policies that produce bizarre behaviors (like avoiding restricing
>>>      access to foreign key data you might need).   On the other hand it
>>>      gives you enough rope to hang yourself and produce weird results
>>>      that don't make sense from a SQL standard point of view if you
>>>      aren't careful matching the SEPostgres rules with your apps.
>>
>>> or
>>>   b) SEPostgreSQL should only give enough rope that you can not
>>>      craft rules that produce unexpected behavior from a SQL point
>>>      of view; and that it would be bad if one can produce SEPostgres
>>>      policies that produce unexpected SQL behavior.
>>
>> With my other hat on (the red one) what I'm concerned about is whether
>> this patch will ever produce a feature that I could turn on in the
>> standard Red Hat/Fedora build of Postgres.  Right at the moment it seems
>> that the potential performance hit, for users who are *not using*
>> SEPostgres but merely have to use a build in which it is present,
>> might be bad enough to guarantee that that will never happen.
>>
> 
> According to the comments in security/sepgsql/core.c:
> 
> 
> /*
>  * sepgsqlIsEnabled
>  *
>  * This function returns the state of SE-PostgreSQL when PGACE hooks
>  * are invoked, to prevent to call sepgsqlXXXX() functions when
>  * SE-PostgreSQL is disabled.
>  *
>  * We can config the state of SE-PostgreSQL in $PGDATA/postgresql.conf.
>  * The GUC option "sepostgresql" can have the following four parameter.
>  *
>  * - default    : It always follows the in-kernel SELinux state. When it
>  *                works in Enforcing mode, SE-PostgreSQL also works in
>  *                Enforcing mode. Changes of in-kernel state are delivered
>  *                to userspace SE-PostgreSQL soon, and SELinux state
>  *                monitoring process updates it rapidly.
>  * - enforcing  : It always works in Enforcing mode. In-kernel SELinux
>  *                has to be enabled.
>  * - permissive : It always works in Permissive mode. In-kernel SELinux
>  *                has to be enabled.
>  * - disabled   : It disables SE-PostgreSQL feature. It works as if
>  *                original PostgreSQL
>  */
> 
> 
> and in the hooks there is a pgace_feature that turns off the checks:
> 
> void
> pgaceGramAlterRelation(Relation rel, HeapTuple tuple, DefElem *defel)
> {
>         switch (pgace_feature)
>         {
> #ifdef HAVE_SELINUX
>         case PGACE_FEATURE_SELINUX:
>                 if (sepgsqlIsEnabled())
>                 {
>                         sepgsqlGramAlterRelation(rel, tuple, defel);
>                         return;
>                 }
>                 break;
> #endif
>         default:
>                 break;
>         }
> 
>         if (defel)
>                 ereport(ERROR,
>                                 (errcode(ERRCODE_PGACE_ERROR),
>                                  errmsg("unable to set security 
> attribute of table "
>                                                 "via ALTER TABLE")));
> }
> 
> 
> So the pgace_feature turns off the backend call, there is an extra 
> function call, and a branch but that shouldn't cause the kind of 
> performance degradation you are talking about.

This hook is only available when an enhanced security feature is
enabled, so I injected ereport() at the tail.
(It is invoked on ALTER TABLE ... SECURITY_LABEL = '...';)

However, most of hooks do nothing or don't change existing behavior
when enhanced security is disabled.

So, I can't understand why it gives adverse affects in performances.

I can admit it needs additional steps to invoke empty functions at least.
However, using "static inline" was arguable in the previous discussion
due to the GCC dependency.

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

On 1/27/09 6:57 PM, "Greg Smith" <gsmith@gregsmith.com> wrote:

> On Tue, 27 Jan 2009, Chad Sellers wrote:
> 
>> I'll speak to this a bit (Josh is also a Tresys employee). I can't say who
>> my customers are, but I can speak to their needs. They really need row-level
>> mandatory access controls (so both).
> 
> From the perspective of what this would buy as far as this feature being a
> PostgreSQL advocacy point, one of the questions Tom asked about a bit
> upthread is still a bit hazy here.  There are commercial database
> offerings selling into the "trusted" space already.  While the use-cases
> you describe make perfect sense, I don't think it's clear to everyone yet
> if there's a unique draw to a PostgreSQL + selinux solution that the class
> of customers you're talking about would prefer it to purchasing one of
> those products.  Is the cost savings the main driver here, or is there
> something else about a secure LAPP stack that makes it particularly
> compelling?
> 
Cost savings may be a driver, but it's certainly not the main driver. A
problem with the current systems (e.g. Oracle) is that they largely operate
in their own world. Oracle Label Security labels do not extend outside the
database to the rest of the system, which makes it impossible to build
certain things. For instance, you can't really build a trusted LAPP stack.
With SE-PostgreSQL, the label used for access control is bound by the kernel
and used by it and the other components of the stack (apache, PostgreSQL).
We can even extend all the way to the end system using labeled IPSec.

Another major drawback to the currently available trusted databases is that
they have hard-coded policy rules rather than flexible ones. While these
hard-coded rules may be appropriate in some scenarios, they often end up
being impractical. For instance, any sort of pipeline (like that of my
example 3) is difficult if not impossible to implement on most of these
hard-coded systems. Additionally, changing the label in most of these
systems requires a privilege that lets you bypass the policy rules, while a
flexible Type Enforcement system (like SE-PostgreSQL) allows you to specify
exactly how relabels should be allowed within the policy.

I'll be happy to provide more details where required. Hopefully that
provided a little light.

Thanks,
Chad Sellers

Robert Treat <xzilla@users.sourceforge.net> writes:
> The more I think about it, the more I feel that where we failed for 8.3 was 
> not having a short 8.4 cycle lined up, which would give more freedom to bump 
> patches to the next release. 

Heh.  The reason we wanted a short 8.3 cycle was so we could push out
patches that had been held over from 8.2.  We are going to have exactly
no credibility if we tell Simon et al "we're pushing these patches to
8.5, but don't worry, it'll be a short release cycle".

I think the best thing we could do overall is to set release dates and
stick to them.  If your patch is not ready, well, at least it will get
out in a defined amount of time.  Right now, the *real* problem with it
being pushed to the next release is you don't know how successful some
other guy will be at persuading us to delay the next release.
        regards, tom lane

On Tue, 2009-01-27 at 21:07 -0500, Tom Lane wrote:
> Robert Treat <xzilla@users.sourceforge.net> writes:
> > The more I think about it, the more I feel that where we failed for 8.3 was 
> > not having a short 8.4 cycle lined up, which would give more freedom to bump 
> > patches to the next release. 
> 
> Heh.  The reason we wanted a short 8.3 cycle was so we could push out
> patches that had been held over from 8.2.  We are going to have exactly
> no credibility if we tell Simon et al "we're pushing these patches to
> 8.5, but don't worry, it'll be a short release cycle".
> 
> I think the best thing we could do overall is to set release dates and
> stick to them.  If your patch is not ready, well, at least it will get
> out in a defined amount of time.  Right now, the *real* problem with it
> being pushed to the next release is you don't know how successful some
> other guy will be at persuading us to delay the next release.

+1

Joshua D. Drake


> 
>             regards, tom lane
> 
-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

Andrew Sullivan wrote:
> On Tue, Jan 27, 2009 at 12:41:36PM -0500, Stephen Frost wrote:
>> * Gregory Stark (stark@enterprisedb.com) wrote:
>>> It does seem weird to simply omit records rather than throw an error and
>>> require the user to use a where clause, even if it's something like WHERE
>>> pg_accessible(tab).
> 
> […]
> 
>> do row-level security and security labels.  Requiring a where clause
>> or you throw an error would certainly make porting applications that
>> depend on that mechanism somewhat difficult, and doesn't really seem
>> like it'd gain you all that much...
> 
> Throwing an error would entail a side-channel leak that would not be
> acceptable to the security community, I bet.  That said, I have
> reservations, along the lines of Peter E's, that the early
> design-level objections to the approach were never answered.  I
> certainly never got any real answer to questions I asked, for what
> it's worth.  
> 
> I will note that I tried to have a look at the literature on this
> topic.  As I started to read, it became obvious that it was copious,
> but pretty well-determined.  What bothered me most about the answers I
> got was that there never seemed to be an answer to "please outline the
> design principles" except for "it's what SE-Linux does".  The OS-level
> control rules seemed to me to be totally foreign to the database
> world, precisely because ACID is a problem in databases in a way it
> isn't for filesystems under the traditional UNIX model.  I formed the
> impression -- only an impression, mind, that there was a poor fit
> between SE-Linux and database systems, and that the proponents had
> decided that enough caulk (in the form of "don't do that") would seal
> the gap.

As I noted before, there is a symmetrical structure between
OS and DBMS.

In operating system, a process accesses objects (like file,
socket, ...) managed by operating system via system calls.
Its security system (filesystem permission, SELinux, ...)
acquires the request and applies its access control rules.

In DBMS, a client accesses database objects managed by DBMS
via SQL queries. Its security system (Database ACL,
SE-PostgreSQL, ...) aquires the request and applies its access
control rules.

We have similar structures everywhere, not only DBMS and OS.
What we should pay attention is a subject entity accesses via
a method provided by object managers (DBMS, OS, ...), and
object managers apply its rules to decide either "allowed"
or "denied" on acquired request.

> I haven't (obviously) been paying much attention to this topic since,
> but I detect something of the same sort of response in the recent
> discussion.  Maybe the goal isn't explicit enough.  If the goal is
> compliance with some set of well-defined tests, what are they?  If the
> goal is buzzword compliance, what are the tests of that (to the extent
> there ever are some)?  If the goal is just "security enhancement", I
> confess that I am still unable to understand the definitions of
> "security" and "enhancement" such that I think we have some
> operationalization of what the patch is supposed to provide.

The most significant feature is centralized access control policy
between OS and DBMS. Did you attend PGcon2008?
I talked here we should consider the value of information asset
is independent from the way to store them.

Needless to say, the value of information asset is decided by its
contents. If your credit card number is recorded on a paper,
do you think it has lesser value than recorded on database?
Thus, from the viewpoint of security, we need to consider the
way to apply unified centralized access controls.

SELinux works as "security server". It enables to provide a centralized
access control decision any other object managers (like, Linux kernel,
X-window, PostgreSQL, ...). It is quite consistent because of
common security policy and common clearance of entities.

It finally enables to apply centralized access control policy on
whole of application stack.
Please note that 95% of attacks in 2008 targeted to web system,
so it gives a nightmare for security folks.

Thanks,

> I know there are people who think this is cool.  I guess, if I were
> running the circus, I'd want to know what's cool about it, and why.
> Then maybe the project would be in a position to understand whether
> that kind of cool is the way it wants to be.  But without a clear
> problem statement, and a roadmap of how the patches solve the problem,
> I'm at a loss.  And last I checked (which was, admittedly, not today),
> the project pages didn't have that information.
> 
> A
> 


-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Tom Lane wrote:
> Heh.  The reason we wanted a short 8.3 cycle was so we could push out
> patches that had been held over from 8.2.  We are going to have exactly
> no credibility if we tell Simon et al "we're pushing these patches to
> 8.5, but don't worry, it'll be a short release cycle".
> 
> I think the best thing we could do overall is to set release dates and
> stick to them.  

On the other hand, we might be better throwing out release dates
and releasing at the end of any Commit Fest where there is enough
demand/interest.

Then we could release 8.4 now, with few objections since people would
know that if Hot Standby has enough demand it could be released within one
commitfest of it being in good shape.   Likewise the SE* guys would be
aware that if they want that patch to drive a release, they'd need to
round up more visible demand.

Heck, 8.4 could have been released a whole commitfest ago if enough
people think FSM is the killer feature in that one.

Tom Lane wrote:
> Joshua Brindle <method@manicmethod.com> writes:
>> Tom Lane wrote:
>>> Right, which is why it's bad for something like a foreign key constraint
>>> to expose the fact that the row does exist after all.
> 
>> Once again, this is not an issue for us.
> 
> Yes it is an issue; claiming it isn't doesn't make it so.  You just
> stated, in effect, that you don't implement data hiding in the
> filesystem because it would break standard Unix filesystem semantics.
> How is that consistent with your opinion that it's okay to break
> standard SQL semantics in order to implement data hiding in a database?
> 
> The question of whether there is a covert channel is only a small part
> of my complaint here.  If it's the judgement of security experts that
> that's an acceptable thing, that's fine, it's their turf.  But SQL
> behavior is my turf, and I'm not happy with discarding fundamental
> semantic properties.

Do you forget a GUC option: sepostgresql_row_leve = on/off ?
It was a requirement from Simon Riggs at Oct 2008, IIRC.

Its original purpose is to reduce storage consumption due to
the security label of each tuples, however, it can allow users
to choose the granularity of access controls in finally.

I can understand you have a complaint about covert channels
via PK/FK due to the row-level granularity.
However, in other hand, we can already see a commercial
producet which provides row-level access controls without
any cares about covert channels.

It shows a fact that not negligible number of users accept
row-level granularity, even if it has covert channels.
What is needed is an explicit documentation about covert
channels and providing a few options to users.
It is not a reasonable stance to deny anything due to a
single item which is acceptable some of people, at least.

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Ron Mayer <rm_pg@cheapcomplexdevices.com> writes:
> Tom Lane wrote:
>> I think the best thing we could do overall is to set release dates and
>> stick to them.  

> On the other hand, we might be better throwing out release dates
> and releasing at the end of any Commit Fest where there is enough
> demand/interest.

> Then we could release 8.4 now, with few objections since people would
> know that if Hot Standby has enough demand it could be released within one
> commitfest of it being in good shape.

Huh?  The people who'd been working with the expectation of completing
their patch in the *next* fest would scream loud and long at having the
release schedule pulled out from under them.  Any development plan with
more than a two-month time horizon would turn into a game of schedule
russian roulette.  The only way this works without having a lot of
pissed-off developers is if we actually release a lot more often, which
is not going to fly.  We don't have the manpower to manage that, and
we don't have users that want it, either.  (They might like it better
if we had a better upgrade-in-place story, but ...)
        regards, tom lane

Joshua Brindle wrote:
> Stephen Frost wrote:
>> * Joshua Brindle (method@manicmethod.com) wrote:
>>> They are separate. If you look at the patches you'll see a pgace 
>>> part, this is where the core interfaces to the security backends, and 
>>> you'll see a rowacl backend and an sepgsql backend.
>>
>> Right, guess it wasn't clear to me that the PGACE bits for row-level
>> access control could be used independently of SELinux (and maybe even on
>> systems that don't have SELinux..?).
>>
> 
> Sure, if you look at pgaceHooks.c you'll see:

It is basically possible to implement something like "PostgreSQL
Label Security" on PGACE framework.
But I don't want to discuss it now, because it surely burst
SE-PostgreSQL until v8.4 beta.

If desired, I'll queue it my todo list next to SE-PostgreSQL...

> bool
> pgaceExecScan(Scan *scan, Relation rel, TupleTableSlot *slot)
> {
>         /* Hardwired DAC checks */
>         if (!rowaclExecScan(scan, rel, slot))
>                 return false;
> 
>         switch (pgace_feature)
>         {
> #ifdef HAVE_SELINUX
>         case PGACE_FEATURE_SELINUX:
>                 if (sepgsqlIsEnabled())
>                         return sepgsqlExecScan(scan, rel, slot);
>                 break;
> #endif
>         default:
>                 break;
>         }
>         return true;
> }
> 
> Notice the rowacl call outside of the HAVE_SELINUX ifdefs

FYI:
In the earlier version, these are mutually exclusive, so we could
not apply SE-PostgreSQL, when a binary is built with RowAcl feature.

However, Bruce Momjian suggested it is not proper manner in
PostgreSQL, because it intend to wrap all available features
into a single binary due to packaging benefit, and all the
available options should be configured by runtime.

In addition, IIRC, Peter E suggested it is not symmetrical
that we cannot apply both of DAC and MAC on tuples simultaneously,
although SE-PostgreSQL applies MAC on tables/columns which
PostgreSQL has DAC features on.
So, I add a support simultaneous DAC&MAC.

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

> I think the best thing we could do overall is to set release dates and
> stick to them.  If your patch is not ready, well, at least it will get
> out in a defined amount of time.  Right now, the *real* problem with it
> being pushed to the next release is you don't know how successful some
> other guy will be at persuading us to delay the next release.

+1, LOL.

Let's not forget that we've already got CTE, window functions, partial
vacuums, and column-level permissions, all of which are major features
that should benefit a lot of people.  I hope Hot Standby gets
committed but even if it doesn't, I'm still going to get a lot of
benefit out of this release, so I'd like it to happen on some sort of
reasonable time scale.

...Robert

On Tue, Jan 27, 2009 at 2:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> This seems to me to be exactly parallel to deciding that SELinux should
> control only table/column permissions within SQL; an approach that would
> be enormously less controversial, less expensive, and more reliable than
> what SEPostgres tries to do.
>

seems that the controversial part of sepgsql is row level permissions,
can we try to commit (obviously with good revision and test) the
table/column privileges part of that patch?

that is still a step on the direction of full centralized security
management on the system...

let the row level privileges part for 8.5, that way the patch will be
smaller now and then...

remember, postponed is not rejected is just a way to give more time to
think (WITH patch comes from the prior release cycle and was committed
in this release), not to think about one scenario but about all
possible scenarios in a more wide audience

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

On Wed, 28 Jan 2009, KaiGai Kohei wrote:

> It shows a fact that not negligible number of users accept row-level 
> granularity, even if it has covert channels.

From my read of the examples that Chad provided, keeping the existence of 
things secret from complete outsiders doesn't look like the prime concern. 
There's plenty of these applications floating around where everyone 
involved is cleared, but to different levels and projects.

The person working on a "secret" project knows perfectly well that there 
are also "top secret" projects floating around they aren't cleared for, 
and that's OK.  They'll probably detect their existence by the doors 
they're not allowed through long before they notice that database rows are 
missing. If you're able to sit in front of a computer that's capable of 
even reaching this information but aren't supposed to be anywhere near it, 
that means there's been a physical security breach.

Where I suspect this is all is going to settle down into is that if 1) the 
SE GUC is on and 2) one of the tables in a join has rows filtered, then 
you can expect that a) it's possible that the result will leak 
information, which certainly need to be documented, and b) the 
optimizations Tom mentioned that "assume foreign key constraints hold" 
will not be possible to implement, so performance will suck compared to a 
similar situation in an unsecured environment.  And all that may very well 
be just fine as far as the people wanting to build applications with 
SEPostgreSQL are concerned.  It will just hint toward using a schema 
design with table-level controls instead if you care about high 
performance on that style of join.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Tom Lane wrote:
> The flaw in that argument is that as you are doing it, the
> de-optimization only happens on queries that actually need the behavior.
> As the SEPostgres patch is constructed, the planner could *never* trust
> an FK for optimization since it would have no way to know whether row
> level permissions might be present (perhaps only for some rows) at
> execution time.

Is the "never" is really correct?

In the following case, it is necessary not to apply optimization:

- SE-PostgreSQL is working, and its row-level access controls are available (sepostgresql_row_level=on).
- Row-level ACL is available on the target relation. It is controlable via table option.

So, it is necessary to add a new security hook to give a hint
the optimizer.
Sorry, I overlooked this optimization. But is is not a fundamental
design issue. PGACE already has a hook to give a hint to optimizer.
It will be a similar one.

See, pgaceAllowFunctionInlined(...);
http://code.google.com/p/sepgsql/source/browse/trunk/sepgsql/src/backend/security/pgaceHooks.c#948

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Jaime Casanova wrote:
> On Tue, Jan 27, 2009 at 2:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> This seems to me to be exactly parallel to deciding that SELinux should
>> control only table/column permissions within SQL; an approach that would
>> be enormously less controversial, less expensive, and more reliable than
>> what SEPostgres tries to do.
>>
> 
> seems that the controversial part of sepgsql is row level permissions,
> can we try to commit (obviously with good revision and test) the
> table/column privileges part of that patch?

Actually, it is already done.
http://code.google.com/p/sepgsql/source/browse/trunk/sepgsql/src/backend/utils/misc/guc.c#1242
http://code.google.com/p/sepgsql/source/browse/trunk/sepgsql/src/backend/security/sepgsql/permissions.c#518

Its original purpose is different, to reduce storage consumption.
But it can be a point of compromise.
See, http://archives.postgresql.org/message-id/492691A8.8030103@ak.jp.nec.com

Is it a reasonable option to allow users to turn on/off?
I can add a documentation about its background and tradeoffs,
for user's correct decision.

Thanks,

> that is still a step on the direction of full centralized security
> management on the system...
> 
> let the row level privileges part for 8.5, that way the patch will be
> smaller now and then...
> 
> remember, postponed is not rejected is just a way to give more time to
> think (WITH patch comes from the prior release cycle and was committed
> in this release), not to think about one scenario but about all
> possible scenarios in a more wide audience


-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Greg Smith wrote:
> Where I suspect this is all is going to settle down into is that if 1) 
> the SE GUC is on and 2) one of the tables in a join has rows filtered, 
> then you can expect that a) it's possible that the result will leak 
> information, which certainly need to be documented, and b) the 
> optimizations Tom mentioned that "assume foreign key constraints hold" 
> will not be possible to implement, so performance will suck compared to 
> a similar situation in an unsecured environment.

c) security feature gives the optimizer a hint "don't optimize out
this table, please!" via a security hook.
I think it is a quite reasonable approach, as I noted in another message.

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

On Tuesday 27 January 2009 21:07:48 Tom Lane wrote:
> Robert Treat <xzilla@users.sourceforge.net> writes:
> > The more I think about it, the more I feel that where we failed for 8.3
> > was not having a short 8.4 cycle lined up, which would give more freedom
> > to bump patches to the next release.
>
> Heh.  The reason we wanted a short 8.3 cycle was so we could push out
> patches that had been held over from 8.2. 

And the reason that didn't work was because when we got to feature freeze, we 
once again had several large, complex patches which people didn't want to 
push for the long 8.4 cycle. (But note: people are willing to push patches 
when they believe the wait time won't be excessive for eventual inclusion)

> We are going to have exactly 
> no credibility if we tell Simon et al "we're pushing these patches to
> 8.5, but don't worry, it'll be a short release cycle".
>

The other options being we stall 8.4 indefinatly waiting for HS (which, 
honestly I am comfortable with), or his patches get pushed and he doesnt get 
them for another 14 months. Seems to me our credibility isn't really even a 
factor here. 

Right now I'm really trying to figure out how to solve this problem for the 
long term. If we say up front now that the next 2 cycles are short cycles, 
then I think people will be more willing to push patches come end-of-8.5 (and 
let's not pretend we're not going to have this same argument over streaming 
replication or synchronous replay or merge command or whatever hot feature is 
almost ready at that time)

> I think the best thing we could do overall is to set release dates and
> stick to them.  If your patch is not ready, well, at least it will get
> out in a defined amount of time.  Right now, the *real* problem with it
> being pushed to the next release is you don't know how successful some
> other guy will be at persuading us to delay the next release.
>

I wont argue that setting release dates and sticking to them is a bad idea, 
but that extra month at the end that that occures due to feature lobbying 
doesn't strike me as the straw breaking the camels back, it's the 12-14 
months in front of that people don't want to wait through. 

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote:
> Marko Kreen wrote:
> > On 1/27/09, Peter Eisentraut <peter_e@gmx.net> wrote:
> >> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote:
> >>  > Such app already exists:
> >>  >
> >>  >   http://ozlabs.org/~jk/projects/patchwork/
> >>  >
> >>  > So it's a matter of just setting it up.
> >>
> >> I was in fact in the process of setting that up just now. :-)
> >
> > Nice to know. :)   I feel that even if we decide to do our own
> > solution it would be good to try existing solution first.
>
> IIRC, we already installed and tried this a while ago. I don't remember
> exactly what it failed on, but there was something pretty clear. But
> maybe it's been fixed by now.

Details?  I find no public record of this.

Greg Smith wrote:
> Where I suspect this is all is going to settle down into is that if 1)
> the SE GUC is on and 2) one of the tables in a join has rows filtered,
> then you can expect that a) it's possible that the result will leak
> information, which certainly need to be documented, 

As far as I can tell this is the case however you hide the information.
If you implemented it with views you'll have the same issue. If you hide
the existence of project p_id="TOPSECRET01" and people can run inserts
then they can spot it. Likewise, it you have fkey references to the row
then deletions can be used to spot it.

--  Richard Huxton Archonet Ltd

Greg Smith wrote:
> PostgreSQL advocacy point, one of the questions Tom asked about a bit 
> upthread is still a bit hazy here.  There are commercial database 
> offerings selling into the "trusted" space already.  While the use-cases 
> you describe make perfect sense, I don't think it's clear to everyone 
> yet if there's a unique draw to a PostgreSQL + selinux solution that the 
> class of customers you're talking about would prefer it to purchasing 
> one of those products.  Is the cost savings the main driver here, or is 
> there something else about a secure LAPP stack that makes it 
> particularly compelling?

According to the data available to me, it is a combination of doing it 
better than the other guys (e.g., a SELinux type interface instead of 
something handcrafted) and the usual cost savings.

Richard Huxton wrote:
> Greg Smith wrote:
>> Where I suspect this is all is going to settle down into is that if 1)
>> the SE GUC is on and 2) one of the tables in a join has rows filtered,
>> then you can expect that a) it's possible that the result will leak
>> information, which certainly need to be documented, 
> 
> As far as I can tell this is the case however you hide the information.
> If you implemented it with views you'll have the same issue. If you hide
> the existence of project p_id="TOPSECRET01" and people can run inserts
> then they can spot it. Likewise, it you have fkey references to the row
> then deletions can be used to spot it.
> 
It is a covert channel discussion.
At least, SE-PostgreSQL does not care about hiding its existence,
so it does not prevent user to infer the existence of a tuple
with same key value, using PK confliction.
(Please note that he must have a info about PK value or lucky
to make a key confliction.)
But, it enables to prevent unclassified user to read the tuple,
and him to know an info the tuple contains "p_id=TOPSECRET01" as
a result of this read action.

Thanks,
-- 
KaiGai Kohei <kaigai@kaigai.gr.jp>

Peter Eisentraut wrote:
> On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote:
>> Marko Kreen wrote:
>>> On 1/27/09, Peter Eisentraut <peter_e@gmx.net> wrote:
>>>> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote:
>>>>  > Such app already exists:
>>>>  >
>>>>  >   http://ozlabs.org/~jk/projects/patchwork/
>>>>  >
>>>>  > So it's a matter of just setting it up.
>>>>
>>>> I was in fact in the process of setting that up just now. :-)
>>> Nice to know. :)   I feel that even if we decide to do our own
>>> solution it would be good to try existing solution first.
>> IIRC, we already installed and tried this a while ago. I don't remember
>> exactly what it failed on, but there was something pretty clear. But
>> maybe it's been fixed by now.
> 
> Details?  I find no public record of this.

I don't recall specifically :-( Which in itself might mean it's
worthwhile to make another try. But i recall trying that one and
reviewboard, and none of them was what we needed.

If you look at Berkus' list of required features (if you haven't seen
it, I'm sure he'll be happy to send you a copy), you will see that it
doesn't come close. We can always argue if his list is reasonable :-),
but that's just a fact. It has nothing about round-robin reviewers. It
has no keep-track-of-nagging features. It has no integration with our
mail archives. At least it didn't then - it also appears to have no
online documentation, so I can't easily check now :-P

//Magnus

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Magnus Hagander a écrit :
> Peter Eisentraut wrote:
>> On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote:
>>> Marko Kreen wrote:
>>>> On 1/27/09, Peter Eisentraut <peter_e@gmx.net> wrote:
>>>>> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote:
>>>>>  > Such app already exists:
>>>>>  >
>>>>>  >   http://ozlabs.org/~jk/projects/patchwork/
>>>>>  >
>>>>>  > So it's a matter of just setting it up.
>>>>>
>>>>> I was in fact in the process of setting that up just now. :-)
>>>> Nice to know. :)   I feel that even if we decide to do our own
>>>> solution it would be good to try existing solution first.
>>> IIRC, we already installed and tried this a while ago. I don't remember
>>> exactly what it failed on, but there was something pretty clear. But
>>> maybe it's been fixed by now.
>> Details?  I find no public record of this.
> 
> I don't recall specifically :-( Which in itself might mean it's
> worthwhile to make another try. But i recall trying that one and
> reviewboard, and none of them was what we needed.
> 
> If you look at Berkus' list of required features (if you haven't seen
> it, I'm sure he'll be happy to send you a copy), 

Josh, can you please give the link to this list of feature ?

> you will see that it
> doesn't come close. We can always argue if his list is reasonable :-),
> but that's just a fact. It has nothing about round-robin reviewers. It
> has no keep-track-of-nagging features. It has no integration with our
> mail archives. At least it didn't then - it also appears to have no
> online documentation, so I can't easily check now :-P
> 
> //Magnus
> 
> 


- --
Cédric Villemain
Administrateur de Base de Données
Cel: +33 (0)6 74 15 56 53
http://dalibo.com - http://dalibo.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkmAOAMACgkQo/dppWjpEvxn6ACg2F5to39Q9fW9vvm25E9fW2Zl
GAAAoOP9yMO3WuT5Rj98s7OyHhDYK4Ui
=rP62
-----END PGP SIGNATURE-----

<div class="gmail_quote">On Wed, Jan 28, 2009 at 4:28 AM, Peter Eisentraut <span dir="ltr"><<a
href="mailto:peter_e@gmx.net">peter_e@gmx.net</a>></span>wrote:<br /><blockquote class="gmail_quote"
style="border-left:1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div
class="Ih2E3d">GregSmith wrote:<br /><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204);
margin:0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> PostgreSQL advocacy point, one of the questions Tom asked about a bit
upthreadis still a bit hazy here.  There are commercial database offerings selling into the "trusted" space already.
 Whilethe use-cases you describe make perfect sense, I don't think it's clear to everyone yet if there's a unique draw
toa PostgreSQL + selinux solution that the class of customers you're talking about would prefer it to purchasing one of
thoseproducts.  Is the cost savings the main driver here, or is there something else about a secure LAPP stack that
makesit particularly compelling?<br /></blockquote><br /></div> According to the data available to me, it is a
combinationof doing it better than the other guys (e.g., a SELinux type interface instead of something handcrafted) and
theusual cost savings.<br /></blockquote></div><br />I don't know about better, but I would definitely say that it's a
moreintegrated (with the OS) solution.  Can you get Oracle to use SELinux policies?  Sure.  But it would take a
combinationof Label Security, Fine Grained Access Control tweaks, custom C functions, and custom policies to handle the
accesscontrol.  And, it would cost a helluva lot of money.<br /><br />In short, this would make Postgres quite a bit
moreappetizing to those who need this functionality, those who prefer SELinux-based policies, and those who don't have
thetime/money to do it in systems like Oracle.  How many people is that?  Based on my consulting experience and
questionsfrom DoD/DoE people specifically, I think the number of people needing this feature is fairly small right
now. But, it wouldn't hurt us to have it.<br clear="all" /><br />Just to make it clear, this feature wouldn't make
Postgresa "trusted" database in any certification sense.  So, using that term would likely cause confusion and get
peoplewho used it thinking it had an EAL certification into trouble.<br /><br />-- <br />Jonah H. Harris, Senior DBA<br
/>myYearbook.com<br/><br /> 

Robert Treat wrote:
> The problem is that the pain point is extremely high for missing a given 
> release cycle. If you don't make a specific release, you have a 12-14 month 
> wait for feature arrival. Given that choice, someone who deperately need (aka 
> wants) HS/SEPostgres/Win32/HOT/IPU/etc... will likely be willing to push a 
> release 3-6 months for that one feature. 

There will always be some features that people are willing to push a
release for, if it's a feature they want. At least I hope so - because
that means we keep adding new features that people want. But as long as
there is some overlap in development timelines - which there will
*always* be - there will always be some patch that is not quite ready on
time.

If the release is pushed back, maybe some other patch could also have
been finished by the new deadline - should that also be included? What
about a completely new feature that isn't even started yet, but that
could easily be finished before the new deadline? What makes those less
worthy?

The question is, how long do we make people wait for *other* features.
It delays this version *and* the next.



> Incidentally, this is probably why people didnt worry about making a given 
> commitfest. The pain point was low, so there was no percieved need to rework 
> a patch for a specific commit, since there was another one just a couple 
> months away. However, we still see a rush of patches at the final freeze 
> because people know that "there is no tommorrow" at that point.  

A problem at this point is that we are basically serializing the project
over one or a couple of patches. All those people who aren't qualified
to review/work on HS or SEPG are left in a position where they can't get
anything done. Sure, they can work on patches offline, and add them to a
hypothetical future commitfest that they have no clue when it's going to
happen, so they don't know when they need to be available to deal with
feedback. And we're back to ending up with a lot of conflicting patches
simply because they sit in the queue for too long. That's a lot of
developer talent wasted.

The commitfests were designed in part to get around this - to get
developers quick feedback so they can get on with more features. The
final commitfest being much longer than the others by design already
makes this hard. Dragging it out even longer makes it an even bigger
failure in this way.

We can't just say that everybody should help with these patches. Not
everybody is qualified to do so, or has an interest to, while they're
still both qualified and interested in working on other things for 8.5.


>> In the third 
>> place, unless we get an upgrade-in-place process that works all the
>> time, we would be looking at maintaining twice as many back branches
>> in order to provide the same kind of release lifespan we have today.
>> We are at the limit of what we can realistically do in back-branch
>> maintenance already :-(
>>
> 
> Yeah, I can't argue with that. I'm not sure it's an unsolvable problem though; 
> if odd/even release maintenance doesn't sound good, we could do something 
> like ubuntus LTS, where we pick 1 release every 3 years to make a long-term 
> support commitment (I think 5 years is our current max), and keep other 
> releases on a shorter lifespan (1 or 2 years). Certainly having IPU (or is 
> that UIP?) would make that an easier decision.  

We're still going to have to pay the full cost of doing a release every
time. With beta/rc management, release notes, announcements, postings,
packaging and all those things.


The PostgreSQL tree tends to be a lot more stable than others. In many
cases, you can just snapshot CVS HEAD and get more or less the same
things. Perhaps if someone were to simply maintain a bunch of tags or
branches against "known feature-points in the system" in a separate SCM
somewhere that'd be enough for people who desperately need the features
- or who just want to test them out earlier. That would be close to zero
cost for the core project to maintain.

//Magnus

Josh Berkus wrote:
> 
>> That's modest. I've talked to several oracle and db2 shops that want a
>> standby for reporting that has relatively easy setup/maintenance
>> (handling ddl is a big part of this) and the HS feature your working
>> on will give them something as good as what they are getting now. So
>> yeah, HS appeals to future users as well.  
> 
> I've talked to some of my clients, and while they *want* synch or
> near-synch HS, even slow HS is useful to them *now*.
> 
> One client is planning on deploying a rather complex FS cloning
> infrastructure just to have a bunch of reporting, testing and read-only
> search databases they need.  They'd be thrilled with an HS feature which
> produced DBs which were an hour out of date (or even 6 hours out of
> date), but ran read-only queries.

I have a lot of clients who would be thrilled to have stuff that's been
in our tree for half a year by now, and they'd be thrilled to have it
*now*. How much extra should we have them wait for the needs of your
clients?


(Yes, I have clients now who would very much like HS as well, of course,
but that's not the point)


//Magnus

Robert Haas wrote:
>> I think the best thing we could do overall is to set release dates and
>> stick to them.  If your patch is not ready, well, at least it will get
>> out in a defined amount of time.  Right now, the *real* problem with it
>> being pushed to the next release is you don't know how successful some
>> other guy will be at persuading us to delay the next release.
> 
> +1, LOL.
> 
> Let's not forget that we've already got CTE, window functions, partial
> vacuums, and column-level permissions, all of which are major features
> that should benefit a lot of people.  I hope Hot Standby gets
> committed but even if it doesn't, I'm still going to get a lot of
> benefit out of this release, so I'd like it to happen on some sort of
> reasonable time scale.

Agreed. Even without HS, this will be one of the biggest releases we've
had in a while, IMHO.

//Magnus

Magnus Hagander <magnus@hagander.net> writes:

> Josh Berkus wrote:
>> 
>> One client is planning on deploying a rather complex FS cloning
>> infrastructure just to have a bunch of reporting, testing and read-only
>> search databases they need.  They'd be thrilled with an HS feature which
>> produced DBs which were an hour out of date (or even 6 hours out of
>> date), but ran read-only queries.
>
> I have a lot of clients who would be thrilled to have stuff that's been
> in our tree for half a year by now, and they'd be thrilled to have it
> *now*. How much extra should we have them wait for the needs of your
> clients?

I really am unconvinced by the argument that delaying existing features is a
big deal. Logically it's less of a big deal than delaying HS a whole release
cycle which I already said I think isn't a big deal either. This is purely a
question of latency between development and release; we still get just as much
in each release, it's just 6-12 months later than it might have been.

What bothers me is delaying work on things like Bitmap Indexes which won't
really start in earnest until Gianni can get feedback from the lists after the
release. Or Join Removal which Simon isn't going to look at until after HS is
committed (not *released* -- once it's *committed* he'll be free to go on to
other things). This would impact *bandwidth* of development which I think is a
much bigger deal. It reduces the amount of new features in each release, not
just which release they fall in.

I'm a bit shocked by how long Tom expects the release cycle to take even if we
froze the code today. I guess I forget how long it takes and how many steps
there are from past releases. If it's going to be 9+ months between Nov 1st
and the first commitfest I'm worried about how many patches will be
languishing in the queue with their authors having moved on to other more
fruitful pastures in the mean time. If we delay further we're talking about
close to a year with developers left hanging.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's RemoteDBA services!

On Wed, Jan 28, 2009 at 11:31:25AM +0900, KaiGai Kohei wrote:

> As I noted before, there is a symmetrical structure between
> OS and DBMS.

Well, you said that before.  I think your analogy is awful.  I don't
think the similarities are nearly as great as you think, and I also
think there are significant differences that _make_ the difference in
these cases.  In particular,

> In operating system, a process accesses objects (like file,
> socket, ...) managed by operating system via system calls.
> Its security system (filesystem permission, SELinux, ...)
> acquires the request and applies its access control rules.
>
> In DBMS, a client accesses database objects managed by DBMS
> via SQL queries. Its security system (Database ACL,
> SE-PostgreSQL, ...) aquires the request and applies its access
> control rules.

the difference here is that in the OS, a process accessing the object
has few to no guarantees about concurrency.  In RDBMS, a very
significant reason even to use the DBMS is the ACID guarantees, which
make a number of claims about concurrency that simply aren't there in
most filesystems.  It's at exactly this architectural point that most
of the in-principle design questions have been aimed.  My personal
view is that those questions haven't really been answered very well,
but as I've already said I mostly stopped paying attention to this
work several months ago; so maybe I overlooked something.  I note that
Peter and Bruce seem to have been satisfied, so maybe they understood
something I don't (that's quite likely).

> The most significant feature is centralized access control policy
> between OS and DBMS. 

Right, I get that; but all the discussion I've seen on this suggest
that, to get the benefit of the centralised access control, I trade
away certain well-understood assumptions of a relational environment,
but without much indication that I've done so.

> I talked here we should consider the value of information asset
> is independent from the way to store them.

Yes, I know that was your premise.  I am not entirely sure I agree
with it, is the thing.

> Needless to say, the value of information asset is decided by its
> contents. 

Nonsense.  The value of an information asset is determined only partly
by its contents.  I'd argue that the value of an information asset is
a function of its use-value.  If the information asset is completely
unusable, then it isn't worth anything at all.  

> If your credit card number is recorded on a paper,
> do you think it has lesser value than recorded on database?

Yes.  The database makes the credit card number available to other
applications, which can then use that data to charge the credit card
with other purchases.  For me, therefore, the piece of paper,
correctly handled, imposes less risk than the database; in addition,
the piece of paper offers a smaller advantage, because it cannot be
leveraged to make other interactions more convenient.  Finally, the
piece of paper offers a different kind of risk, because if it is
mishandled and then becomes the basis on which the number ends up in a
database, I have a new problem for which I was not prepared.

I believe my fundamental objection was that, as far as I was able to
tell, SELinux simply didn't have anything useful to say about
concurrent actions on data under SE controls; that's because it was
aimed at a fairly primitive database (a filesystem) without the rich
concurrency support of RDBMS.  I still don't see anywhere in your
discussions an extension of the SELinux model to account for that
concurrency richness, so I think there's something wrong with the
principles from which your're starting.  I'm totally prepared to admit
I've missed something, however.  Also, since this isn't really my
problem any more, I'm unlikely to spend much time reading more design
notes or anything of the sort.  

Finally,

> It finally enables to apply centralized access control policy on
> whole of application stack.
> Please note that 95% of attacks in 2008 targeted to web system,
> so it gives a nightmare for security folks.

this argument gets to the heart of what you seem to want, which is a
centralized system that guarantees the controls you want.  I'm
actually dubious that such centralization is actually the benefit that
its proponents seem to think it is; but if it is, then the centralised
system needs to be exactly as rich as the richest system under
control.  By starting with SELinux, I argue that the approach starts
with a too-poor model.  (See above.)  

More fundamentally, the premise that the database is just a part of an
"application stack" is, in my view, exactly _why_ these systems are so
vulnerable to attack.  Database management systems are not designed to
be dumb storage for a single application, and they're actually very
poorly adapted to such a role.  My impression is that SEPostgres is an
attempt to finally force the database system under such controls, as
though it were a glorified filesystem.  I have no idea whether it will
work; but to my way of thinking, it's a mindset foreign to the
principles of RDBM system design.  That could be why some of us react
to the proposal with perplexed looks.

A

-- 
Andrew Sullivan
ajs@crankycanuck.ca

On Wed, 2009-01-28 at 14:55 +0100, Magnus Hagander wrote:

> If the release is pushed back, maybe some other patch could also have
> been finished by the new deadline - should that also be included? What
> about a completely new feature that isn't even started yet, but that
> could easily be finished before the new deadline? What makes those less
> worthy?

Committers have always added features after freeze...

For example, Virtual TransactionIds were added to 8.3 almost exactly 5
months after feature freeze. Not even suggested until about 5 months
after, in fact. I argued against such a change late in the cycle, but we
did it. It's a great feature and I'm glad we did.

If we try to shorten the release cycle, we just end up missing out on
tuning opportunities that emerge in beta. IIRC 8.2 was delayed while we
changed index cost models. Thankfully.

8.0 was shipped with a completely ineffective bgwriter, so the above
changes seem like common sense in comparison.

The only way to keep the dev window open longer is to overlap the start
of the next cycle with the previous one. i.e. branch new version before
final release.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

On Wednesday 28 January 2009 08:55:56 Magnus Hagander wrote:
> We're still going to have to pay the full cost of doing a release every
> time. With beta/rc management, release notes, announcements, postings,
> packaging and all those things.
>

As I pointed out to Tom, by percentage the additional beta/release cycles 
wouldn't be very different than what we have now; the more churn you have 
during development, the longer it takes to beta/release. 

I'm pretty sure that if we had pushed everything not committed on December 
1st, we would be very close to release right now, and that's with more dev 
cycles than I'm talking about for 8.5.  And I think most people (aka not the 
patch authors :-) would have been willing to push the stuff we're dealing 
with now if they knew the next release would be closer to 6 months than 14 
months. 

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

Robert Treat <xzilla@users.sourceforge.net> writes:
> On Wednesday 28 January 2009 08:55:56 Magnus Hagander wrote:
>> We're still going to have to pay the full cost of doing a release every
>> time. With beta/rc management, release notes, announcements, postings,
>> packaging and all those things.

> As I pointed out to Tom, by percentage the additional beta/release cycles 
> wouldn't be very different than what we have now; the more churn you have 
> during development, the longer it takes to beta/release. 

I don't believe that thesis in itself, because it ignores economies of
scale and parallelism for beta testing.  And in any case it's complete
nonsense in respect to back-branch maintenance costs.  If we double
the frequency of releases we are going to be pretty much forced to halve
the support lifetime, and ain't nobody going to be happy with us.
        regards, tom lane

Le 28 janv. 09 à 16:22, Simon Riggs <simon@2ndQuadrant.com> a écrit :
> The only way to keep the dev window open longer is to overlap the
> start
> of the next cycle with the previous one. i.e. branch new version
> before
> final release.

This is the second time the idea is raised and I like it.

Do we have anywhere near enough resources for this to happen?
That would mean first 8.5 commit-fest begins e.g. April 1st, while
hopefully 8.4 enters beta or get ready for it.

If no commiter is available for reviewed patch they just get postponed
as ready to commit on next commit fest. This way our Round Robin
Reviewers team is still at work while in beta, and more importantly
developpers still get feedback.

--
dim

Dimitri Fontaine <dfontaine@hi-media.com> writes:
> Le 28 janv. 09 à 16:22, Simon Riggs <simon@2ndQuadrant.com> a écrit :
>> The only way to keep the dev window open longer is to overlap the  
>> start of the next cycle with the previous one. i.e. branch new version  
>> before final release.

> This is the second time the idea is raised and I like it.

> Do we have anywhere near enough resources for this to happen?

No.  The key committers are overstressed already.

It would also pretty much guarantee that we get *no* help during review
and beta, because everyone else will find it more interesting/fun to
work on new patches instead.  The current system at least gives
non-committer developers some motivation to help with that stuff,
because they know their patches won't be looked at until beta is over.
        regards, tom lane

Zdenek Kotala wrote:
> 
> Bruce Momjian p??e v po 26. 01. 2009 v 23:02 -0500:
> > OK, time for me to chime in.
> > 
> > I think the outstanding commit-fest items can be broken down into four
> > sections:
> > 
> >     o  Log streaming
> >     o  Hot standby
> >     o  SE-PostgreSQL
> >     o  Others
> 
> You omit pg_upgrade. Does it mean that this project is already killed
> for 8.4?

I considered pg_upgrade one of the "others" on the list;  it is not as
complex as the previous three.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

> I considered pg_upgrade one of the "others" on the list;  it is not as
> complex as the previous three.

LOL.

...Robert

On Wednesday 28 January 2009 12:35:42 Tom Lane wrote:
> Robert Treat <xzilla@users.sourceforge.net> writes:
> > On Wednesday 28 January 2009 08:55:56 Magnus Hagander wrote:
> >> We're still going to have to pay the full cost of doing a release every
> >> time. With beta/rc management, release notes, announcements, postings,
> >> packaging and all those things.
> >
> > As I pointed out to Tom, by percentage the additional beta/release cycles
> > wouldn't be very different than what we have now; the more churn you have
> > during development, the longer it takes to beta/release.
>
> I don't believe that thesis in itself, because it ignores economies of
> scale and parallelism for beta testing.  And in any case it's complete 
> nonsense in respect to back-branch maintenance costs.  If we double
> the frequency of releases we are going to be pretty much forced to halve
> the support lifetime, and ain't nobody going to be happy with us.
>

Yes, back branch maintanance is an issue, but I'd bet that as long as we 
occasionally designate specific releases as long term support releases (my 
guess is 1 every 4 releases, though I haven't done the math), people would be 
comfortable with this.  We've already had short maintainance windows for 
win32 support, and that has gone over without significant uproar. Also other 
projects (some much larger than ours) have implemented similar schemes, and 
it's been fairly well recieved. People understand the trade-offs of new 
features verses stability, and as long as you give them a long term option, 
they're happy.  

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Tue, Jan 27, 2009 at 12:52 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> Right, but you expect that to be a small and predictable cost, say in
> >> the single-digits-percentage range.  Plan optimizations that
> >> suddenly stop happening can cost you multiple orders of magnitude.
> 
> > Well, look at it another way.  If we don't accept row-level security
> > into PostgreSQL, then people will have to implement it themselves.  In
> > fact, I currently have a real application that does exactly this.  The
> > row-filtering is done, in essence, by having the web application add
> > certain conditions to the WHERE clause of certain queries depending on
> > which user is making the request.  And if those WHERE clauses happen
> > to mention columns from table X, then table X won't be a candidate for
> > join removal.  The only difference is that the logic is in my app
> > rather than in the database itself.
> 
> > To put that another way, row-level permissions are just another
> > attribute of a table that could potentially affect the query result,
> > and the impact of referring to that attribute will be exactly the same
> > as the impact of referring to any other attribute in that table.
> 
> The flaw in that argument is that as you are doing it, the
> de-optimization only happens on queries that actually need the behavior.
> As the SEPostgres patch is constructed, the planner could *never* trust
> an FK for optimization since it would have no way to know whether row
> level permissions might be present (perhaps only for some rows) at
> execution time.  You could only get back the optimization in builds with
> SEPostgres compiled out.  That's pretty nasty, especially for packagers
> who have to decide which build setting will displease fewer users.

I am afraid that SQL-level row permissions would also cause that
problem, and I thought they were enabled by default.  (The configure
flag --enable-selinux only controls SE-Linux support.)

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Simon Riggs wrote:
> 
> On Tue, 2009-01-27 at 14:18 -0500, Tom Lane wrote:
> > Joshua Brindle <method@manicmethod.com> writes:
> > > Tom Lane wrote:
> > >> Right, which is why it's bad for something like a foreign key constraint
> > >> to expose the fact that the row does exist after all.
> > 
> > > Once again, this is not an issue for us.
> > 
> > Yes it is an issue
> 
> > The question of whether there is a covert channel is only a small part
> > of my complaint here.  If it's the judgement of security experts that
> > that's an acceptable thing, that's fine, it's their turf.  But SQL
> > behavior is my turf, and I'm not happy with discarding fundamental
> > semantic properties.
> 
> Why did we bother to invite Joshua here if we aren't going to listen to
> him?
> 
> Thanks for coming to help Joshua, much appreciated.

I agree.  This is exactly the type of feedback I was hoping for.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Robert Haas wrote:
> > The flaw in that argument is that as you are doing it, the
> > de-optimization only happens on queries that actually need the behavior.
> > As the SEPostgres patch is constructed, the planner could *never* trust
> > an FK for optimization since it would have no way to know whether row
> > level permissions might be present (perhaps only for some rows) at
> > execution time.  You could only get back the optimization in builds with
> > SEPostgres compiled out.  That's pretty nasty, especially for packagers
> > who have to decide which build setting will displease fewer users.
> 
> OK, I think I am starting to understand your concern now.
> 
> My understanding of how the world works is SE-PostgreSQL would always
> be compiled in but could be turned off at run-time with a GUC.  I know
> that the original design called for a compile-time switch, but
> everyone hated it and I am pretty sure KaiGai changed it.  If he
> hasn't, he will.  :-)
> 
> There was also talk of having a table-level option to include/exclude
> the security ID (I'm not sure if it's currently implemented that way).
>  Obviously that wouldn't be relevant for row-level MAC (because
> presumably you would need/want that turned on for all tables) but it
> would be very relevant for row-level DAC (because it's easy, at least
> for me, to imagine that you would only turn this on for a subset,
> possibly quite a small subset, of your tables where you knew that it
> was really needed).
> 
> If, by default, we make sepostgresql disabled, MAC security IDs on
> newly created tables off, and DAC security IDs on newly created tables
> off, then the pain will be confined to people who explicitly request
> sepostgresql or row-level DAC.

Yes, if there is concern about row-level security turning off
optimizations, some flag would have to be checked so the optimization
would be possible for sites not using row-level security;  ideally there
would be a table-level flag, and I think the current patch implements it
that way.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce@momjian.us> writes:
> Tom Lane wrote:
>> As the SEPostgres patch is constructed, the planner could *never* trust
>> an FK for optimization since it would have no way to know whether row
>> level permissions might be present (perhaps only for some rows) at
>> execution time.  You could only get back the optimization in builds with
>> SEPostgres compiled out.  That's pretty nasty, especially for packagers
>> who have to decide which build setting will displease fewer users.

> I am afraid that SQL-level row permissions would also cause that
> problem, and I thought they were enabled by default.  (The configure
> flag --enable-selinux only controls SE-Linux support.)

So they would.  However, I've already determined that I'm against
row-level permissions of either flavor ;-)
        regards, tom lane

Peter Eisentraut wrote:
> On Tuesday 27 January 2009 16:36:50 Stephen Frost wrote:
> > * Peter Eisentraut (peter_e@gmx.net) wrote:
> > > As one of the earlier reviewers, I think the design is OK, but the way
> > > the implementation is presented was not acceptable, and very little has
> > > been accomplished in terms of reacting to our comments.  For example,
> > > where is the SQL row security feature, which should have been designed,
> > > implemented, and committed separately, in the opinion of most
> > > commentaries.
> >
> > Eh?  Are you thinking of column-level privileges, which was committed
> > last week?
> 
> No.

The point is we would have preferred to see SQL-level row permissions as
a separate patch first; that just didn't happen in this case, and it
makes the discussion a little more complex.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Simon Riggs wrote:
> 
> On Tue, 2009-01-27 at 15:46 -0500, Tom Lane wrote:
> > Peter Eisentraut <peter_e@gmx.net> writes:
> > > Not to pick on you personally, but this is the kind of review that should have 
> > > happened six months ago, not during a "why is our development process 
> > > inadequate" discussion on the eve of beta.
> > 
> > Right now, today, in this thread, is the first time that we've had any
> > opportunity to debate the design of SEPostgres with knowledgeable people
> > other than KaiGai-san.  It would likely be better if we started a new
> > thread with a more appropriate title, but I see nothing wrong with
> > asking pretty fundamental questions.
> 
> Except that Bruce and I already checked detailed documentation
> references on this very topic months ago. Check with Bruce; he was
> careful to point those things out to me, so I'm sure he'll do the same
> for you. I'm satisfied, on this point.

Sure, here is the email:
http://archives.postgresql.org/pgsql-hackers/2008-09/msg01750.php


--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On Wed, 2009-01-28 at 17:04 -0500, Tom Lane wrote:

> The key committers are overstressed already.

Some developers are too.

I'm sure there's a way to avoid it being a zero-sum game.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support

Robert Treat wrote:
> The revisionism was that of "remarkable failure".  That was our shortest 
> release cycle in the modern era. And it didn't have the advantage of the 
> commitfest process. 
> 
> But I think what is important here is to recognize why it didn't work. Once 
> again we ended up with large, complex features (HOT, tsearch) that people 
> didn't want to wait 14 months to see if they missed the 8.3 release. And yes, 
> most of these same arguements were raised then... "full text search is killer 
> feature", "whole applications are waiting for in-core full text search", "hot 
> will give allow existing customers to use postgres on a whole new 
> level", "not fair to push back patches so long when developers followed the 
> rules", "sponsors wont want to pay for features they wont see for 
> years", "developers dont want to wait so long to see features committed", and 
> on and on...  

I think the big reminder for me from above is that we will always have
big stuff that doesn't make a certain major release, and trying to
circumvent our existing process is usually a mistake.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Robert Treat wrote:
> > We are going to have exactly 
> > no credibility if we tell Simon et al "we're pushing these patches to
> > 8.5, but don't worry, it'll be a short release cycle".
> >
> 
> The other options being we stall 8.4 indefinatly waiting for HS (which, 
> honestly I am comfortable with), or his patches get pushed and he doesnt get 
> them for another 14 months. Seems to me our credibility isn't really even a 
> factor here. 
> 
> Right now I'm really trying to figure out how to solve this problem for the 
> long term. If we say up front now that the next 2 cycles are short cycles, 
> then I think people will be more willing to push patches come end-of-8.5 (and 
> let's not pretend we're not going to have this same argument over streaming 
> replication or synchronous replay or merge command or whatever hot feature is 
> almost ready at that time)

You want the fix --- have people complete their patches long before the
last commit fest --- no matter of adjustment is going to fix that. If
they can't complete it early, fine, but don't expect we can reach some
ideal development schedule by adjusting things --- sometimes ideal just
isn't possible.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Gregory Stark wrote:
> I'm a bit shocked by how long Tom expects the release cycle to take even if we
> froze the code today. I guess I forget how long it takes and how many steps
> there are from past releases. If it's going to be 9+ months between Nov 1st
> and the first commitfest I'm worried about how many patches will be
> languishing in the queue with their authors having moved on to other more
> fruitful pastures in the mean time. If we delay further we're talking about
> close to a year with developers left hanging.

What makes the period long are the many things we don't know are
broken in CVS, but we know we will find before the final release.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On Wednesday 28 January 2009 20:12:40 Bruce Momjian wrote:
> Robert Treat wrote:
> > The revisionism was that of "remarkable failure".  That was our shortest
> > release cycle in the modern era. And it didn't have the advantage of the
> > commitfest process.
> >
> > But I think what is important here is to recognize why it didn't work.
> > Once again we ended up with large, complex features (HOT, tsearch) that
> > people didn't want to wait 14 months to see if they missed the 8.3
> > release. And yes, most of these same arguements were raised then... "full
> > text search is killer feature", "whole applications are waiting for
> > in-core full text search", "hot will give allow existing customers to use
> > postgres on a whole new level", "not fair to push back patches so long
> > when developers followed the rules", "sponsors wont want to pay for
> > features they wont see for years", "developers dont want to wait so long
> > to see features committed", and on and on...
>
> I think the big reminder for me from above is that we will always have
> big stuff that doesn't make a certain major release, and trying to
> circumvent our existing process is usually a mistake.
>

Our usual process *is* to try and circumvent our usual process. And I believe 
it will continue to be that way until we lower the incentive to lobby for 
circumvention. 

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

Robert Treat wrote:
> On Wednesday 28 January 2009 20:12:40 Bruce Momjian wrote:
>   
>> Robert Treat wrote:
>>     
>>> The revisionism was that of "remarkable failure".  That was our shortest
>>> release cycle in the modern era. And it didn't have the advantage of the
>>> commitfest process.
>>>
>>> But I think what is important here is to recognize why it didn't work.
>>> Once again we ended up with large, complex features (HOT, tsearch) that
>>> people didn't want to wait 14 months to see if they missed the 8.3
>>> release. And yes, most of these same arguements were raised then... "full
>>> text search is killer feature", "whole applications are waiting for
>>> in-core full text search", "hot will give allow existing customers to use
>>> postgres on a whole new level", "not fair to push back patches so long
>>> when developers followed the rules", "sponsors wont want to pay for
>>> features they wont see for years", "developers dont want to wait so long
>>> to see features committed", and on and on...
>>>       
>> I think the big reminder for me from above is that we will always have
>> big stuff that doesn't make a certain major release, and trying to
>> circumvent our existing process is usually a mistake.
>>
>>     
>
> Our usual process *is* to try and circumvent our usual process. And I believe 
> it will continue to be that way until we lower the incentive to lobby for 
> circumvention. 
>   

Anybody lobbying to get the process circumvented gets their feature 
reverted? :-)

cheers

andrew

Peter Eisentraut wrote:
> On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote:
> > Marko Kreen wrote:
> > > On 1/27/09, Peter Eisentraut <peter_e@gmx.net> wrote:
> > >> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote:
> > >>  > Such app already exists:
> > >>  >
> > >>  >   http://ozlabs.org/~jk/projects/patchwork/
> > >>  >
> > >>  > So it's a matter of just setting it up.
> > >>
> > >> I was in fact in the process of setting that up just now. :-)
> > >
> > > Nice to know. :)   I feel that even if we decide to do our own
> > > solution it would be good to try existing solution first.
> >
> > IIRC, we already installed and tried this a while ago. I don't remember
> > exactly what it failed on, but there was something pretty clear. But
> > maybe it's been fixed by now.
>
> Details?  I find no public record of this.

I think it was Keystone;  Marc set it up.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

> Our usual process *is* to try and circumvent our usual process. And I believe
> it will continue to be that way until we lower the incentive to lobby for
> circumvention.

I think Tom and Bruce have both pretty much stated that they're not
keen on a shorter release cycle, and they're the ones who would have
to do the work, so I think this argument is going nowhere.  Moreover,
I agree with them.  Having short release cycles would probably be a
good idea if we had a larger community with more patch authors, more
reviewers, and more committers.  As it is, I think it would simply
mean that the committers would spend more time doing releases and
back-branch maintenance, and correspondingly less time to do what we
really want them to do: review and commit patches.  That problem is
already pretty severe, and it would be a bad thing if it got worse.

If anyone really can't wait a year for a new feature, they can
backport it to the previous release, or pay the patch author to do it.If they were paying the patch author to develop
thefeature in the
 
first place, it shouldn't be a horribly expensive proposition.

At the moment, what we really should be doing is conducting final
reviews of as many patches as possible and trying to make sure that
they are in good shape to be committed so that the people who have put
in hard work for THIS release have a chance to see that work go out
the door in a somewhat timely fashion.

...Robert

On Wed, 28 Jan 2009, Bruce Momjian wrote:

>> Details?  I find no public record of this.
>
> I think it was Keystone;  Marc set it up.

If that was it, god, that was what, 10 years ago when we tried that?  And
yes, it was atrocious ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org                              MSN . scrappy@hub.org
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664

Marc G. Fournier wrote:
> On Wed, 28 Jan 2009, Bruce Momjian wrote:
>
> >> Details?  I find no public record of this.
> >
> > I think it was Keystone;  Marc set it up.
>
> If that was it, god, that was what, 10 years ago when we tried that?  And
> yes, it was atrocious ...

Yep, that matches my recollection.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

KaiGai Kohei wrote:
> Bruce Momjian wrote:
> > OK, time for me to chime in.
> > 
> > I think the outstanding commit-fest items can be broken down into four
> > sections:
> > 
> >     o  Log streaming
> >     o  Hot standby
> >     o  SE-PostgreSQL
> >     o  Others
> 
>   - snip -
> 
> > SE-PostgreSQL has been in steady development for a year so this is the
> > time to decide about it.  My feeling is if we don't accept it now, we
> > are never going to have SE-Linux or row-level security.  The next week
> > should show us the right direction when we start discussion on
> > Wednesday, noon GMT.
> 
> Hmm...?
> This conditional punishment of death seems to me not a reasonable one.
> If we can found a matter as a result of discussion, which is impossible
> to fix within reasonable term, I'll agree it being postponed to v8.5.
> However, why is the punishment of death necessary here?

What I meant was that this features is not going to get any better than
the work you have done, so if we reject it odds are we will never accept
it.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On Wednesday 28 January 2009 23:42:11 Robert Haas wrote:
> > Our usual process *is* to try and circumvent our usual process. And I
> > believe it will continue to be that way until we lower the incentive to
> > lobby for circumvention.
>
> I think Tom and Bruce have both pretty much stated that they're not
> keen on a shorter release cycle, and they're the ones who would have
> to do the work, so I think this argument is going nowhere. Moreover, 
> I agree with them.  Having short release cycles would probably be a
> good idea if we had a larger community with more patch authors, more
> reviewers, and more committers.  

more reviewers and more committers would actually be an argument against 
shorter release cycles, since we'd have a better shot at actually getting all 
patches in in a timely fasion.  we dont, and we cant change that. again, 
thats the whole point of this... look at the variables and see which ones we 
can and cant change, and if those changes would address the causes. 

> As it is, I think it would simply 
> mean that the committers would spend more time doing releases and
> back-branch maintenance, and correspondingly less time to do what we
> really want them to do: review and commit patches.  That problem is
> already pretty severe, and it would be a bad thing if it got worse.
>

read up-thread, i've already shown that this would not be the case. remember, 
we reduce the pressure from the large, complex patches that bottleneck the 
process, which allows more parralell review/commit. 

> If anyone really can't wait a year for a new feature, they can
> backport it to the previous release, or pay the patch author to do it.
>  If they were paying the patch author to develop the feature in the
> first place, it shouldn't be a horribly expensive proposition.
>

i dont think we as a community should encourage people to pay for private 
releases. that is a *really* bad idea. 

> At the moment, what we really should be doing is conducting final
> reviews of as many patches as possible and trying to make sure that
> they are in good shape to be committed so that the people who have put
> in hard work for THIS release have a chance to see that work go out
> the door in a somewhat timely fashion.
>

not that i disagree, but if we can improve things for the people working on 
the next release, well, I think that's a good idea too, and I dont see how 
doing nothing is going to help. 

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

On 29 jan 2009, at 05.35, Bruce Momjian <bruce@momjian.us> wrote:

> Peter Eisentraut wrote:
>> On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote:
>>> Marko Kreen wrote:
>>>> On 1/27/09, Peter Eisentraut <peter_e@gmx.net> wrote:
>>>>> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote:
>>>>>> Such app already exists:
>>>>>>
>>>>>>  http://ozlabs.org/~jk/projects/patchwork/
>>>>>>
>>>>>> So it's a matter of just setting it up.
>>>>>
>>>>> I was in fact in the process of setting that up just now. :-)
>>>>
>>>> Nice to know. :)   I feel that even if we decide to do our own
>>>> solution it would be good to try existing solution first.
>>>
>>> IIRC, we already installed and tried this a while ago. I don't
>>> remember
>>> exactly what it failed on, but there was something pretty clear. But
>>> maybe it's been fixed by now.
>>
>> Details?  I find no public record of this.
>
> I think it was Keystone;  Marc set it up.

Not at all. That was *ages* ago. This was recently, and I was part of
setting it up myself...

/Magnus

Magnus Hagander wrote:
> 
> 
> On 29 jan 2009, at 05.35, Bruce Momjian <bruce@momjian.us> wrote:
> 
>> Peter Eisentraut wrote:
>>> On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote:
>>>> Marko Kreen wrote:
>>>>> On 1/27/09, Peter Eisentraut <peter_e@gmx.net> wrote:
>>>>>> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote:
>>>>>>> Such app already exists:
>>>>>>>
>>>>>>>  http://ozlabs.org/~jk/projects/patchwork/
>>>>>>>
>>>>>>> So it's a matter of just setting it up.
>>>>>>
>>>>>> I was in fact in the process of setting that up just now. :-)
>>>>>
>>>>> Nice to know. :)   I feel that even if we decide to do our own
>>>>> solution it would be good to try existing solution first.
>>>>
>>>> IIRC, we already installed and tried this a while ago. I don't remember
>>>> exactly what it failed on, but there was something pretty clear. But
>>>> maybe it's been fixed by now.
>>>
>>> Details?  I find no public record of this.
>>
>> I think it was Keystone;  Marc set it up.
> 
> Not at all. That was *ages* ago. This was recently, and I was part of 
> setting it up myself...

In fact we still have the jail running ...


Stefan

Stefan Kaltenbrunner wrote:
> Magnus Hagander wrote:
>>
>>
>> On 29 jan 2009, at 05.35, Bruce Momjian <bruce@momjian.us> wrote:
>>
>>> Peter Eisentraut wrote:
>>>> On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote:
>>>>> Marko Kreen wrote:
>>>>>> On 1/27/09, Peter Eisentraut <peter_e@gmx.net> wrote:
>>>>>>> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote:
>>>>>>>> Such app already exists:
>>>>>>>>
>>>>>>>>  http://ozlabs.org/~jk/projects/patchwork/
>>>>>>>>
>>>>>>>> So it's a matter of just setting it up.
>>>>>>>
>>>>>>> I was in fact in the process of setting that up just now. :-)
>>>>>>
>>>>>> Nice to know. :)   I feel that even if we decide to do our own
>>>>>> solution it would be good to try existing solution first.
>>>>>
>>>>> IIRC, we already installed and tried this a while ago. I don't
>>>>> remember
>>>>> exactly what it failed on, but there was something pretty clear. But
>>>>> maybe it's been fixed by now.
>>>>
>>>> Details?  I find no public record of this.
>>>
>>> I think it was Keystone;  Marc set it up.
>>
>> Not at all. That was *ages* ago. This was recently, and I was part of
>> setting it up myself...
> 
> In fact we still have the jail running ...

That's reviewboard AFAIK, or do we also have one with patchwork? Which one?

//Magnus

Magnus Hagander wrote:
> Stefan Kaltenbrunner wrote:
>> Magnus Hagander wrote:
>>>
>>> On 29 jan 2009, at 05.35, Bruce Momjian <bruce@momjian.us> wrote:
>>>
>>>> Peter Eisentraut wrote:
>>>>> On Tuesday 27 January 2009 23:59:46 Magnus Hagander wrote:
>>>>>> Marko Kreen wrote:
>>>>>>> On 1/27/09, Peter Eisentraut <peter_e@gmx.net> wrote:
>>>>>>>> On Tuesday 27 January 2009 15:51:02 Marko Kreen wrote:
>>>>>>>>> Such app already exists:
>>>>>>>>>
>>>>>>>>>  http://ozlabs.org/~jk/projects/patchwork/
>>>>>>>>>
>>>>>>>>> So it's a matter of just setting it up.
>>>>>>>> I was in fact in the process of setting that up just now. :-)
>>>>>>> Nice to know. :)   I feel that even if we decide to do our own
>>>>>>> solution it would be good to try existing solution first.
>>>>>> IIRC, we already installed and tried this a while ago. I don't
>>>>>> remember
>>>>>> exactly what it failed on, but there was something pretty clear. But
>>>>>> maybe it's been fixed by now.
>>>>> Details?  I find no public record of this.
>>>> I think it was Keystone;  Marc set it up.
>>> Not at all. That was *ages* ago. This was recently, and I was part of
>>> setting it up myself...
>> In fact we still have the jail running ...
> 
> That's reviewboard AFAIK, or do we also have one with patchwork? Which one?

well from a quick glance there is the bugzilla demo install as well as 
pieces of reviewboard and patchwork on the trackerdemo jail.


Stefan

I thought reviewboard looked pretty good for code quality patch review. It
would be cool if someone could write a mail filter which automatically added
any patches posted to the list to reviewboard.

Incidentally one issue with reviewboard/patchwork/whatever is that they tend
to encourage the review to do a line-by-line review of the code itself rather
than the overall architecture. That's fine for the kind of code-quality
reviews some authors (like myself I admit :( ) need sometimes. And it would
make it easier to do hit-and-run comments when you see minor issues not worth
pestering authors about on-list. But I don't think it really helps with the
hard reviews.


But that's just a cute tool for one particular part of the work. I don't think
it addresses workflow management like RT or debbugs (or trac?) would.


--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's RemoteDBA services!

> read up-thread, i've already shown that this would not be the case. remember,
> we reduce the pressure from the large, complex patches that bottleneck the
> process, which allows more parralell review/commit.

I read what you wrote - I just don't believe it.  My own experience is
that doing more releases is more work.  Also, two commitfests per
release means that if you can't get your patch up to snuff in two
iterations, you're bumped.  The diminished pain of being bumped will,
I think, be more than balanced out by the increased frequency of
bumps.  Many patches needed 2 or 3 commitfests to get committed; all
of the people who need 3 iterations, and anyone who needs 2 iterations
and doesn't submit until the second commitfest of the cycle, will take
two releases to get out the door.  I don't believe you're ever going
to make beta/release so low-impact that that won't be disruptive or
irritating to people.

...Robert

Robert Haas <robertmhaas@gmail.com> writes:

>> read up-thread, i've already shown that this would not be the case. remember,
>> we reduce the pressure from the large, complex patches that bottleneck the
>> process, which allows more parralell review/commit.
>
> I read what you wrote - I just don't believe it.  My own experience is
> that doing more releases is more work.  

Yeah, more releases than once a year is kind of crazy.

> Also, two commitfests per release means that if you can't get your patch up
> to snuff in two iterations, you're bumped.

I wish we could get rid of the whole concept and stigma of being "bumped" your
patch will be released in the next release after it's committed. What does it
matter if there's been another release since you started or not?

ISTM there are two ways to make the release step take less time. Either we
sacrifice quality or we change the development process so more testing and
review happens during development. I see the latter as realistic if we hold
off committing (but not reviewing) any major patches submitted after
commitfest#1 until the next commitfest#1. That means when it comes time to
release there won't be any major changes in it that people haven't had months
to experiment with already.

I would still like an answer to my question about what steps there are that
take so many months for a release, but I expect most of them boil down to
(justified) paranoia about testing major features that people haven't already
tested outside of development environments.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production
Tuning

On Thursday 29 January 2009 11:40:48 Stefan Kaltenbrunner wrote:
> well from a quick glance there is the bugzilla demo install as well as
> pieces of reviewboard and patchwork on the trackerdemo jail.

So what's the URL and where can we sign up?

On Thursday 29 January 2009 08:39:48 Gregory Stark wrote:
> I wish we could get rid of the whole concept and stigma of being "bumped"
> your patch will be released in the next release after it's committed. What
> does it matter if there's been another release since you started or not?
>

This is the whole point. It isn't that there is a stigma to getting bumped; It 
matters becase missing a release means 12-14 months before your feature will 
be released, even when it takes far less time than that to complete the work. 
That 12-14 month delay has real implications on a number of levels for users 
and developers. 

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

> I wish we could get rid of the whole concept and stigma of being "bumped" your
> patch will be released in the next release after it's committed. What does it
> matter if there's been another release since you started or not?

It matters because the intervening beta/release cycle is likely to sap
some focus from the ongoing process of patch review.  If nothing else,
it's a longer period of time before the patch gets another look, and
authors, reviewers, and committers have moved on to other things and
forgotten details that then need to be rehashed.  If we could have
commitfests every two months regardless of the release schedule, then
I think the timing of releases really wouldn't matter as much.  But
then we'd need multiple branches and I don't think Tom et. al. want to
go that route.  And I understand why.  Merging in CVS is the suck, but
even if you can make that aspect of things easier, it's still going to
be some work, and you still have the problem that people might not do
enough testing on release N if they're already in the midst of heavy
development for release N+1.

Linux solves this problem by having back-branch maintainers and
subsystem maintainers who have roles that are intermediate between
random patch submitter and full committer.  We don't really have quite
that much structure, maybe because we're a small project.  But it's
worth thinking about, because if Tom or Peter or Alvaro or Heikki
called me up and said, "If you agree to do be responsible for task X
over the next year, which I estimate will save me 40 hours of work, I
will agree to spend an additional 10 hours over that same time period
reviewing and potentially committing your patches" - I would probably
take that deal. And if I didn't, I bet there are at least five other
people who would be more than happy to get in line.  Of course, making
that work depends on one of those people having a well-defined task
that they trust me (or whoever) to do and that can be severed from the
rest of their work, and there may not be anything of that nature (or
maybe it's at a higher increment, like 200 hours for 50 hours, in
which case it would be more than I could take on, but someone else
might be interested).  But I think that's what we have to look for.  I
don't believe that you can speed a project up much by adjusting the
length of the release cycle, but it is *sometimes* possible to speed
up a project by dividing up the work over more people.

> I would still like an answer to my question about what steps there are that
> take so many months for a release, but I expect most of them boil down to
> (justified) paranoia about testing major features that people haven't already
> tested outside of development environments.

That I'm not sure about.

...Robert

Peter Eisentraut wrote:
> On Thursday 29 January 2009 11:40:48 Stefan Kaltenbrunner wrote:
>> well from a quick glance there is the bugzilla demo install as well as
>> pieces of reviewboard and patchwork on the trackerdemo jail.
> 
> So what's the URL and where can we sign up?

note the "pieces" part of my mail :-) As far as I recall the patchworks 
install somehow collided with the reviewboard one so it was disabled 
because Zdenek was still actively using reviewboard.


Stefan

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:

> Peter Eisentraut wrote:
>> On Thursday 29 January 2009 11:40:48 Stefan Kaltenbrunner wrote:
>>> well from a quick glance there is the bugzilla demo install as well as
>>> pieces of reviewboard and patchwork on the trackerdemo jail.
>>
>> So what's the URL and where can we sign up?
>
> note the "pieces" part of my mail :-) As far as I recall the patchworks install
> somehow collided with the reviewboard one so it was disabled because Zdenek was
> still actively using reviewboard.

For what it's worth Google seems to have rolled out reviewboard as a Google
App. We could use it hosted on there if we wanted.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production
Tuning

Gregory Stark wrote:
> Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:
> 
>> Peter Eisentraut wrote:
>>> On Thursday 29 January 2009 11:40:48 Stefan Kaltenbrunner wrote:
>>>> well from a quick glance there is the bugzilla demo install as well as
>>>> pieces of reviewboard and patchwork on the trackerdemo jail.
>>> So what's the URL and where can we sign up?
>> note the "pieces" part of my mail :-) As far as I recall the patchworks install
>> somehow collided with the reviewboard one so it was disabled because Zdenek was
>> still actively using reviewboard.
> 
> For what it's worth Google seems to have rolled out reviewboard as a Google
> App. We could use it hosted on there if we wanted.

well the problem is more that we don't seem to want reviewboard at all - 
getting it hosted as a proper service is not really an issue.



Stefan

All,

Thing is, our review/commit process is so peculiar to our project that 
using *any* prebuilt solution would require us to change our process to 
support the tool. And I can't imagine this group doing that.

--Josh

On Thu, 2009-01-29 at 10:18 -0800, Josh Berkus wrote:
> All,
> 
> Thing is, our review/commit process is so peculiar to our project that 
> using *any* prebuilt solution would require us to change our process to 
> support the tool. And I can't imagine this group doing that.

I am not sure I agree with this.

Someone submits patchticket is createdreviewer takes ticketcomments
submitter takes ticketfixes based on comments
review takes ticketapproves
if reviewer is a committers, he commits.
if reviewer isn't he set the ticket to "need final review"
tickets that are in that state are reviewed by commiters.

Sounds like standard stuff to me.

Joshua D. Drake


> 
> --Josh
> 
-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

Josh,

> Someone submits patch
>  ticket is created
>  reviewer takes ticket
>  comments
> submitter takes ticket
>  fixes based on comments
> review takes ticket
>  approves
> if reviewer is a committers, he commits.
> if reviewer isn't he set the ticket to "need final review"
> tickets that are in that state are reviewed by commiters.
> 
> Sounds like standard stuff to me.

But that's *not* actually how we do things.  So you're making my point.


--Josh

Josh Berkus <josh@agliodbs.com> writes:
> But that's *not* actually how we do things.  So you're making my point.

Well, the stuff around the wiki status board is pretty new and I don't
think anyone feels that it's set in stone yet.  The thing we don't want
to compromise on, IMHO, is that the long-term record of what's happened
is in the mailing list archives and *not* in the internal state of some
tool we happen to be using.  (One obvious reason for not compromising
on that is that we'd be locked into whatever tool we first pick.)
But it doesn't really matter whether the tool thinks it has archival
state, as long as we can make it link to the archives conveniently.
        regards, tom lane

Stefan Kaltenbrunner píše v čt 29. 01. 2009 v 18:29 +0100:
> Peter Eisentraut wrote:
> > On Thursday 29 January 2009 11:40:48 Stefan Kaltenbrunner wrote:
> >> well from a quick glance there is the bugzilla demo install as well as
> >> pieces of reviewboard and patchwork on the trackerdemo jail.
> > 
> > So what's the URL and where can we sign up?
> 
> note the "pieces" part of my mail :-) As far as I recall the patchworks 
> install somehow collided with the reviewboard one so it was disabled 
> because Zdenek was still actively using reviewboard.

I don't use it at this moment. You can disable reviewboard if you want.
Zdenek

On Thursday 29 January 2009 12:03:45 Robert Haas wrote:
> I
> don't believe that you can speed a project up much by adjusting the
> length of the release cycle, but it is *sometimes* possible to speed
> up a project by dividing up the work over more people.
>

This is interesting. We had a problem in 8.3 (and most of the releases before 
that) of too many patches in the queue at the end of the development cycle. 
Most everyone agreed that more reviewers/committers would help, but given no 
way to conjure them up, they realized that wasn't a solution. Instead, we 
went to a tighter development cycle, with one month of dev and then a 
commifest. This allowed us to better parralelize both reviews and commits, 
allowed a number of patches to get bumped through multiple fests with 
relatively few compliants (after all, the next fest was just a month down the 
line), keep the patch queue pretty manageable (right up untill the end, when 
we stopped the cycle), and also delivered us some really big features along 
the way.   

-- 
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

Peter Eisentraut wrote:
> On Thursday 29 January 2009 11:40:48 Stefan Kaltenbrunner wrote:
>> well from a quick glance there is the bugzilla demo install as well as
>> pieces of reviewboard and patchwork on the trackerdemo jail.
> 
> So what's the URL and where can we sign up?

resurrected the install and subscribed it to pgsql-hackers:

http://trackerdemo.postgresql.org

however it seems that it won't deal with patches that just have
Content-Type: text/plain (like: 
http://archives.postgresql.org/pgsql-hackers/2009-01/msg02586.php) - 
seems not to hard to fix from a quick glance at the code however.


Stefan

On Tue, Jan 27, 2009 at 11:23 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Dave Page <dpage@pgadmin.org> writes:
>> On Tue, Jan 27, 2009 at 3:40 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> I already pointed out some pretty serious problems with the updatable
>>> views patch.  Are you claiming they are trivial to fix?
>
>> Not at all. I think the deferral of that particular patch is the
>> correct thing to do because there are confirmed, real problems with it
>> that are not realistic to fix in an appropriate timeframe for the
>> release. The primary case that I'm objecting to is HS which you've
>> been saying will take 10 - 12 months to complete having by your own
>> admission not looked at the code or followed the discussion
>> particularly closely.
>
> Well, perhaps I'm being pessimistic, or perhaps you're being
> optimistic.  What is undeniable fact is that HS will not be committable
> this week, which will make it three months since feature freeze.
> As for when it *will* be committable --- Heikki is saying two weeks
> if no new problems crop up, but given the rate at which new problems
> have been found so far, what are the odds of that?  We've seen this
> movie before.
>
> Since it's going to take us two weeks to clean up the other loose ends
> anyway, there's no harm in letting Simon and Heikki try to complete the
> patch by then.  But I'll happily lay a side bet with you about what the
> situation will be two weeks from now.

I think Tom wins this bet, since it's now been four weeks.  In fact,
he was being optimistic: the loose ends aren't cleaned up either
(based on a review of the wiki, we're about half way there: six
patches have been committed in the intervening time and seven remain
in the queue).

...Robert

On Wed, 2009-02-25 at 22:56 -0500, Robert Haas wrote:

> > Since it's going to take us two weeks to clean up the other loose ends
> > anyway, there's no harm in letting Simon and Heikki try to complete the
> > patch by then.  But I'll happily lay a side bet with you about what the
> > situation will be two weeks from now.
> 
> I think Tom wins this bet, since it's now been four weeks.  In fact,
> he was being optimistic: the loose ends aren't cleaned up either
> (based on a review of the wiki, we're about half way there: six
> patches have been committed in the intervening time and seven remain
> in the queue).

I agree. It is time to move on.

Joshua D. Drake

> 
> ...Robert
> 
-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

Joshua D. Drake wrote:
> On Wed, 2009-02-25 at 22:56 -0500, Robert Haas wrote:
> 
>>> Since it's going to take us two weeks to clean up the other loose ends
>>> anyway, there's no harm in letting Simon and Heikki try to complete the
>>> patch by then.  But I'll happily lay a side bet with you about what the
>>> situation will be two weeks from now.
>> I think Tom wins this bet, since it's now been four weeks.  In fact,
>> he was being optimistic: the loose ends aren't cleaned up either
>> (based on a review of the wiki, we're about half way there: six
>> patches have been committed in the intervening time and seven remain
>> in the queue).
> 
> I agree. It is time to move on.

Huh? What features are you saying about to be moved on?
Is it the updatable-view feature? or rest of all pending ones?

Thanks,
-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

On Thu, 2009-02-26 at 14:17 +0900, KaiGai Kohei wrote:
> Joshua D. Drake wrote:
> > On Wed, 2009-02-25 at 22:56 -0500, Robert Haas wrote:
> > 
> >>> Since it's going to take us two weeks to clean up the other loose ends
> >>> anyway, there's no harm in letting Simon and Heikki try to complete the
> >>> patch by then.  But I'll happily lay a side bet with you about what the
> >>> situation will be two weeks from now.
> >> I think Tom wins this bet, since it's now been four weeks.  In fact,
> >> he was being optimistic: the loose ends aren't cleaned up either
> >> (based on a review of the wiki, we're about half way there: six
> >> patches have been committed in the intervening time and seven remain
> >> in the queue).
> > 
> > I agree. It is time to move on.
> 
> Huh? What features are you saying about to be moved on?
> Is it the updatable-view feature? or rest of all pending ones?

Tom originally stated (as I recall, no flames please) that we would wait
for 2 weeks for the hot standby stuff. It has now been four. That is
what I and I believe Robert Haas are talking about.

Sincerely,

Joshua D. Drake

> 
> Thanks,
-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

Joshua D. Drake wrote:
> On Thu, 2009-02-26 at 14:17 +0900, KaiGai Kohei wrote:
>> Joshua D. Drake wrote:
>>> On Wed, 2009-02-25 at 22:56 -0500, Robert Haas wrote:
>>>
>>>>> Since it's going to take us two weeks to clean up the other loose ends
>>>>> anyway, there's no harm in letting Simon and Heikki try to complete the
>>>>> patch by then.  But I'll happily lay a side bet with you about what the
>>>>> situation will be two weeks from now.
>>>> I think Tom wins this bet, since it's now been four weeks.  In fact,
>>>> he was being optimistic: the loose ends aren't cleaned up either
>>>> (based on a review of the wiki, we're about half way there: six
>>>> patches have been committed in the intervening time and seven remain
>>>> in the queue).
>>> I agree. It is time to move on.
>> Huh? What features are you saying about to be moved on?
>> Is it the updatable-view feature? or rest of all pending ones?
> 
> Tom originally stated (as I recall, no flames please) that we would wait
> for 2 weeks for the hot standby stuff. It has now been four. That is
> what I and I believe Robert Haas are talking about.

Thanks, it helps me clear.

P.S,
I would like folks not to forget SE-PostgreSQL.

-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

On Thu, 2009-02-26 at 14:43 +0900, KaiGai Kohei wrote:
> Joshua D. Drake wrote:

> > Tom originally stated (as I recall, no flames please) that we would wait
> > for 2 weeks for the hot standby stuff. It has now been four. That is
> > what I and I believe Robert Haas are talking about.
> 
> Thanks, it helps me clear.
> 
> P.S,
> I would like folks not to forget SE-PostgreSQL.
> 

I assure you that I have not.

Sincerely,

Joshua D. Drake

-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

Robert Haas wrote:
> On Tue, Jan 27, 2009 at 11:23 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> As for when it *will* be committable --- Heikki is saying two weeks
>> if no new problems crop up, but given the rate at which new problems
>> have been found so far, what are the odds of that?  We've seen this
>> movie before.
>>
>> Since it's going to take us two weeks to clean up the other loose ends
>> anyway, there's no harm in letting Simon and Heikki try to complete the
>> patch by then.  But I'll happily lay a side bet with you about what the
>> situation will be two weeks from now.
> 
> I think Tom wins this bet, since it's now been four weeks.  In fact,
> he was being optimistic: the loose ends aren't cleaned up either
> (based on a review of the wiki, we're about half way there: six
> patches have been committed in the intervening time and seven remain
> in the queue).

Agreed. Simon has finished the pending items he had four weeks ago, but 
the code clearly isn't ready for commit yet as new issues are cropping 
up. And I think the way subtransactions are handled, which has been a 
difficult part of the patch all along, still needs more thinking.

--   Heikki Linnakangas  EnterpriseDB   http://www.enterprisedb.com

Heikki,

> Agreed. Simon has finished the pending items he had four weeks ago, but 
> the code clearly isn't ready for commit yet as new issues are cropping 
> up. And I think the way subtransactions are handled, which has been a 
> difficult part of the patch all along, still needs more thinking.

Are there issues other than subtransactions?

--Josh

On Wed, 2009-02-25 at 22:11 -0800, Josh Berkus wrote:
> Heikki,
> 
> > Agreed. Simon has finished the pending items he had four weeks ago, but 
> > the code clearly isn't ready for commit yet as new issues are cropping 
> > up. And I think the way subtransactions are handled, which has been a 
> > difficult part of the patch all along, still needs more thinking.
> 
> Are there issues other than subtransactions?

I think his reply states that. The long and short is, what Tom was
concerned about is true and Heikki has confirmed it. This patch as nice
as it would be to have, isn't ready for prime time. It is time to push
this patch to 8.5, close up the rest of whatever is left with other
patches and move to Beta.

Sincerely,

Joshua D. Drake

> 
> --Josh
> 
> 
-- 
PostgreSQL - XMPP: jdrake@jabber.postgresql.org  Consulting, Development, Support, Training  503-667-4564 -
http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997
 

Josh Berkus wrote:
>> Agreed. Simon has finished the pending items he had four weeks ago, 
>> but the code clearly isn't ready for commit yet as new issues are 
>> cropping up. And I think the way subtransactions are handled, which 
>> has been a difficult part of the patch all along, still needs more 
>> thinking.
> 
> Are there issues other than subtransactions?

Subtransactions, and transaction tracking in general. I haven't looked 
at the other parts in detail yet.

--   Heikki Linnakangas  EnterpriseDB   http://www.enterprisedb.com

On Thu, Feb 26, 2009 at 7:09 AM, Joshua D. Drake <jd@commandprompt.com> wrote:

> I think his reply states that. The long and short is, what Tom was
> concerned about is true and Heikki has confirmed it. This patch as nice
> as it would be to have, isn't ready for prime time. It is time to push
> this patch to 8.5, close up the rest of whatever is left with other
> patches and move to Beta.

The decision on whether it is ready lies with Heikki, and whilst he is
prepared to work on it, and there are other patches still in the queue
being worked on by other committers there is no reason to bump it yet
(unless Heikki wants to look at the other patches).

I agree that this patch alone should not delay the release further though.

-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

On Thu, Feb 26, 2009 at 3:47 AM, Dave Page <dpage@pgadmin.org> wrote:
>> I think his reply states that. The long and short is, what Tom was
>> concerned about is true and Heikki has confirmed it. This patch as nice
>> as it would be to have, isn't ready for prime time. It is time to push
>> this patch to 8.5, close up the rest of whatever is left with other
>> patches and move to Beta.
>
> The decision on whether it is ready lies with Heikki, and whilst he is
> prepared to work on it, and there are other patches still in the queue
> being worked on by other committers there is no reason to bump it yet
> (unless Heikki wants to look at the other patches).

Well, this is a key point.  If Heikki thinks that the patch isn't
going to be ready for 8.4 (and it sounds like that's the case), then
it would be more beneficial to the project for him to lay it aside for
a few weeks and help clean out the rest of the patch queue rather than
continuing to work on that patch and leaving the other patches to the
other committers.  Our rate of progress on cleaning out the November
CommitFest seems to be asymptotically approaching zero (9 patches
committed in December, 6 in January, 4 so far in February...) so
benching one of the main committers for a feature that won't be ready
anyway is not a good trade-off.

Of course, if Heikki can't or isn't interested in working on any of
the remaining patches, then he might as well keep plugging away at Hot
Standby rather than leaving it for another day.  It can live in git
until 8.4 is released and then get merged after the tree is branched,
and anyone else who wants to do further development in that area can
get started knowing what the final shape of that code will be.

...Robert

Robert Haas wrote:
> On Thu, Feb 26, 2009 at 3:47 AM, Dave Page <dpage@pgadmin.org> wrote:
> >> I think his reply states that. The long and short is, what Tom was
> >> concerned about is true and Heikki has confirmed it. This patch as nice
> >> as it would be to have, isn't ready for prime time. It is time to push
> >> this patch to 8.5, close up the rest of whatever is left with other
> >> patches and move to Beta.
> >
> > The decision on whether it is ready lies with Heikki, and whilst he is
> > prepared to work on it, and there are other patches still in the queue
> > being worked on by other committers there is no reason to bump it yet
> > (unless Heikki wants to look at the other patches).
> 
> Well, this is a key point.  If Heikki thinks that the patch isn't
> going to be ready for 8.4 (and it sounds like that's the case), then
> it would be more beneficial to the project for him to lay it aside for
> a few weeks and help clean out the rest of the patch queue rather than
> continuing to work on that patch and leaving the other patches to the
> other committers.  Our rate of progress on cleaning out the November
> CommitFest seems to be asymptotically approaching zero (9 patches
> committed in December, 6 in January, 4 so far in February...) so
> benching one of the main committers for a feature that won't be ready
> anyway is not a good trade-off.

Agreed, we could use Heikki's help in closing the other outstanding
patches, and if hot standby isn't going to make 8.4, which I agree with,
it would be good for him to put hot standby aside and help on other
patches.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +