Thread: Heroku early upgrade is raising serious questions

I think we have a problem here :

https://status.heroku.com/incidents/510

Disclaimer : I don't know what thursday security fix is about and I
don't have much information on the "Heroku Postgres Official
Maintenance". So for now, I won't discuss wether or not Heroku should do
that upgrade earlier than everyone. This is why why I'm sending this on
pgsql-advocacy instead of pgsql-hackers

What I know is that Heroku's announcement is raising many questions all
over the place:

http://techcrunch.com/2013/04/01/heroku-forces-customer-upgrade-to-fix-critical-postgresql-security-hole/
https://news.ycombinator.com/item?id=5475619

Among these questions, the 3 below are recurring :

Which companies have access to the patch before the official release ?
What does a company have to do to have access to this patch ?
Who decides to allow this "early access" ?


Now my guess is that Heroku is treated here as a distributer such as Red
Hat, the Debian packagers, etc. Once again I am not discussing wether or
not they should have access to the code earlier.

What I am discussing is that most people consider that Heroku is a
"database as a service" company, not a distributor of software. And the
overall feeling among DBA can be described as :

"Why is Heroku so special ? Why do I have to wait 4 days while they are
allowed to upgrade before the security breach is fully disclosed ?"

In other words, we are sending a terrible message to our users. I
understand that this bug cannot be discussed in public but the Heroku
upgrade is public and therefore the PostgreSQL community needs to come
up with an explanation to make things clear and avoid misunderstandings
and frustration.

On Tue, Apr  2, 2013 at 11:41:46PM +0200, damien clochard wrote:
> What I am discussing is that most people consider that Heroku is a
> "database as a service" company, not a distributor of software. And the
> overall feeling among DBA can be described as :
>
> "Why is Heroku so special ? Why do I have to wait 4 days while they are
> allowed to upgrade before the security breach is fully disclosed ?"
>
> In other words, we are sending a terrible message to our users. I
> understand that this bug cannot be discussed in public but the Heroku
> upgrade is public and therefore the PostgreSQL community needs to come
> up with an explanation to make things clear and avoid misunderstandings
> and frustration.

We realize this issue has become public and the core team is planning to
post an updated set of rules on how major security releases are
distributed, probably on or shortly after the Thursday release.  I will
send this email to core so they are aware of it.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

> What I know is that Heroku's announcement is raising many questions all
> over the place:
>
> http://techcrunch.com/2013/04/01/heroku-forces-customer-upgrade-to-fix-critical-postgresql-security-hole/
> https://news.ycombinator.com/item?id=5475619

Just to keep this in scope, those are two places, and the first sources
the second, so basically "Hacker News is complaining".  I'll also point
out that many of the comments on the HN thread are supportive. Also,
contrast this Slashdot thread:

http://news.slashdot.org/story/13/03/29/1519208/security-fix-leads-to-postgresql-lock-down

... which praises us for taking reasonable security precautions as a
consensus of the comments.

> In other words, we are sending a terrible message to our users. I
> understand that this bug cannot be discussed in public but the Heroku
> upgrade is public and therefore the PostgreSQL community needs to come
> up with an explanation to make things clear and avoid misunderstandings
> and frustration.

I don't think this is as big of an issue as you seem to.  I do think we
should have some messaging around this, but I don't agree that it should
happen before Thursday, when we will be doing PR around the security
update anyway.

I'm also happy that we're getting all this press, because it means
people will actually apply the darned updates.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 04/02/2013 03:40 PM, Josh Berkus wrote:

>> In other words, we are sending a terrible message to our users. I
>> understand that this bug cannot be discussed in public but the Heroku
>> upgrade is public and therefore the PostgreSQL community needs to come
>> up with an explanation to make things clear and avoid misunderstandings
>> and frustration.
>
> I don't think this is as big of an issue as you seem to.  I do think we
> should have some messaging around this, but I don't agree that it should
> happen before Thursday, when we will be doing PR around the security
> update anyway.
>
> I'm also happy that we're getting all this press, because it means
> people will actually apply the darned updates.

I think the overriding point of concern here is that there is an
impression that somehow Heroku got special access to the fix before
anyone else. Of course this isn't true, but our communication as a
project has been sorely lacking this time around and this has caused
some confusion about what is actually going on.

Joshua D. Drake



--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On Apr 2, 2013, at 6:52 PM, Joshua D. Drake wrote:

> On 04/02/2013 03:40 PM, Josh Berkus wrote:
>
>>> In other words, we are sending a terrible message to our users. I
>>> understand that this bug cannot be discussed in public but the Heroku
>>> upgrade is public and therefore the PostgreSQL community needs to come
>>> up with an explanation to make things clear and avoid misunderstandings
>>> and frustration.
>>
>> I don't think this is as big of an issue as you seem to.  I do think we
>> should have some messaging around this, but I don't agree that it should
>> happen before Thursday, when we will be doing PR around the security
>> update anyway.
>>
>> I'm also happy that we're getting all this press, because it means
>> people will actually apply the darned updates.
>
> I think the overriding point of concern here is that there is an impression that somehow Heroku got special access to
thefix before anyone else. Of course this isn't true, but our communication as a project has been sorely lacking this
timearound and this has caused some confusion about what is actually going on. 

+1 - with a more outside perspective on the overall issue, I do have to say that I'm okay to any entity operating
"criticalinfrastructure" or the like having access to a critical security patch before the source is made available.  I
thinkto reiterate what JD said, we should just communicate that better in the future. 

* Jonathan S. Katz (jonathan.katz@excoventures.com) wrote:
> +1 - with a more outside perspective on the overall issue, I do have to say that I'm okay to any entity operating
"criticalinfrastructure" or the like having access to a critical security patch before the source is made available.  I
thinkto reiterate what JD said, we should just communicate that better in the future. 

Having some kind of documentation / policy regarding who can get access,
or what they have to do to get access, would certainly help address
these concerns.

For my 2c, I really don't feel this went very well and I absolutely
think that it's a black-eye that the common impression is that we gave
the fix to Heroku ahead of the public release.

Not sure what the best place to discuss this is, but, basically, I think
we can do better.

    Thanks,

        Stephen

On 03/04/2013 08:11, damien clochard wrote:

> Among these questions, the 3 below are recurring :
>
> Which companies have access to the patch before the official release
> ? What does a company have to do to have access to this patch ? Who
> decides to allow this "early access" ?

While no-one seems to address these questions - here's my opinion,

1. An issue is found, a solution is developed and committed to source
code repository.
2. Source is downloaded and compiled by release build servers.
3. Official release binaries are made available to download.

Once 1 is complete an announcement of upcoming release can be made.
There is a delay till release as the release builds are being compiled.

After 1. is complete anyone can download and build their own fixed
version of postgresql.

If an announcement is made after 1 then there is a delay that anyone can
use to build their own update before the *official release*

If an announcement is made after 3 then anyone also following the
development can also release their own updates before the official
announcement.

That's a benefit of open source.
There is no preference given to special customers.
Those that care and pay attention can look after their customers as
they see fit.

Jonathan,

> Here is a wiki I through together combining elements of both our
> current security page and thoughts from the Django one:

Thanks for getting this started!  I've revised it heavily.

> One suggestion (not in the draft) is that when we do make release
> announcements containing security fixes, we do include the URL to our
> security policy to make it clear what it is.

Actually, we usually do provide a link.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Le 03/04/2013 02:43, Jonathan S. Katz a écrit :
> On Apr 2, 2013, at 8:14 PM, Jonathan S. Katz wrote:
>
>> On Apr 2, 2013, at 8:03 PM, Selena Deckelmann wrote:
>>
>>> I agree that we should have a well-documented security release
>>> process. There are existing processes documented that we might use as
>>> a starting point, and I personally think largely match what we
>>> currently do, like:
>>> https://docs.djangoproject.com/en/1.5/internals/security/
>>
>> The Django security release guide is good - I think we could almost
>> copy & paste it.  I could throw something up on our wiki where we can
>> fill in the blanks on what we want the actually policy to be and allow
>> people to comment + add modifications.
>
> Here is a wiki I through together combining elements of both our current
> security page and thoughts from the Django one:
>
> https://wiki.postgresql.org/wiki/PostgreSQL_Security_Release_Policy_Draft
>
> I separated between our current policy and the draft.  The draft really
> needs some blanks to be filled in.
>

Thanks for this draft, it's an improvement !

Here's a few comments :

A/ I think the names of "The Packagers List" should be public. I think
it's an important infomation when you choose a distibution system or a
service provider. One should be able to check if a package/service
provider is connected to the Security Team or not.

B/ I feel that all "Packagers" should respect the "embargo date". They
should not produce the packages prior to the official realease. This is
what RPM and DEB packagers do and it's a good thing. Once again the
problem is not that Heroku had early access to the security fix. The
problem is that they "released" it 3 days before others packagers. I
don't know if they did that on purpose but the message they are sending
is "Heroku Postgres is more secure than vanilla PostgreSQL, because you
get upgrades before full disclosure"

C/ The Packagers list could be extended to companies providing
PostgreSQL support. If the term "Packagers" include not only
organizations that distribute the code but also organizations that
provide PostgreSQL as a services, then PostgreSQL Support services
should be included too.

 If you produce binaries you're not supposed to make them accessible
prior to the embargo date.

On Wed, Apr 3, 2013 at 3:55 AM, damien clochard <damien@dalibo.info> wrote:
>
>
> A/ I think the names of "The Packagers List" should be public. I think
> it's an important infomation when you choose a distibution system or a
> service provider. One should be able to check if a package/service
> provider is connected to the Security Team or not.

The packagers list and security team are different groups.

> B/ I feel that all "Packagers" should respect the "embargo date". They
> should not produce the packages prior to the official realease. This is
> what RPM and DEB packagers do and it's a good thing. Once again the
> problem is not that Heroku had early access to the security fix. The
> problem is that they "released" it 3 days before others packagers. I
> don't know if they did that on purpose but the message they are sending
> is "Heroku Postgres is more secure than vanilla PostgreSQL, because you
> get upgrades before full disclosure"

How would that work? The reason we have a number of days between the
tarballs being rolled and the embargo date is that it takes time to
build and properly QA the packages. In the case of the installers,
each branch gets tested on 30 - 40 different platforms in total. It is
simply not possible to "not produce the packages prior to the official
realease".

> C/ The Packagers list could be extended to companies providing
> PostgreSQL support. If the term "Packagers" include not only
> organizations that distribute the code but also organizations that
> provide PostgreSQL as a services, then PostgreSQL Support services
> should be included too.

No, most definitely not. The packagers list is a working/coordination
list, not one for discussion. We need to keep that list tightly
purposed and focussed on those actually creating packages for public
distribution and arguably in the future, deployment on public DBaaS
platforms (the key word in both cases, being "public").

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Apr 3, 2013 at 9:55 AM, damien clochard <damien@dalibo.info> wrote:
> Le 03/04/2013 02:43, Jonathan S. Katz a écrit :
>> On Apr 2, 2013, at 8:14 PM, Jonathan S. Katz wrote:
>>
>>> On Apr 2, 2013, at 8:03 PM, Selena Deckelmann wrote:
>>>
>>>> I agree that we should have a well-documented security release
>>>> process. There are existing processes documented that we might use as
>>>> a starting point, and I personally think largely match what we
>>>> currently do, like:
>>>> https://docs.djangoproject.com/en/1.5/internals/security/
>>>
>>> The Django security release guide is good - I think we could almost
>>> copy & paste it.  I could throw something up on our wiki where we can
>>> fill in the blanks on what we want the actually policy to be and allow
>>> people to comment + add modifications.
>>
>> Here is a wiki I through together combining elements of both our current
>> security page and thoughts from the Django one:
>>
>> https://wiki.postgresql.org/wiki/PostgreSQL_Security_Release_Policy_Draft
>>
>> I separated between our current policy and the draft.  The draft really
>> needs some blanks to be filled in.
>>
>
> Thanks for this draft, it's an improvement !
>
> Here's a few comments :
>
> A/ I think the names of "The Packagers List" should be public. I think
> it's an important infomation when you choose a distibution system or a
> service provider. One should be able to check if a package/service
> provider is connected to the Security Team or not.

Listing which packages, at least, seems reasonable. Doesn't have to be
the people, but wihch projects/packagies are included does.


> B/ I feel that all "Packagers" should respect the "embargo date". They
> should not produce the packages prior to the official realease. This is
> what RPM and DEB packagers do and it's a good thing. Once again the
> problem is not that Heroku had early access to the security fix. The
> problem is that they "released" it 3 days before others packagers. I
> don't know if they did that on purpose but the message they are sending
> is "Heroku Postgres is more secure than vanilla PostgreSQL, because you
> get upgrades before full disclosure"
>
> C/ The Packagers list could be extended to companies providing
> PostgreSQL support. If the term "Packagers" include not only
> organizations that distribute the code but also organizations that
> provide PostgreSQL as a services, then PostgreSQL Support services
> should be included too.

In that case, you can just make it public in the first place. Any
company can claim to do postgres support. There are thousands of them
out there that do, at a lower level.


>  If you produce binaries you're not supposed to make them accessible
> prior to the embargo date.

Yes.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

>>
>> Here's a few comments :
>>
>> A/ I think the names of "The Packagers List" should be public. I think
>> it's an important infomation when you choose a distibution system or a
>> service provider. One should be able to check if a package/service
>> provider is connected to the Security Team or not.
>
> Listing which packages, at least, seems reasonable. Doesn't have to be
> the people, but wihch projects/packagies are included does.
>

Yes this is what I meant : Listing the names of organization/companies
inside the Packagers List.

>
>> B/ I feel that all "Packagers" should respect the "embargo date". They
>> should not produce the packages prior to the official realease. This is
>> what RPM and DEB packagers do and it's a good thing. Once again the
>> problem is not that Heroku had early access to the security fix. The
>> problem is that they "released" it 3 days before others packagers. I
>> don't know if they did that on purpose but the message they are sending
>> is "Heroku Postgres is more secure than vanilla PostgreSQL, because you
>> get upgrades before full disclosure"
>>
>> C/ The Packagers list could be extended to companies providing
>> PostgreSQL support. If the term "Packagers" include not only
>> organizations that distribute the code but also organizations that
>> provide PostgreSQL as a services, then PostgreSQL Support services
>> should be included too.
>
> In that case, you can just make it public in the first place. Any
> company can claim to do postgres support. There are thousands of them
> out there that do, at a lower level.
>

Yes just like anyone can claim to build its own distro or a "cloud
database". Actually it's even easier to claim you do DBaaS than
pretending to offer PostgreSQL support :-)

I never said the list should be extended to anyone asking. The Packagers
List needs to stay small and the Security Team is free to reject
requests that don't seem appropriate.

All I'm saying is that the difference between a DBaaS plateform and a
Production Support provider can be very thin. Some PostgreSQL companies
high level support including remote admin, monitoring, upgrades, etc. At
this level of service the difference with a cloud database is just the
location of the server.

Le 03/04/2013 10:07, Dave Page a écrit :
> On Wed, Apr 3, 2013 at 3:55 AM, damien clochard <damien@dalibo.info> wrote:
>>
>>
>> A/ I think the names of "The Packagers List" should be public. I think
>> it's an important infomation when you choose a distibution system or a
>> service provider. One should be able to check if a package/service
>> provider is connected to the Security Team or not.
>
> The packagers list and security team are different groups.
>

Yes i'm talking of the packagers list.

>> B/ I feel that all "Packagers" should respect the "embargo date". They
>> should not produce the packages prior to the official realease. This is
>> what RPM and DEB packagers do and it's a good thing. Once again the
>> problem is not that Heroku had early access to the security fix. The
>> problem is that they "released" it 3 days before others packagers. I
>> don't know if they did that on purpose but the message they are sending
>> is "Heroku Postgres is more secure than vanilla PostgreSQL, because you
>> get upgrades before full disclosure"
>
> How would that work? The reason we have a number of days between the
> tarballs being rolled and the embargo date is that it takes time to
> build and properly QA the packages. In the case of the installers,
> each branch gets tested on 30 - 40 different platforms in total. It is
> simply not possible to "not produce the packages prior to the official
> realease".
>

Ok maybe I was not clear enough here. With the word "produce" I meant
"making available to public". I'm awara the packagers need time to build
and test their packages.

What I am saying is that the packagers should not release publicly the
packages before the official release date.

I feel that if Red Hat had released the new RPM on monday, many people
here would have been unpleased. So why can Heroku do it ?


>> C/ The Packagers list could be extended to companies providing
>> PostgreSQL support. If the term "Packagers" include not only
>> organizations that distribute the code but also organizations that
>> provide PostgreSQL as a services, then PostgreSQL Support services
>> should be included too.
>
> No, most definitely not. The packagers list is a working/coordination
> list, not one for discussion. We need to keep that list tightly
> purposed and focussed on those actually creating packages for public
> distribution and arguably in the future, deployment on public DBaaS
> platforms (the key word in both cases, being "public").
>

Meh. What do you mean by "public" ? To me something that is "available
to everyone" or "open to general view". If you include paying services
sucha as Red Hat and Heroku in this "public" definition, than I guess
PostgreSQL support company is "public" too ? Where's the difference ?

Once again I'm not arguing that the packagers list should be extended to
anyone asking. But if we include "PostgreSQL as a service" in this list
than we need to come up with a precise definition of what "PostgreSQL
service" means...

On Wed, Apr 3, 2013 at 4:48 AM, damien clochard <damien@dalibo.info> wrote:
>
>> How would that work? The reason we have a number of days between the
>> tarballs being rolled and the embargo date is that it takes time to
>> build and properly QA the packages. In the case of the installers,
>> each branch gets tested on 30 - 40 different platforms in total. It is
>> simply not possible to "not produce the packages prior to the official
>> realease".
>>
>
> Ok maybe I was not clear enough here. With the word "produce" I meant
> "making available to public". I'm awara the packagers need time to build
> and test their packages.
>
> What I am saying is that the packagers should not release publicly the
> packages before the official release date.

Right, I agree with that.

>> No, most definitely not. The packagers list is a working/coordination
>> list, not one for discussion. We need to keep that list tightly
>> purposed and focussed on those actually creating packages for public
>> distribution and arguably in the future, deployment on public DBaaS
>> platforms (the key word in both cases, being "public").
>>
>
> Meh. What do you mean by "public" ? To me something that is "available
> to everyone" or "open to general view". If you include paying services
> sucha as Red Hat and Heroku in this "public" definition, than I guess
> PostgreSQL support company is "public" too ? Where's the difference ?

PostgreSQL support companies do not generally produce PostgreSQL
binary packages that are available for anyone to use (for a service
fee or otherwise) either via download or on a platform like a cloud
service. There are a handful of exceptions to that rule (EDB for
example, as we produce the installers), but most, if not all of those
companies are on the packagers list already.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Apr 03, 2013 at 05:06:08AM -0400, Dave Page wrote:
> PostgreSQL support companies do not generally produce PostgreSQL
> binary packages that are available for anyone to use (for a service
> fee or otherwise) either via download or on a platform like a cloud
> service. There are a handful of exceptions to that rule (EDB for
> example, as we produce the installers), but most, if not all of those
> companies are on the packagers list already.

So that means if said support company creates packages for its customers it
should be on the packagers list? After all anyone could get the packages from
that company, couldn't they? Is there a any description as to who is eligible
for the packages list? And of course I take it there is a code of conduct for
this list, albeit Heroku didn't honor that one.

Michael

--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at gmail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On Wed, Apr 3, 2013 at 5:31 AM, Michael Meskes <meskes@postgresql.org> wrote:
> On Wed, Apr 03, 2013 at 05:06:08AM -0400, Dave Page wrote:
>> PostgreSQL support companies do not generally produce PostgreSQL
>> binary packages that are available for anyone to use (for a service
>> fee or otherwise) either via download or on a platform like a cloud
>> service. There are a handful of exceptions to that rule (EDB for
>> example, as we produce the installers), but most, if not all of those
>> companies are on the packagers list already.
>
> So that means if said support company creates packages for its customers it
> should be on the packagers list? After all anyone could get the packages from
> that company, couldn't they? Is there a any description as to who is eligible
> for the packages list?

First; I'm giving about my personal opinion at the moment, not
representing -core.

I do not believe that regular support companies should be included,
because there are too many of them, and they will likely be packaging
for a very small audience who in most cases could easily be using the
community packages. With so many people on the list, security and
confidentiality becomes impossible to enforce.

I support having the packagers of the mainstream packages on the list,
e.g. installers, RPMs, DEBs, Postgres.app, OS vendor packages etc
(e.g. Palle who provides the FreeBSD ports) etc.

I also support having the large scale DBaaS providers on the list, as
they provide Postgres instances for thousands of users, very publicly
- Heroku, as the obvious example, have hundreds of thousands of
databases on their platform.

> And of course I take it there is a code of conduct for
> this list, albeit Heroku didn't honor that one.

Let me state this very clearly:

*** Heroku have done nothing wrong ***

I cannot go into details at the moment, but their actions have been
taken following talks with the core team, in a difficult time, with no
precedence within the community to follow and very little time for
in-depth discussion. We have had similar discussions with other large
DBaaS providers, who have different architectures with different
implications to consider.

In hindsight, I'm sure the rest of core will agree we might have
handled this better in some respects, but as we all know, hindsight is
a wonderful thing. We will be working on policies to guide us in the
future in the event that something similar happens again (and as
you've probably seen, that's already started).

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Apr 03, 2013 at 06:14:25AM -0400, Dave Page wrote:
> I do not believe that regular support companies should be included,
> because there are too many of them, and they will likely be packaging
> for a very small audience who in most cases could easily be using the
> community packages. With so many people on the list, security and
> confidentiality becomes impossible to enforce.

I do agree on the confidentiality problem, I do not agree in the "in most cases could easily be using the
community packages" part. Honestly, why would anyone pay a support company to
build packages if they can use the free community ones instead? Ok, maybe some
do because they simple don't know, but in general itÄs because they have some
special needs.

> I support having the packagers of the mainstream packages on the list,
> e.g. installers, RPMs, DEBs, Postgres.app, OS vendor packages etc
> (e.g. Palle who provides the FreeBSD ports) etc.
>
> I also support having the large scale DBaaS providers on the list, as
> they provide Postgres instances for thousands of users, very publicly
> - Heroku, as the obvious example, have hundreds of thousands of
> databases on their platform.

So then it's a matter of users? What is the number of user one has to have to
qualify? How do we count users? Installations, db admins, real database users
as in customers? As a support company (bad wording actually as support doesn't
mean you're creating packages) serving a customers with thousands of instances,
do I qualify?

Don't get me wrong, I'm not complaining and I don't have a ready made solution
for this, but I do strongly believe we should come up with a set of rules, that
are discussed and decided upon in the open. We definitely don't want this to
look like a (strange?) decision made behind closed doors, or do we?

> Let me state this very clearly:
>
> *** Heroku have done nothing wrong ***

That much I learned so far, sorry for accusing Heroku. It seems that press
report wasn't theirs and they received permission to start deploying early.
Although again I wonder why all this decision making hasn't been open. Why are
their customers more important than ours or EDB's or any other company that
hasn't rolled out the patches yet.

> I cannot go into details at the moment, but their actions have been

Why? I can see a reason why we don't talk about the bug or the fix in the open.
Sure that makes sense because we have to have the fixed version out first. But
why does the same hold for communication about deployment embargo?

> taken following talks with the core team, in a difficult time, with no
> precedence within the community to follow and very little time for

You mean the PostgreSQL community, right? We're not the first project that
discovers a nasty security hole. And we won't be the last.

> in-depth discussion. We have had similar discussions with other large
> DBaaS providers, who have different architectures with different
> implications to consider.

Sure and those details don't belong into the open.

> In hindsight, I'm sure the rest of core will agree we might have
> handled this better in some respects, but as we all know, hindsight is
> a wonderful thing. We will be working on policies to guide us in the
> future in the event that something similar happens again (and as
> you've probably seen, that's already started).

Yes, hindsight is always 20/20. Again, I hope you see my email is part of a
constructive discussion to get to a better policy for the next time, that
hopefully will never come. :)

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at gmail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On Wed, Apr 3, 2013 at 1:22 PM, Michael Meskes <meskes@postgresql.org> wrote:
> On Wed, Apr 03, 2013 at 06:14:25AM -0400, Dave Page wrote:
>> I cannot go into details at the moment, but their actions have been
>
> Why? I can see a reason why we don't talk about the bug or the fix in the open.
> Sure that makes sense because we have to have the fixed version out first. But
> why does the same hold for communication about deployment embargo?

Because talking about it in public in a way to make it make sense,
would leak information about what and where the bug is, and thus give
people who are looking to exploit it a much easier job in finding it
before people have had a chance to apply the patches.

If you are willing to wait a few days until such details can be made
public, there is no reason why we can't talk about it in the open -
and we should. But for now, the risk of actually putting all users at
risk because someone uses that information to figure out where exactly
the bug is before the patches are applied is pretty big.


>> taken following talks with the core team, in a difficult time, with no
>> precedence within the community to follow and very little time for
>
> You mean the PostgreSQL community, right? We're not the first project that
> discovers a nasty security hole. And we won't be the last.

Yes.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Wed, 2013-04-03 at 06:14 -0400, Dave Page wrote:
> On Wed, Apr 3, 2013 at 5:31 AM, Michael Meskes <meskes@postgresql.org> wrote:
> > On Wed, Apr 03, 2013 at 05:06:08AM -0400, Dave Page wrote:
> >> PostgreSQL support companies do not generally produce PostgreSQL
> >> binary packages that are available for anyone to use (for a service
> >> fee or otherwise) either via download or on a platform like a cloud
> >> service. There are a handful of exceptions to that rule (EDB for
> >> example, as we produce the installers), but most, if not all of those
> >> companies are on the packagers list already.
> >
> > So that means if said support company creates packages for its customers it
> > should be on the packagers list? After all anyone could get the packages from
> > that company, couldn't they? Is there a any description as to who is eligible
> > for the packages list?
>
> First; I'm giving about my personal opinion at the moment, not
> representing -core.
>
> I do not believe that regular support companies should be included,
> because there are too many of them, and they will likely be packaging
> for a very small audience who in most cases could easily be using the
> community packages. With so many people on the list, security and
> confidentiality becomes impossible to enforce.
>
> I support having the packagers of the mainstream packages on the list,
> e.g. installers, RPMs, DEBs, Postgres.app, OS vendor packages etc
> (e.g. Palle who provides the FreeBSD ports) etc.
>
> I also support having the large scale DBaaS providers on the list, as
> they provide Postgres instances for thousands of users, very publicly
> - Heroku, as the obvious example, have hundreds of thousands of
> databases on their platform.
>
> > And of course I take it there is a code of conduct for
> > this list, albeit Heroku didn't honor that one.
>
> Let me state this very clearly:
>
> *** Heroku have done nothing wrong ***
>
> I cannot go into details at the moment, but their actions have been
> taken following talks with the core team, in a difficult time, with no
> precedence within the community to follow and very little time for
> in-depth discussion. We have had similar discussions with other large
> DBaaS providers, who have different architectures with different
> implications to consider.
>
> In hindsight, I'm sure the rest of core will agree we might have
> handled this better in some respects, but as we all know, hindsight is
> a wonderful thing. We will be working on policies to guide us in the
> future in the event that something similar happens again (and as
> you've probably seen, that's already started).
>

FWIW, I completely agree with Dave. Kudos to -core and the security team
for handling this.


--
Guillaume
http://blog.guillaume.lelarge.info
http://www.dalibo.com

On Wed, Apr 03, 2013 at 01:26:22PM +0200, Magnus Hagander wrote:
> > Why? I can see a reason why we don't talk about the bug or the fix in the open.
> > Sure that makes sense because we have to have the fixed version out first. But
> > why does the same hold for communication about deployment embargo?
>
> Because talking about it in public in a way to make it make sense,
> would leak information about what and where the bug is, and thus give
> people who are looking to exploit it a much easier job in finding it
> before people have had a chance to apply the patches.

I wasn't talking about the discussion about the bug etc., I was just talking
about the discussion about the permission to deploy. But if these were so
tightly intervened I will gladly wait.

> If you are willing to wait a few days until such details can be made
> public, there is no reason why we can't talk about it in the open -
> and we should. But for now, the risk of actually putting all users at
> risk because someone uses that information to figure out where exactly
> the bug is before the patches are applied is pretty big.

Sure, thanks.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at gmail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On Wed, Apr 3, 2013 at 1:49 PM, Michael Meskes <meskes@postgresql.org> wrote:
> On Wed, Apr 03, 2013 at 01:26:22PM +0200, Magnus Hagander wrote:
>> > Why? I can see a reason why we don't talk about the bug or the fix in the open.
>> > Sure that makes sense because we have to have the fixed version out first. But
>> > why does the same hold for communication about deployment embargo?
>>
>> Because talking about it in public in a way to make it make sense,
>> would leak information about what and where the bug is, and thus give
>> people who are looking to exploit it a much easier job in finding it
>> before people have had a chance to apply the patches.
>
> I wasn't talking about the discussion about the bug etc., I was just talking
> about the discussion about the permission to deploy. But if these were so
> tightly intervened I will gladly wait.

If you want an explanation for exactly what was done this time, and
why, then yes, that's hard to do without explaining the whole thing.
Which would leak it.


>> If you are willing to wait a few days until such details can be made
>> public, there is no reason why we can't talk about it in the open -
>> and we should. But for now, the risk of actually putting all users at
>> risk because someone uses that information to figure out where exactly
>> the bug is before the patches are applied is pretty big.
>
> Sure, thanks.

For the record, we have no intention whatsoever to keep any of this
information secret past the embargo date. Never had.


--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Wed, Apr 3, 2013 at 7:22 AM, Michael Meskes <meskes@postgresql.org> wrote:
> On Wed, Apr 03, 2013 at 06:14:25AM -0400, Dave Page wrote:
>> I do not believe that regular support companies should be included,
>> because there are too many of them, and they will likely be packaging
>> for a very small audience who in most cases could easily be using the
>> community packages. With so many people on the list, security and
>> confidentiality becomes impossible to enforce.
>
> I do agree on the confidentiality problem, I do not agree in the "in most cases could easily be using the
> community packages" part. Honestly, why would anyone pay a support company to
> build packages if they can use the free community ones instead? Ok, maybe some
> do because they simple don't know, but in general itÄs because they have some
> special needs.

Even if you put that aside though, there are nearly 300
support/services companies in our directory, most of whom are likely
to be unknown to the majority of us. Obviously there's no way we would
include all of them on the -packagers list, so how do we decide
fairly?

>> I support having the packagers of the mainstream packages on the list,
>> e.g. installers, RPMs, DEBs, Postgres.app, OS vendor packages etc
>> (e.g. Palle who provides the FreeBSD ports) etc.
>>
>> I also support having the large scale DBaaS providers on the list, as
>> they provide Postgres instances for thousands of users, very publicly
>> - Heroku, as the obvious example, have hundreds of thousands of
>> databases on their platform.
>
> So then it's a matter of users? What is the number of user one has to have to
> qualify? How do we count users? Installations, db admins, real database users
> as in customers?

Yes, that is what needs debate. I don't know how we can write down
specific criteria. I do know that when we tried to do something
similar to define who goes on the sponsors page, we got nowhere at all
because it's *really* hard to write such rules, without almost
immediately finding some exception that we'd have to make.

>> I cannot go into details at the moment, but their actions have been
>
> Why? I can see a reason why we don't talk about the bug or the fix in the open.
> Sure that makes sense because we have to have the fixed version out first. But
> why does the same hold for communication about deployment embargo?

Magnus has answered that nicely, so I won't repeat him.

>> taken following talks with the core team, in a difficult time, with no
>> precedence within the community to follow and very little time for
>
> You mean the PostgreSQL community, right?

Yes.

> Yes, hindsight is always 20/20. Again, I hope you see my email is part of a
> constructive discussion to get to a better policy for the next time, that
> hopefully will never come. :)

Of course - I've known you long enough to expect nothing less :-)

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Apr 03, 2013 at 07:52:33AM -0400, Dave Page wrote:
> Even if you put that aside though, there are nearly 300
> support/services companies in our directory, most of whom are likely
> to be unknown to the majority of us. Obviously there's no way we would
> include all of them on the -packagers list, so how do we decide
> fairly?

Yep, that's the tricky point. And we may run into the same problem DBaaS when
more companies offer a hosted PostgreSQL version.

> > So then it's a matter of users? What is the number of user one has to have to
> > qualify? How do we count users? Installations, db admins, real database users
> > as in customers?
>
> Yes, that is what needs debate. I don't know how we can write down
> specific criteria. I do know that when we tried to do something
> similar to define who goes on the sponsors page, we got nowhere at all
> because it's *really* hard to write such rules, without almost
> immediately finding some exception that we'd have to make.

True.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at gmail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

>
> On Wed, Apr 3, 2013 at 1:49 PM, Michael Meskes <meskes@postgresql.org>
> wrote:
>> On Wed, Apr 03, 2013 at 01:26:22PM +0200, Magnus Hagander wrote:
>>> > Why? I can see a reason why we don't talk about the bug or the fix in
>>> the open.
>>> > Sure that makes sense because we have to have the fixed version out
>>> first. But
>>> > why does the same hold for communication about deployment embargo?
>>>
>>> Because talking about it in public in a way to make it make sense,
>>> would leak information about what and where the bug is, and thus give
>>> people who are looking to exploit it a much easier job in finding it
>>> before people have had a chance to apply the patches.
>>
>> I wasn't talking about the discussion about the bug etc., I was just
>> talking
>> about the discussion about the permission to deploy. But if these were
>> so
>> tightly intervened I will gladly wait.
>
> If you want an explanation for exactly what was done this time, and
> why, then yes, that's hard to do without explaining the whole thing.
> Which would leak it.
>
>
>>> If you are willing to wait a few days until such details can be made
>>> public, there is no reason why we can't talk about it in the open -
>>> and we should. But for now, the risk of actually putting all users at
>>> risk because someone uses that information to figure out where exactly
>>> the bug is before the patches are applied is pretty big.
>>
>> Sure, thanks.
>
> For the record, we have no intention whatsoever to keep any of this
> information secret past the embargo date. Never had.



In my opinion we should STRENGTHEN early warning system as core members,
with greater visibility in communities. This will help us to manage
information to regard.


Saludos,
Gilberto Castillo
La Habana, Cuba
---
This message was processed by Kaspersky Mail Gateway 5.6.28/RELEASE running at host imx3.etecsa.cu
Visit our web-site: <http://www.kaspersky.com>, <http://www.viruslist.com>

Hi Josh,

On Apr 3, 2013, at 12:57 AM, Josh Berkus wrote:

> Jonathan,
>
>> Here is a wiki I through together combining elements of both our
>> current security page and thoughts from the Django one:
>
> Thanks for getting this started!  I've revised it heavily.

Thanks for working on it - it looks very good overall.

My one question regarding policy is related to distribution.  I do agree with the evaluation criteria for choosing
distributors,but my question pertains to entities that could be classified as "critical infrastructure" that use
Postgres,e.g. utilities, hospitals, etc.  Though it is still up to the management of those entities to handle the
upgrades,I think it would be in their best interests to have a critical security fix available to them so they have
thatopportunity before it goes live. 

I also presume that these organizations receive their releases from distributors - so if we were to enable such
organizationsto also receive an early release, what would the policy be? 

>> One suggestion (not in the draft) is that when we do make release
>> announcements containing security fixes, we do include the URL to our
>> security policy to make it clear what it is.
>
> Actually, we usually do provide a link.

I've looked through the news announcements to the last few releases.  There are links to the versioning policy and if
thereis a CVE a link to the CVE listing site itself, but nothing pointing to our security policy.  I strongly suggest
weadd that link to our template (don't know where that exists) and make sure it's in any future email pertaining to a
securityannouncement and/or release. 

Jonathan

> My one question regarding policy is related to distribution.  I do
> agree with the evaluation criteria for choosing distributors, but my
> question pertains to entities that could be classified as "critical
> infrastructure" that use Postgres, e.g. utilities, hospitals, etc.
> Though it is still up to the management of those entities to handle
> the upgrades, I think it would be in their best interests to have a
> critical security fix available to them so they have that opportunity
> before it goes live.
>
> I also presume that these organizations receive their releases from
> distributors - so if we were to enable such organizations to also
> receive an early release, what would the policy be?

There's a whole set of questions regarding early access to security
updates which we're not yet ready to tackle, and may never be ready to
tackle.  This includes:

- large commercial support vendors (e.g. SRA)
- distributors of embedded Postgres (on devices) (e.g. Apple)
- critical infrastructure users (e.g. the FAA)
- large-scale end users with high security profiles (e.g. Enova)

All of the above have legitimate, and sometimes compelling, reasons to
need to be able to apply security updates in advance of them becoming
public.  Deciding who gets to be on an early notification list and who
doesn't, while keeping the list small enough to not effectively make
things public, will be very hard and potentially impossible. And
ultimately we are a non-profit, volunteer project and can't devote 100
full time staff to managing security disclosure the way Microsoft can.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 04/08/2013 11:49 AM, Jonathan S. Katz wrote:
> On Apr 3, 2013, at 1:46 PM, Josh Berkus wrote:

> Josh - I think your points are valid concerning the vetting of who would
> be included in early releases.  I believe the best way to address who
> would be on that list is having a committee to vet those applications -
> I believe that is similar to how other OSS communities handle it.  I do
> not think the amount of submissions for requesting early access would be
> so great that we would need a full-time team to maintain it, and I think
> most of us have a good idea already about which types of organizations
> truly would need an early access release.
>
> With that said, if there are no overwhelming objections over the next 36
> hours, I can submit a patch to our security policy on the website using
> the language that is in the wiki above.

Since it is -core that primarily decides when things will be released,
it seems that we would need their approval for this policy?

Joshua D. Drake


>
> Jonathan
>
>


--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On Apr 8, 2013, at 2:53 PM, Joshua D. Drake wrote:

>
> On 04/08/2013 11:49 AM, Jonathan S. Katz wrote:
>> On Apr 3, 2013, at 1:46 PM, Josh Berkus wrote:
>
>> Josh - I think your points are valid concerning the vetting of who would
>> be included in early releases.  I believe the best way to address who
>> would be on that list is having a committee to vet those applications -
>> I believe that is similar to how other OSS communities handle it.  I do
>> not think the amount of submissions for requesting early access would be
>> so great that we would need a full-time team to maintain it, and I think
>> most of us have a good idea already about which types of organizations
>> truly would need an early access release.
>>
>> With that said, if there are no overwhelming objections over the next 36
>> hours, I can submit a patch to our security policy on the website using
>> the language that is in the wiki above.
>
> Since it is -core that primarily decides when things will be released, it seems that we would need their approval for
thispolicy? 

Well, that would be valid.  Could someone in -core please put the policy through the proper procedural work for
approval?

Jonathan

>> Since it is -core that primarily decides when things will be released, it seems that we would need their approval
forthis policy? 
>
> Well, that would be valid.  Could someone in -core please put the policy through the proper procedural work for
approval?

DIscussion on -core currently in progress.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

>
> Now that a few days have passed, I'd like to revisit this before too
> much time lapses.
>
> (The link again for the security policy
> draft: https://wiki.postgresql.org/wiki/PostgreSQL_Security_Release_Policy_Draft)
>

Jonathan,

Thanks for this page again !

I would like to add a paragraph about the release date (or "embargo
date"). It seems important to me that all packagers agree to synchronize
and distribute/deploy the security fix at the same date. For packager
who distribute the source code this is obvious. But that's also true for
DBaaS providers.

The Heroku announcement caused many confusions. The worst confusion is
that it sounds like Heroku gets a special treament and is allowed to
upgrade 3 days before full disclosure, while the rest of us have to wait
the official release date.

So basically the message we're sending is : Heroku Postgres is safer
than Vanilla PostgreSQL because in case of an high-exposure security
vulnerability, Heroku will upgrade before everyone else.

BTW you can replace Heroku by the DBaaS provider of your choice... I
have nothing against Heroku and I have great respect for the
contribution to our community.

I'm taking them as an exemple, because they've been very transparent
about all this (see
https://blog.heroku.com/archives/2013/4/4/heroku_postgres_databases_patched)
and that's a good thing because it helps us improving our Security
Release Policy.

Now I understand that Heroku (and other DBaaS providers) may host
hundreds of thousand PostgreSQL servers and I understand that upgrading
so many servers in a few hours is something very hard to acheive. But
the responsability of building a security maintenance process like that
is on Heroku (and other DBaaS providers). The PostgreSQL community
should keep some neutrality and should not compensate the lack of
upgrade machinery of a private company. Even if that means thousand of
their customers will be exposed for a while.

> I would like to add a paragraph about the release date (or "embargo
> date"). It seems important to me that all packagers agree to synchronize
> and distribute/deploy the security fix at the same date. For packager
> who distribute the source code this is obvious. But that's also true for
> DBaaS providers.

Very good point.

> The Heroku announcement caused many confusions. The worst confusion is
> that it sounds like Heroku gets a special treament and is allowed to
> upgrade 3 days before full disclosure, while the rest of us have to wait
> the official release date.
>
> So basically the message we're sending is : Heroku Postgres is safer
> than Vanilla PostgreSQL because in case of an high-exposure security
> vulnerability, Heroku will upgrade before everyone else.

It was the most expected response from users, I think.

> BTW you can replace Heroku by the DBaaS provider of your choice... I
> have nothing against Heroku and I have great respect for the
> contribution to our community.
>
> I'm taking them as an exemple, because they've been very transparent
> about all this (see
> https://blog.heroku.com/archives/2013/4/4/heroku_postgres_databases_patched)
> and that's a good thing because it helps us improving our Security
> Release Policy.
>
> Now I understand that Heroku (and other DBaaS providers) may host
> hundreds of thousand PostgreSQL servers and I understand that upgrading
> so many servers in a few hours is something very hard to acheive. But
> the responsability of building a security maintenance process like that
> is on Heroku (and other DBaaS providers). The PostgreSQL community
> should keep some neutrality and should not compensate the lack of
> upgrade machinery of a private company. Even if that means thousand of
> their customers will be exposed for a while.

Agreed.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

On Apr 8, 2013, at 5:27 PM, damien clochard wrote:

>> Now that a few days have passed, I'd like to revisit this before too
>> much time lapses.
>>
>> (The link again for the security policy
>> draft: https://wiki.postgresql.org/wiki/PostgreSQL_Security_Release_Policy_Draft)
>>
>
> Now I understand that Heroku (and other DBaaS providers) may host
> hundreds of thousand PostgreSQL servers and I understand that upgrading
> so many servers in a few hours is something very hard to acheive. But
> the responsability of building a security maintenance process like that
> is on Heroku (and other DBaaS providers). The PostgreSQL community
> should keep some neutrality and should not compensate the lack of
> upgrade machinery of a private company. Even if that means thousand of
> their customers will be exposed for a while.

I do not entirely agree with this.  Applications that run as critical infrastructure (examples again: hospitals,
utilities,etc.) should be permitted early access to releases - as a community, we would not want to expose critical
infrastructureto unnecessary risk while there is a fix available prior to public disclosure of a bug.  I think most
peoplehave good judgment and understanding that certain groups should have early access to a potentially dangerous
softwareissue as it could end up causing much larger issues should the software be exploited.  I think if we as a
communitycommunicate this message well, the people who have issues with some groups having early access would be
minimal.

In this specific case, DBaaS providers were exposed to a bug that is relatively easy to exploit with potentially dire
consequencesthat could potentially ruin many, many businesses (I do not want to give a bad estimate, so I won't provide
anumber).  Let's say this horrible scenario happened: sure, people could say that a DBaaS provider did not adequately
securetheir system, but fingers could also be pointed at the community for a) having a security hole in the first place
(asludicrous as that sounds to us as we know that software is flawed AND Postgres has an *excellent* track record for
security)and b) not recognizing the damage that could be caused by not permitting systems considered to be "critical
infrastructure"early access to a fix. 

In an ideal world - yes - everyone should have access to critical security bugfixes at identical times.  However, due
tothe sensitivity of certain systems (I do not think any of us would want to see a hospital's system go down due to a
securityexploit), there does need to be an early access list.  The way keep it fair for that is to have a committee of
variousPGDG members with different backgrounds to review applications and vote to decide who should have early access
(whereaccess is not even necessarily the source - access could even be just to the binaries).  I know one of the goals
isto keep that list small, and I do agree with that, but more importantly, there should be guidelines on how to
maintainmembership to that list. 

Damien,

> The Heroku announcement caused many confusions. The worst confusion is
> that it sounds like Heroku gets a special treament and is allowed to
> upgrade 3 days before full disclosure, while the rest of us have to wait
> the official release date.

So Heroku had permission from the core team to start their update early,
partly because of required deployment times, and partly because they
could supply testing and feedback on the patch, as they have with other
patches they've backported from future versions of PostgreSQL.  We were
also conscious of the fact that, as far as we knew, Heroku represented
the single largest organizational vulnerability to this particular
issue, and that due to their "port-only access" the possibility of
accidental disclosure was minimal.

We didn't anticipate that the early notification we did combined with
the Heroku outage notification would make it obvious that an early
deployment was happening.  We don't generally do early warnings, and
this whole "cloud" thing is still new to us organizationally.  ALso, we
went from discovery to release in 3 weeks, so there wasn't a lot of
discussion time around policy and procedure.

Clearly we can't do it that way again.

-core is currently hashing out thoughts on what might be a reasonable
early notification process for high-risk users, and if it's feasible for
us to have one.  As well as other aspects of our security release procedure.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Mon, Apr 8, 2013 at 06:58:57PM -0400, Jonathan S. Katz wrote:
> In an ideal world - yes - everyone should have access to critical
> security bugfixes at identical times.  However, due to the sensitivity
> of certain systems (I do not think any of us would want to see a
> hospital's system go down due to a security exploit), there does need
> to be an early access list.  The way keep it fair for that is to have
> a committee of various PGDG members with different backgrounds to
> review applications and vote to decide who should have early access
> (where access is not even necessarily the source - access could even
> be just to the binaries).  I know one of the goals is to keep that
> list small, and I do agree with that, but more importantly, there
> should be guidelines on how to maintain membership to that list.

Good analysis --- let me add a few things.  First, as others have
pointed out, Heroku did nothing wrong.  They followed the rules set by
the core/security teams --- if they had been given different rules, they
would have acted differently --- so any blame on the outcome has to be
assigned to the core/security teams.

Second, not only is Heroku uniquely exposed to the bug, they don't
supply source or even binaries to users, hence an early deployment had
little risk of a security leak.  (They also produce a custom build with
additional features, e.g. JSON.)  Also, they did a rolling deployment
and provided daily reports to the security team about their progress and
lack of problems, so their early deployment had some testing benefit to
the community.  (The buildfarm did no testing of the security patches
due to the git mirror blackout.)

I think the unfortunate net effect is that, in this case, distributors
who had to provide source or release notes were actually hampered in
providing early deployment binaries to users.

If you look at each issue individually, allowing binary-only,
no-release-note distributors to deploy early actually makes sense, but
seen in a larger context, it looks more doubtful.  There was some sense
that this was such an unusual release that _safety_ was given such great
importance that some other criteria were overlooked --- perhaps rightly,
perhaps wrongly.

As Josh Berkus already mentioned, the core team is still debriefing all
these issues and how this was handled and trying to improve its
processes.  This is only my personal analysis, not that of the core team
as a whole.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

On 04/08/2013 04:12 PM, Josh Berkus wrote:
>
> Damien,
>
>> The Heroku announcement caused many confusions. The worst confusion is
>> that it sounds like Heroku gets a special treament and is allowed to
>> upgrade 3 days before full disclosure, while the rest of us have to wait
>> the official release date.
>
> So Heroku had permission from the core team to start their update early,
> partly because of required deployment times, and partly because they
> could supply testing and feedback on the patch, as they have with other
> patches they've backported from future versions of PostgreSQL.  We were
> also conscious of the fact that, as far as we knew, Heroku represented
> the single largest organizational vulnerability to this particular
> issue, and that due to their "port-only access" the possibility of
> accidental disclosure was minimal.
>
> We didn't anticipate that the early notification we did combined with
> the Heroku outage notification would make it obvious that an early
> deployment was happening.  We don't generally do early warnings, and
> this whole "cloud" thing is still new to us organizationally.  ALso, we
> went from discovery to release in 3 weeks, so there wasn't a lot of
> discussion time around policy and procedure.
>
> Clearly we can't do it that way again.
>
> -core is currently hashing out thoughts on what might be a reasonable
> early notification process for high-risk users, and if it's feasible for
> us to have one.  As well as other aspects of our security release procedure.
>

I think that one thing that is very important to remember here, is -core
doesn't make decisions lightly. I understand that some may have felt
that communication wasn't perfect (it wasn't) but I know several members
of core and they all take the position IMHO too seriously. So let's
relax, realize there is improvement to be made (which they and we do)
and just work the problem.

The rest is just noise.

Sincerely,

JD





--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On 04/08/2013 04:32 PM, Joshua D. Drake wrote:
>
> On 04/08/2013 04:12 PM, Josh Berkus wrote:
>>
>
>> Clearly we can't do it that way again.
>>
>> -core is currently hashing out thoughts on what might be a reasonable
>> early notification process for high-risk users, and if it's feasible for
>> us to have one.  As well as other aspects of our security release
>> procedure.
>>
>
> I think that one thing that is very important to remember here, is -core
> doesn't make decisions lightly. I understand that some may have felt
> that communication wasn't perfect (it wasn't) but I know several members
> of core and they all take the position IMHO too seriously. So let's
> relax, realize there is improvement to be made (which they and we do)
> and just work the problem.

Agreed. As far as I can see things where handled in the Postgres way,
when in doubt err on the side of caution. I applaud the efforts of those
concerned and trust in their ability to build on the experience.

>
> The rest is just noise.
>
> Sincerely,
>
> JD
>
>
>
>
>


--
Adrian Klaver
adrian.klaver@gmail.com

> Agreed. As far as I can see things where handled in the Postgres way,
> when in doubt err on the side of caution. I applaud the efforts of those
> concerned and trust in their ability to build on the experience.

Mostly I'd rather be arguing as to whether or not we should have given
Heroku early deployment vs. arguing whether or not we could have
prevented them from being hacked.  The same goes for other users, which
is why we're discussing policy now.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 04/08/2013 05:50 PM, Josh Berkus wrote:
>
>
>> Agreed. As far as I can see things where handled in the Postgres way,
>> when in doubt err on the side of caution. I applaud the efforts of those
>> concerned and trust in their ability to build on the experience.
>
> Mostly I'd rather be arguing as to whether or not we should have given
> Heroku early deployment

No.

That isn't to say I didn't understand that theory of doing so. I do.
However, you essentially created a class of user that is above other
users and that is explicitly not Open Source.

Sincerely,

Joshua D. Drake

--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On 04/08/2013 05:50 PM, Josh Berkus wrote:
>
>> Agreed. As far as I can see things where handled in the Postgres way,
>> when in doubt err on the side of caution. I applaud the efforts of those
>> concerned and trust in their ability to build on the experience.
>
> Mostly I'd rather be arguing as to whether or not we should have given
> Heroku early deployment vs. arguing whether or not we could have
> prevented them from being hacked.  The same goes for other users, which
> is why we're discussing policy now.

I am going to admit to being dense, but is it not the same thing?

>


--
Adrian Klaver
adrian.klaver@gmail.com

On Tue, Apr 9, 2013 at 2:20 AM, Joshua D. Drake <jd@commandprompt.com> wrote:
>
> On 04/08/2013 05:50 PM, Josh Berkus wrote:
>>
>>
>>
>>> Agreed. As far as I can see things where handled in the Postgres way,
>>> when in doubt err on the side of caution. I applaud the efforts of those
>>> concerned and trust in their ability to build on the experience.
>>
>>
>> Mostly I'd rather be arguing as to whether or not we should have given
>> Heroku early deployment
>
>
> No.
>
> That isn't to say I didn't understand that theory of doing so. I do.
> However, you essentially created a class of user that is above other users
> and that is explicitly not Open Source.

I'm not arguing either way here... but if you think of DBaaS as "the
new packaging", it starts to seem more reasonable to give folks like
Heroku access at the same time as the traditional packagers.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Le 09/04/2013 10:21, Dave Page a écrit :
> On Tue, Apr 9, 2013 at 2:20 AM, Joshua D. Drake <jd@commandprompt.com> wrote:
>>
>> On 04/08/2013 05:50 PM, Josh Berkus wrote:
>>>
>>>
>>>
>>>> Agreed. As far as I can see things where handled in the Postgres way,
>>>> when in doubt err on the side of caution. I applaud the efforts of those
>>>> concerned and trust in their ability to build on the experience.
>>>
>>>
>>> Mostly I'd rather be arguing as to whether or not we should have given
>>> Heroku early deployment
>>
>>
>> No.
>>
>> That isn't to say I didn't understand that theory of doing so. I do.
>> However, you essentially created a class of user that is above other users
>> and that is explicitly not Open Source.
>
> I'm not arguing either way here... but if you think of DBaaS as "the
> new packaging", it starts to seem more reasonable to give folks like
> Heroku access at the same time as the traditional packagers.
>

Just to make things clear : my previous message is not about wether or
nor a DBaaS provider should have early access to security releases (even
if that's a good question).

My message is about wether or not a DBaaS provider should be allowed to
deploy the security release days before the official release date.

On Mon, Apr 08, 2013 at 06:58:57PM -0400, Jonathan S. Katz wrote:
> In this specific case, DBaaS providers were exposed to a bug that is
> relatively easy to exploit with potentially dire consequences that could
> potentially ruin many, many businesses (I do not want to give a bad estimate,
> so I won't provide a number).  Let's say this horrible scenario happened:

So you're saying we make it dependant on how many business critical
installations a provider runs? In theory that makes a lot of sense, but in
reality I fail to see how to do this.

> sure, people could say that a DBaaS provider did not adequately secure their
> system, but fingers could also be pointed at the community for a) having a
> security hole in the first place (as ludicrous as that sounds to us as we
> know that software is flawed AND Postgres has an *excellent* track record for
> security) and b) not recognizing the damage that could be caused by not
> permitting systems considered to be "critical infrastructure" early access to
> a fix.

How about a big corporate user where PostgreSQL is the backbone? Wouldn't look
good for us either, but not being a DBaaS provider they are not in our focus
here. Makes me wonder why.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at gmail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On Tue, Apr 09, 2013 at 09:21:59AM +0100, Dave Page wrote:
> I'm not arguing either way here... but if you think of DBaaS as "the
> new packaging", it starts to seem more reasonable to give folks like
> Heroku access at the same time as the traditional packagers.

Good point. Isn't this along the same lines as Damien's posting, in that we
should not be on the hook for their deployment time?

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at gmail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On Mon, Apr 08, 2013 at 11:27:53PM +0200, damien clochard wrote:
> Now I understand that Heroku (and other DBaaS providers) may host
> hundreds of thousand PostgreSQL servers and I understand that upgrading
> so many servers in a few hours is something very hard to acheive. But

And the other question that comes up is why are we only talking about this in
the context of a DBaaS provider. Lets just create an example. Assume you have a
distributed system storing all the data about the whole population for a
country with one database per municipality. This could lead to a lot of
databases and obviously the same problems, but no one, at least so far, would
consider them a packager. So they would have the same deployment problem but no
access to the fix before the end of our embargo time.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at gmail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On 04/09/2013 01:38 AM, damien clochard wrote:

>>> No.
>>>
>>> That isn't to say I didn't understand that theory of doing so. I do.
>>> However, you essentially created a class of user that is above other users
>>> and that is explicitly not Open Source.
>>
>> I'm not arguing either way here... but if you think of DBaaS as "the
>> new packaging", it starts to seem more reasonable to give folks like
>> Heroku access at the same time as the traditional packagers.
>>
>
> Just to make things clear : my previous message is not about wether or
> nor a DBaaS provider should have early access to security releases (even
> if that's a good question).
>
> My message is about wether or not a DBaaS provider should be allowed to
> deploy the security release days before the official release date.
>

Right. Which I don't think they should be able to.

Joshua D. Drake

--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On 04/09/2013 02:55 AM, Michael Meskes wrote:
>
> On Tue, Apr 09, 2013 at 09:21:59AM +0100, Dave Page wrote:
>> I'm not arguing either way here... but if you think of DBaaS as "the
>> new packaging", it starts to seem more reasonable to give folks like
>> Heroku access at the same time as the traditional packagers.
>
> Good point. Isn't this along the same lines as Damien's posting, in that we
> should not be on the hook for their deployment time?

Well no because traditional packagers all release at the same time so
that there is no disparity between when Ubuntu gets the fix and Solaris
gets the fix.

Sincerely,

Joshua D. Drake

>
> Michael
>


--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On Tue, Apr 09, 2013 at 08:39:03AM -0700, Joshua D. Drake wrote:
>
> On 04/09/2013 02:55 AM, Michael Meskes wrote:
> >
> >On Tue, Apr 09, 2013 at 09:21:59AM +0100, Dave Page wrote:
> >>I'm not arguing either way here... but if you think of DBaaS as "the
> >>new packaging", it starts to seem more reasonable to give folks like
> >>Heroku access at the same time as the traditional packagers.
> >
> >Good point. Isn't this along the same lines as Damien's posting, in that we
> >should not be on the hook for their deployment time?
>
> Well no because traditional packagers all release at the same time
> so that there is no disparity between when Ubuntu gets the fix and
> Solaris gets the fix.

So what do I misunderstand? As far as I read it, Damien said all should get the
fix at the same time, right? Which is what you say and also what Dave said,
isn't it? I think the question we're dancing around here is, should anyone be
allowed to deploy before the embargo is over? I don't mind DBaaS providers
getting the fix early, but I mind seeing it deployed early.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at gmail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On 04/09/2013 09:01 AM, Michael Meskes wrote:

>> Well no because traditional packagers all release at the same time
>> so that there is no disparity between when Ubuntu gets the fix and
>> Solaris gets the fix.
>
> So what do I misunderstand? As far as I read it, Damien said all should get the
> fix at the same time, right? Which is what you say and also what Dave said,
> isn't it? I think the question we're dancing around here is, should anyone be
> allowed to deploy before the embargo is over? I don't mind DBaaS providers
> getting the fix early, but I mind seeing it deployed early.

Maybe I wasn't clear, sorry. No. I do not believe that ANY entity should
be able to deploy before the embargo is over.

Joshua D. Drake


>
> Michael
>


--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

* Joshua D. Drake (jd@commandprompt.com) wrote:
> On 04/09/2013 09:01 AM, Michael Meskes wrote:
> >>Well no because traditional packagers all release at the same time
> >>so that there is no disparity between when Ubuntu gets the fix and
> >>Solaris gets the fix.
> >
> >So what do I misunderstand? As far as I read it, Damien said all should get the
> >fix at the same time, right? Which is what you say and also what Dave said,
> >isn't it? I think the question we're dancing around here is, should anyone be
> >allowed to deploy before the embargo is over? I don't mind DBaaS providers
> >getting the fix early, but I mind seeing it deployed early.
>
> Maybe I wasn't clear, sorry. No. I do not believe that ANY entity
> should be able to deploy before the embargo is over.

Then perhaps I'm missing something, but what's the point in getting the
update if you can't actually apply it until everyone (including the bad
guys) know about it?  Particularly when applying it is going to take a
whole lot more time than it takes for the bad guys to probe your systems
and figure out which aren't patched yet...

    Thanks,

        Stephen

On 04/09/2013 09:29 AM, Stephen Frost wrote:
> * Joshua D. Drake (jd@commandprompt.com) wrote:
>> On 04/09/2013 09:01 AM, Michael Meskes wrote:
>>>> Well no because traditional packagers all release at the same time
>>>> so that there is no disparity between when Ubuntu gets the fix and
>>>> Solaris gets the fix.
>>>
>>> So what do I misunderstand? As far as I read it, Damien said all should get the
>>> fix at the same time, right? Which is what you say and also what Dave said,
>>> isn't it? I think the question we're dancing around here is, should anyone be
>>> allowed to deploy before the embargo is over? I don't mind DBaaS providers
>>> getting the fix early, but I mind seeing it deployed early.
>>
>> Maybe I wasn't clear, sorry. No. I do not believe that ANY entity
>> should be able to deploy before the embargo is over.
>
> Then perhaps I'm missing something, but what's the point in getting the
> update if you can't actually apply it until everyone (including the bad
> guys) know about it?  Particularly when applying it is going to take a
> whole lot more time than it takes for the bad guys to probe your systems
> and figure out which aren't patched yet...

I don't know that there is a all-in-one solution for this particular
scenario.

Joshua D. Drake




--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

On 2013-04-09 12:29:37 -0400, Stephen Frost wrote:
> * Joshua D. Drake (jd@commandprompt.com) wrote:
> > On 04/09/2013 09:01 AM, Michael Meskes wrote:
> > >>Well no because traditional packagers all release at the same time
> > >>so that there is no disparity between when Ubuntu gets the fix and
> > >>Solaris gets the fix.
> > >
> > >So what do I misunderstand? As far as I read it, Damien said all should get the
> > >fix at the same time, right? Which is what you say and also what Dave said,
> > >isn't it? I think the question we're dancing around here is, should anyone be
> > >allowed to deploy before the embargo is over? I don't mind DBaaS providers
> > >getting the fix early, but I mind seeing it deployed early.
> >
> > Maybe I wasn't clear, sorry. No. I do not believe that ANY entity
> > should be able to deploy before the embargo is over.
>
> Then perhaps I'm missing something, but what's the point in getting the
> update if you can't actually apply it until everyone (including the bad
> guys) know about it?  Particularly when applying it is going to take a
> whole lot more time than it takes for the bad guys to probe your systems
> and figure out which aren't patched yet...

Patching, packaging and verifying that the package works takes time,
especially if you run a modified version of postgres.

Greetings,

Andres Freund

--
 Andres Freund                       http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

* Joshua D. Drake (jd@commandprompt.com) wrote:
> I don't know that there is a all-in-one solution for this particular
> scenario.

Perhaps not, but I feel we can, and should, do our best to try and get
everyone updated before giving attackers the information they need to
exploit people.

    Thanks,

        Stephen

On 04/09/2013 10:06 AM, Stephen Frost wrote:
> * Joshua D. Drake (jd@commandprompt.com) wrote:
>> I don't know that there is a all-in-one solution for this particular
>> scenario.
>
> Perhaps not, but I feel we can, and should, do our best to try and get
> everyone updated before giving attackers the information they need to
> exploit people.

Well I certainly agree with that.

>
>     Thanks,
>
>         Stephen
>


--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

* Andres Freund (andres@2ndquadrant.com) wrote:
> On 2013-04-09 12:29:37 -0400, Stephen Frost wrote:
> > Then perhaps I'm missing something, but what's the point in getting the
> > update if you can't actually apply it until everyone (including the bad
> > guys) know about it?  Particularly when applying it is going to take a
> > whole lot more time than it takes for the bad guys to probe your systems
> > and figure out which aren't patched yet...
>
> Patching, packaging and verifying that the package works takes time,
> especially if you run a modified version of postgres.

I agree with that.  For individuals who are primairly responsible for
providing packages getting access early to do those tasks is great.

That does not address the large-scale deployments where upgrades also
take a very signifigant amount of time.  If we are to provide them with
the information ahead of the release, as they are trusted, I do not
believe it makes any sense to prevent them from upgrading their systems
until the information is out in the open.

Weighing the needs of various communities along with their risk profiles
and trustworthiness is a very difficult thing, but once vetted and
approved for early access, they should be encouraged to do as much as
they can to ensure they are not vulnerable provided that they are able
to do so without disclosing sensetive information.

    Thanks,

        Stephen

On 2013-04-09 13:14:18 -0400, Stephen Frost wrote:
> * Andres Freund (andres@2ndquadrant.com) wrote:
> > On 2013-04-09 12:29:37 -0400, Stephen Frost wrote:
> > > Then perhaps I'm missing something, but what's the point in getting the
> > > update if you can't actually apply it until everyone (including the bad
> > > guys) know about it?  Particularly when applying it is going to take a
> > > whole lot more time than it takes for the bad guys to probe your systems
> > > and figure out which aren't patched yet...
> >
> > Patching, packaging and verifying that the package works takes time,
> > especially if you run a modified version of postgres.
>
> I agree with that.  For individuals who are primairly responsible for
> providing packages getting access early to do those tasks is great.
>
> That does not address the large-scale deployments where upgrades also
> take a very signifigant amount of time.  If we are to provide them with
> the information ahead of the release, as they are trusted, I do not
> believe it makes any sense to prevent them from upgrading their systems
> until the information is out in the open.

Installing the packages somewhere where far more people have a chance to
gain access to reduces the likelihood that somebody figures out where
the vulnerability is noticeably. Figuring out which parts of a binary
have changed is easy enough, even if its stripped.

Also, it changes how privileged the people that get access to the
vulnerability are. If they are allowed to install at the same time as
everyone else its somewhat fair game, otherwise there will be people
making a marketing distinction out of their privileged access.

Greetings,

Andres Freund

--
 Andres Freund                       http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

On Apr 9, 2013, at 1:41 PM, Andres Freund wrote:

> On 2013-04-09 13:14:18 -0400, Stephen Frost wrote:
>> * Andres Freund (andres@2ndquadrant.com) wrote:
>>> On 2013-04-09 12:29:37 -0400, Stephen Frost wrote:
>>>> Then perhaps I'm missing something, but what's the point in getting the
>>>> update if you can't actually apply it until everyone (including the bad
>>>> guys) know about it?  Particularly when applying it is going to take a
>>>> whole lot more time than it takes for the bad guys to probe your systems
>>>> and figure out which aren't patched yet...
>>>
>>> Patching, packaging and verifying that the package works takes time,
>>> especially if you run a modified version of postgres.
>>
>> I agree with that.  For individuals who are primairly responsible for
>> providing packages getting access early to do those tasks is great.
>>
>> That does not address the large-scale deployments where upgrades also
>> take a very signifigant amount of time.  If we are to provide them with
>> the information ahead of the release, as they are trusted, I do not
>> believe it makes any sense to prevent them from upgrading their systems
>> until the information is out in the open.
>
> Installing the packages somewhere where far more people have a chance to
> gain access to reduces the likelihood that somebody figures out where
> the vulnerability is noticeably. Figuring out which parts of a binary
> have changed is easy enough, even if its stripped.
>
> Also, it changes how privileged the people that get access to the
> vulnerability are. If they are allowed to install at the same time as
> everyone else its somewhat fair game, otherwise there will be people
> making a marketing distinction out of their privileged access.

Well, part of the policy of getting early access should be "do not publicize that you have early access" - that would
eliminateany publicity / marketing advantages an entity could take. 

Jonathan

On 2013-04-09 13:46:43 -0400, Jonathan S. Katz wrote:
> On Apr 9, 2013, at 1:41 PM, Andres Freund wrote:
>
> > On 2013-04-09 13:14:18 -0400, Stephen Frost wrote:
> >> * Andres Freund (andres@2ndquadrant.com) wrote:
> >>> On 2013-04-09 12:29:37 -0400, Stephen Frost wrote:
> >>>> Then perhaps I'm missing something, but what's the point in getting the
> >>>> update if you can't actually apply it until everyone (including the bad
> >>>> guys) know about it?  Particularly when applying it is going to take a
> >>>> whole lot more time than it takes for the bad guys to probe your systems
> >>>> and figure out which aren't patched yet...
> >>>
> >>> Patching, packaging and verifying that the package works takes time,
> >>> especially if you run a modified version of postgres.
> >>
> >> I agree with that.  For individuals who are primairly responsible for
> >> providing packages getting access early to do those tasks is great.
> >>
> >> That does not address the large-scale deployments where upgrades also
> >> take a very signifigant amount of time.  If we are to provide them with
> >> the information ahead of the release, as they are trusted, I do not
> >> believe it makes any sense to prevent them from upgrading their systems
> >> until the information is out in the open.
> >
> > Installing the packages somewhere where far more people have a chance to
> > gain access to reduces the likelihood that somebody figures out where
> > the vulnerability is noticeably. Figuring out which parts of a binary
> > have changed is easy enough, even if its stripped.
> >
> > Also, it changes how privileged the people that get access to the
> > vulnerability are. If they are allowed to install at the same time as
> > everyone else its somewhat fair game, otherwise there will be people
> > making a marketing distinction out of their privileged access.
>
> Well, part of the policy of getting early access should be "do not publicize that you have early access" - that would
eliminateany publicity / marketing advantages an entity could take. 

Things like the heroku downtime notice make that pretty clear
though. They hardly could not announce that they have a downtime though,
so I am not blaming them for that, but its still obvious.

Greetings,

Andres Freund

--
 Andres Freund                       http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

All,

* Selena Deckelmann (selena@chesnok.com) wrote:
> On Tue, Apr 2, 2013 at 4:42 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > Having some kind of documentation / policy regarding who can get access,
> > or what they have to do to get access, would certainly help address
> > these concerns.
>
> This is a key point.

Here's what I've been kicking around for a general plan (though -advocacy
still seems like an odd place to discuss this, but whatever):

Tiered release-
First to people who can FIX the problem, eg: -security
Second to people who maintain things downstream:
  This would include both packagers for major distros and large-scale
  DBaaS environments; approved by -core or a similar committee.
Public notification of a general release to be forthcoming.
Third to the general public as binaries/packages
Lastly, full disclosure, sources, etc.

This would only apply in cases where there is no known exploit and the
bug is not generally known, and perhaps only for major bugs.

Ideally, we would be able to minimize impact from this process on the
developers, perhaps through an independent/security repo or similar.

Anyway, that's my 2c.

    Thanks,

        Stephen

* Andres Freund (andres@2ndquadrant.com) wrote:
> Also, it changes how privileged the people that get access to the
> vulnerability are. If they are allowed to install at the same time as
> everyone else its somewhat fair game, otherwise there will be people
> making a marketing distinction out of their privileged access.

I do not consider this a game where everyone should be treated 'fairly'
wrt their exposure to attackers.  I would be open to including something
in the policy which discourages members from advertising their
membership as a marketing distinction, but I'm not convinced that it's
necessary.

    Thanks,

        Stephen

* Selena Deckelmann (selena@chesnok.com) wrote:
> Another important aspect of PostgreSQL is that we are a collective, rather
> than a company. We don't have, for example, a legal entity of record that
> could legitimately accept NDAs on behalf of our developers. (More than one
> vendor brought up "sign an NDA" as a way to get early access, and that's
> not a reasonable option for adding people to pgsql-security or
> pgsql-packagers.)

I wouldn't encourage this- but we do have a legal entity through SPI.
Were we, as a community, open to using 'signed an NDA' as sufficient
trust, using SPI as the entity could work.  To be honest, I don't think
that we, collectively, feel that a signed NDA is sufficient.

    Thanks,

        Stephen

On 2013-04-09 13:59:29 -0400, Stephen Frost wrote:
> * Andres Freund (andres@2ndquadrant.com) wrote:
> > Also, it changes how privileged the people that get access to the
> > vulnerability are. If they are allowed to install at the same time as
> > everyone else its somewhat fair game, otherwise there will be people
> > making a marketing distinction out of their privileged access.
>
> I do not consider this a game where everyone should be treated 'fairly'
> wrt their exposure to attackers.  I would be open to including something
> in the policy which discourages members from advertising their
> membership as a marketing distinction, but I'm not convinced that it's
> necessary.

Note that I am not saying that it has to be fair. I haven't yet made up
my mind about it, I am just saying its a fair point to make. And I think
the increased exposure and thus increased likelihood of leakage due to
more widespread usage holds some weight, completely independent of the
argument of fairness.

Greetings,

Andres Freund

--
 Andres Freund                       http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

* Selena Deckelmann (selena@chesnok.com) wrote:
> None are empowered to sign contracts or legally represent the developers
> who make up PGDG.

It is not the developers comprised of PGDG who are required to sign into
an NDA.  It is company A, B, or C who would need to sign an NDA with "some
legal entity", giving that legal entity the power/right to sue company A,
B, or C, were they to release the information provided to them
inappropriately under a breach of contract.

Notionally, perhaps a PGDG developer would provide the data to the
'legal entity' who would then provide the information to the company, to
legitimatize the contract, but that would not require any NDA or
contract to be signed by the PGDG developer.

    Thanks,

        Stephen

* Andres Freund (andres@2ndquadrant.com) wrote:
> And I think the increased exposure and thus increased likelihood of
> leakage due to more widespread usage holds some weight

This is most appropriately addressed on a case-by-case basis regarding
the specific situation.  Such classification should be done by -core (or
some similar committee) and then we should have a policy which can be
followed based on that classification.  My hope is that the very general
policy which I outlined could simply be tailored based on the
classification.

    Thanks,

        Stephen

On Tue, Apr 9, 2013 at 11:01 AM, Stephen Frost <sfrost@snowman.net> wrote:

* Selena Deckelmann (selena@chesnok.com) wrote:

> Another important aspect of PostgreSQL is that we are a collective, rather
> than a company. We don't have, for example, a legal entity of record that
> could legitimately accept NDAs on behalf of our developers. (More than one
> vendor brought up "sign an NDA" as a way to get early access, and that's
> not a reasonable option for adding people to pgsql-security or
> pgsql-packagers.)

I wouldn't encourage this- but we do have a legal entity through SPI.
Were we, as a community, open to using 'signed an NDA' as sufficient
trust, using SPI as the entity could work.  To be honest, I don't think
that we, collectively, feel that a signed NDA is sufficient.

As far as I know, our association with SPI hasn't been empowered to sign contracts on behalf of PGDG. They don't even hold any trademarks for us. PGDG's association with SPI is to receive donations and disperse grants.  Happy to be corrected if I am mistaken on those points.

We also have several other non-profits whose missions are varied.

None are empowered to sign contracts or legally represent the developers who make up PGDG.

On 04/09/2013 11:22 AM, Dave Page wrote:

>> We also have several other non-profits whose missions are varied.
>>
>> None are empowered to sign contracts or legally represent the
>> developers who make up PGDG.
>
> PGDG is not a 'real' entity at all, so it has no way of empowering
> anyone to do anything really. The closest we have is The PostgreSQL
> Community Association of Canada which has been blessed by -core to hold
> domains and trademarks etc. (and doesn't do anything else).

Well there is SPI too.

Joshua D. Drake



--
Command Prompt, Inc. - http://www.commandprompt.com/
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, Postgres-XC
@cmdpromptinc - 509-416-6579

Stephen Frost <sfrost@snowman.net> writes:
> That does not address the large-scale deployments where upgrades also
> take a very signifigant amount of time.  If we are to provide them with
> the information ahead of the release, as they are trusted, I do not
> believe it makes any sense to prevent them from upgrading their systems
> until the information is out in the open.

+1

> Weighing the needs of various communities along with their risk profiles
> and trustworthiness is a very difficult thing, but once vetted and
> approved for early access, they should be encouraged to do as much as
> they can to ensure they are not vulnerable provided that they are able
> to do so without disclosing sensetive information.

+1

And no ssh access to the servers seems like it applied.

The trust problem has just been presented to me in another phrasing that
we might want to be adressing: the level of trust we have into those
people who receive the information early obviously includes they not
perusing the information to exploit users (e.g. from competitive
places).

As obvious as it sounds, we have to write it down in the docs currently
being edited, I think.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

On Tue, Apr 09, 2013 at 01:14:18PM -0400, Stephen Frost wrote:
> That does not address the large-scale deployments where upgrades also
> take a very signifigant amount of time.  If we are to provide them with
> the information ahead of the release, as they are trusted, I do not
> believe it makes any sense to prevent them from upgrading their systems
> until the information is out in the open.

But this does not only apply to the Heroku's of this world. What about the not
so hypothecial example I brought earlier? There are actually a lot of companies
out there that deploy Postgres on a large scale but are not DBaaS providers.
There are also alot of companies that somehow bundle Postgres with their
product and deliver it to *a lot* of customers. Their upgrade problem is even
worse. Do we add them all?

Besides some of these might get their packages from
service providers. Ok, in theory we could add those. But how about those who
use packages from  one of the distros? With the same argument we would have to
go for a two step embargo.

Michael

--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at gmail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

Michael,

* Michael Meskes (meskes@postgresql.org) wrote:
> But this does not only apply to the Heroku's of this world. What about the not
> so hypothecial example I brought earlier? There are actually a lot of companies
> out there that deploy Postgres on a large scale but are not DBaaS providers.
> There are also alot of companies that somehow bundle Postgres with their
> product and deliver it to *a lot* of customers. Their upgrade problem is even
> worse. Do we add them all?

Who gets added and who doesn't would be the committee's responsibility.
Risk and exposure would weigh into that decision.  DBaaS providers had a
much higher from this most recent bug than even very large scale
internal deployments.  When asking "do we add them all?", the answer
will have to be 'no' or there would end up being little point.

> Besides some of these might get their packages from
> service providers. Ok, in theory we could add those. But how about those who
> use packages from  one of the distros? With the same argument we would have to
> go for a two step embargo.

I don't entirely follow this.  Upthread I had suggested a multi-phase
approach which sounds like what you mean by 'two step embargo'.  I
continue to feel that makes sense, to give everyone the best chance at
upgrading prior to exploits being generally available.

    Thanks,

        Stephen

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160


Stephen Frost replied:
> Who gets added and who doesn't would be the committee's responsibility.
> Risk and exposure would weigh into that decision.  DBaaS providers had a
> much higher from this most recent bug than even very large scale
> internal deployments.  When asking "do we add them all?", the answer
> will have to be 'no' or there would end up being little point.

Still sounds like a huge mess. Who gets put on the committee? Wouldn't the
committee need to have a huge database of potential people to notify, including
details of their systems (e.g. do they use tsearch, if this is a tsearch bug).
Would they be deciding on people on a case by case basis, or just deciding
on what class of people get notified. If the latter, why not just have
core continue to do it? If the former, how can that be agile enough for a
quick turnaround? Would companies have to be requested to be added to
this database, in the hopes they get notified of a serious bug before it
is released?

Perhaps we can just agree that the way this was handled was a mistake, and
strive for more transparency and egalitarianism next time? Sure, Heroku has
a huge list of servers listening on 5432, but do did that place in Poland,
and apparently they did not get a early warning.

- --
Greg Sabino Mullane greg@turnstep.com
End Point Corporation http://www.endpoint.com/
PGP Key: 0x14964AC8 201304112203
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAlFna3IACgkQvJuQZxSWSsi3FQCdHjlrxnS+izZTay7dd2eVvk/l
mQEAoIda6OkcpbZ9Y59nubSg0faVzUO3
=PSSA
-----END PGP SIGNATURE-----

* Greg Sabino Mullane (greg@turnstep.com) wrote:
> Still sounds like a huge mess. Who gets put on the committee?

The initial suggestion was to put -core on it.

> Wouldn't the
> committee need to have a huge database of potential people to notify, including
> details of their systems (e.g. do they use tsearch, if this is a tsearch bug).

No; if it's that specific, then it's not a high-exposure security bug.

There will certainly be some amount of trial-and-error associated with
this, but that's the nature of security issues- they aren't all going
to be the same.

> Would they be deciding on people on a case by case basis, or just deciding
> on what class of people get notified. If the latter, why not just have
> core continue to do it?

My comments around "-core or a committee" was more geared towards if
-core felt a seperate committee would be better.  I'm fine with it being
left with -core, but I do feel that we should publicize and perhaps
formalize a bit more that policy.

> If the former, how can that be agile enough for a
> quick turnaround? Would companies have to be requested to be added to
> this database, in the hopes they get notified of a serious bug before it
> is released?

Yes, organizations which felt they met the requirements outlined in the
policy would need to request to be added and then a decision by -core
(or whatever group it is) would need to be made regarding that request.

> Perhaps we can just agree that the way this was handled was a mistake, and
> strive for more transparency and egalitarianism next time? Sure, Heroku has
> a huge list of servers listening on 5432, but do did that place in Poland,
> and apparently they did not get a early warning.

I agree that, in hindsight, we could have done better.  That's what this
discussion is about- figuring out how to do better in the future.  I
don't think that means we should give up on having a security policy
which allows early access to trusted organizations.

    Thanks,

        Stephen

> Perhaps we can just agree that the way this was handled was a mistake, and
> strive for more transparency and egalitarianism next time? Sure, Heroku has
> a huge list of servers listening on 5432, but do did that place in Poland,
> and apparently they did not get a early warning.

We didn't find out about Home.PL until the day of release.  And we
wouldn't put them on an early notification list in any case, since
despite our efforts, we don't have a direct, trusted contact there.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

>> Perhaps not, but I feel we can, and should, do our best to try and get
>> everyone updated before giving attackers the information they need to
>> exploit people.
>
> Well I certainly agree with that.

... which was the goal in doing early notification of the cloud
providers.  They were indisputably the biggest potential targets for the
recent vulnerability.  And they *didn't* get hacked, so the strategy was
materially successful.  Whether or not a different approach would have
been equally/more successful is, at this point, "monday morning
quarterbacking" as we say in the 'States.

I'm a pragmatist.  I'm looking for the policy which protects the most
users from script kiddies.  If that policy is fair and democratic that's
also good, but less important than preventing people from being hacked.
 This is where I, personally, am coming from.

The problem with early notification from this perspective is that the
more organizations receiving early notification, the greater the chance
of a leak, at which point you've done the opposite of protecting users.
 On the other hand, the problem with no notification is that you create
a race between black hats and admins as to who can deploy the fix vs.
the exploit faster, which isn't good either.  I don't know that any
organization has a clear answer to this year, including large commercial
software vendors.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Hi Stephen, Hi all,


First, Stephen, please apology the short quote of your mail below.

Le vendredi 12 avril 2013 à 09:18 -0400, Stephen Frost a écrit :
> I
> don't think that means we should give up on having a security policy
> which allows early access to trusted organizations.

So I just quoted this sentence. Actually, I wanted to quote only 2
words: "trusted organizations".

If we want to still deliver early accesses to some and not to the
others, then, yes, we would want "trusted organizations".

The fundamental question then, is how organizations qualify to become
"trusted organizations" ?

In the commercial~business'world that's quite obvious. Some pay for it,
others signs Non-Disclosure Agreement, often both.

But who will pay for what, given our organization doesn't have a single
legal and central entity? If someone tells me about PostgreSQL Canada:
do this organization has lawyers or is willing to pay for some ? Will
this be appliable globally ? US or Can laws applies everywhere, really?

Yeah, this is becoming awfully difficult IMHO.

Lots of people on this list, and Im part of it, want to have users
treated equally and carrefully.

Saying one organization matters more than another just because it has
more users or postmasters is wrong to me. We all know lots of places
where a single postmaster holds such important data, sometimes managing
somewhat people's life!

Will we then compare among databases, who has the most important? How we
will do that?

How will you 'trust' a company which has 5,50,500,5000 people in it ?

All these questions leads to undecidability, IMHO.

To me the only way to do is give the access to all at the same time,
despite all the problems that may occurs. Yes, it's the "hard way", but
it's the only one leading to the equalty we want.

It's not a community matter to care about commercial issues, to validate
or invalidate one's business plan or whatever.

People who really care about the security of their users will have to do
the necessary efforts and machinery to think about a deployment plan
when a security patch is commited.

Don't read me too fast: I like Heroku a lot. I really appreciate all
their efforts, sponsoring and incentive, putting the spotlights on
PostgreSQL. I also like having more beer tickets like you all on the
events :-P

But do we, as a community, have to care about how they do business with
PostgreSQL ? I don't think so.


My 2 cents.

--
Jean-Paul Argudo
www.PostgreSQL.fr
www.Dalibo.com

Jean-Paul Argudo <jean-paul@postgres.fr> writes:
> The fundamental question then, is how organizations qualify to become
> "trusted organizations" ?

>From my understanding of the current situation, it's quite easy and
clear, arrange to be subscriber on pgsql-packagers.

Maybe what we need to do is document that to get early access to
security updates you need to be registered as a packager, and that we
only accept trusted person in there.

Then any packager is trusted to release the upgrade either in the open
following the public rules, or otherwise as he sees fit with *explicit
approval* from core.

The procedure certainly would need to be specific that should you fail
to follow those 2 easy to document cases, you can get removed from the
packagers list.

> Lots of people on this list, and Im part of it, want to have users
> treated equally and carrefully.

I can't resist, please don't mix equality and equity. See attached, as
pictures work so much better than written words.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

On Mon, Apr 15, 2013 at 10:23:09AM +0200, Dimitri Fontaine wrote:
> Jean-Paul Argudo <jean-paul@postgres.fr> writes:
> > The fundamental question then, is how organizations qualify to become
> > "trusted organizations" ?
>
> >From my understanding of the current situation, it's quite easy and
> clear, arrange to be subscriber on pgsql-packagers.

People will not be happy if we add people to packagers and someone leaks
information to hackers before the official release.

> Maybe what we need to do is document that to get early access to
> security updates you need to be registered as a packager, and that we
> only accept trusted person in there.
>
> Then any packager is trusted to release the upgrade either in the open
> following the public rules, or otherwise as he sees fit with *explicit
> approval* from core.
>
> The procedure certainly would need to be specific that should you fail
> to follow those 2 easy to document cases, you can get removed from the
> packagers list.

Again, the damage is done if someone leaks information, and being
removed from packagers doesn't fix the security problem for everyone
else. We just can't have an iterative process here were we guess who is
trust-worthy and vulnerable, and then remove people when we are wrong.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Adrian Klaver-3 wrote
> On 04/08/2013 05:50 PM, Josh Berkus wrote:
>>
>>> Agreed. As far as I can see things where handled in the Postgres way,
>>> when in doubt err on the side of caution. I applaud the efforts of those
>>> concerned and trust in their ability to build on the experience.
>>
>> Mostly I'd rather be arguing as to whether or not we should have given
>> Heroku early deployment vs. arguing whether or not we could have
>> prevented them from being hacked.  The same goes for other users, which
>> is why we're discussing policy now.
>
> I am going to admit to being dense, but is it not the same thing?

I agree it's better to forego open source doctrine in the interest of
preventing a larger evil.  If we had not given Heroku early acces and they
got hacked then the discussion would revolve around how said hack could have
been prevented instead.  The is decidedly a worse discussion then making
them a special class of user.  If not every person could be given "early
access" then whether someone is given access is irrelevant to another's
circumstance.  Did anyone else even ask the question of special early access
terms?  These kinds of decisions are why a -core group exists instead of
there simply being a communal repository to which anyone can contribute and
use.  Not everything can be planned for in advance and so those unplanned
situations are delegated to a previously designated group of decision
makers.  If people feel the current default process is insufficient they
should have spoken up before now and not when a special case was decided as
being necessary.  This will hopefully impact the future but looking back the
process worked well and as intended.

David J.




--
View this message in context:
http://postgresql.1045698.n5.nabble.com/Heroku-early-upgrade-is-raising-serious-questions-tp5750503p5751367.html
Sent from the PostgreSQL - advocacy mailing list archive at Nabble.com.

Jean-Paul Argudo-6 wrote
> Yeah, this is becoming awfully difficult IMHO.
>
> Lots of people on this list, and Im part of it, want to have users
> treated equally and carrefully.

If it is felt that a legal hammer hanging over their head is necessary to
get someone abide by the terms of the early release then that company/person
should simply not be given access.

There is a happy medium between "do nothing special" and "have an ironclad
policy in place" that is worth exploring.  Those who do not make the
"special" listing are only minimally worse off in that some people have the
code and could exploit that fact.  If the risk of such pre-exploitation is
considerably less than the risk of normal exploitation once the code is
released then the risk-reward balance for the community as a whole suggests
that early release is preferable.

The question I guess is whether you believe the people being dealt with are
inherently good or bad.  People with long track records of contributing to
the project and with high-profile stacks in the project succeeding should
have enough self-preservation interest in seeing that the code is kept
secure just to maintain their reputation, credibility, and business.

It would be worth inspecting the release policy and making sure that the
fewest number of people have access to the source code during the embargo
period.  In effect Heroku should have a single person apply the patch and
build their internal distributions and then invoke their own internal
embargo so that no-one in the company would be allowed to see that
patch/source; they are only allowed to deploy the binary distributions.

David J.




--
View this message in context:
http://postgresql.1045698.n5.nabble.com/Heroku-early-upgrade-is-raising-serious-questions-tp5750503p5752192.html
Sent from the PostgreSQL - advocacy mailing list archive at Nabble.com.

Bruce Momjian <bruce@momjian.us> writes:
> People will not be happy if we add people to packagers and someone leaks
> information to hackers before the official release.

Indeed. That's the way it works today, though.

> Again, the damage is done if someone leaks information, and being
> removed from packagers doesn't fix the security problem for everyone
> else. We just can't have an iterative process here were we guess who is
> trust-worthy and vulnerable, and then remove people when we are wrong.

Agreed. It's a problem of trust, not of procedure, and that's what I
wanted to stress in my previous email by saying that we already have the
procedure. Thanks for underlining it.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Hi All,


First, thanks for your comments. This discussion is very interesting.

Le mardi 16 avril 2013 à 09:21 +0200, Dimitri Fontaine a écrit :
> Bruce Momjian <bruce@momjian.us> writes:
> > People will not be happy if we add people to packagers and someone leaks
> > information to hackers before the official release.
>
> Indeed. That's the way it works today, though.

Yes, true. I see no solution to this problem. Thats why I suggested our
community doesn't deal with it, since every solution we may find will be
surely incomplete if not wrong.

I really doubt we find some kind of solution like "one fits all".

One can play with words (or pictures :-P), but is it really to us, as a
community, to fix one's particular problems?

>> Again, the damage is done if someone leaks information, and being
> > removed from packagers doesn't fix the security problem for everyone
> > else. We just can't have an iterative process here were we guess who is
> > trust-worthy and vulnerable, and then remove people when we are wrong.
>
> Agreed. It's a problem of trust, not of procedure, and that's what I
> wanted to stress in my previous email by saying that we already have the
> procedure. Thanks for underlining it.

So you both agreed on the 1st mail of this thread, at least on the
problem I tried to explain (apologies, I'm quoting myself):

  The fundamental question then, is how organizations qualify to become
  "trusted organizations" ?

On this point, AFAIK, there's still no answer.

> Regards,
> --
> Dimitri Fontaine
> http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support


--
Jean-Paul Argudo
www.PostgreSQL.fr

Hi Selena, Hi all !


Le lundi 15 avril 2013 à 08:39 -0700, Selena Deckelmann a écrit :
> Hi!
>
>
> On Mon, Apr 15, 2013 at 12:42 AM, Jean-Paul Argudo
> <jean-paul@postgres.fr> wrote:
>
>         To me the only way to do is give the access to all at the same
>         time,
>         despite all the problems that may occurs. Yes, it's the "hard
>         way", but
>         it's the only one leading to the equalty we want.
>
>
> PostgreSQL is written and maintained by a 6-member core team, a group
> of about 20 committers, and somewhere between 300-400 developers who
> send in code each year.  Plus many other volunteers who run
> conferences, meetups and participate in mailing lists like this one.

Thanks for this summary.

> From a security standpoint, the decisions made should weigh:

Let me comment this:

> * Risk to the general public
> * Risk to the *known* users of PostgreSQL

Why do you make a difference between the general public (unknown users,
that's what you mean?) and "known" users. Known by who?

Why is this distinction important in your eyes?

What kind of consequences would happen if we keep this distinction?

> * Risk to our core committers, developers and volunteers

What kind of risk? Is this statement a general purpose? Does it apply to
the code generally or just to a security patch?

> * Risk to the survival of the open source project

I don't fear anything here. I trust us to not do things that may result
in this kind of risk.

> and:
> * Do we have a good patch for the problem?
> * Are there possible workarounds without patching?

On those two statements I doubt we lack anything. This community has
probably the best coders I know of. I trust them completely to find
workarounds, patches on any bugs found, etc.

> What is "fair" in that context is not the same thing as "treating
> everyone equally".

So as Dimitri stated, you're more happy with "equity" than "equality".
So I am in the real life too, but I think than finding a solution that
fits all to our problem here is an illusion.

If PaaS users are upgraded before others, whatever the reasons are, this
will lead in another mail thread like this one.

I also think it will be a nightmare to write down something simple and
understandable by all.

> Personally, I do not agree that "equality is what we want" in the
> context of managing security vulnerability disclosure.

If you're talking strictly about how to manage security vulnerability
disclosure, then I agree.

On this point, -core did well to me.

> We are open source, so eventually everyone will have access to patches
> to security vulnerabilities.

Sure.

> However, it's important to use well-understood risk mitigation
> techniques in deciding how to share information about
> vulnerabilities.

Agreed.

> Despite how the disclosure and communication made contributors to this
> thread *feel*, the consensus from security experts that I talked to
> was: PGDG handled this security issue well. We also drew enough
> attention that it *appears* that many of our users upgraded or took
> mitigation action - with minimal compromise exposure after we fully
> > disclosed the bug.

Yeah, I agree too there.

> And now, -core is working to change our security policy to better
> address the concerns of PaaS and security-sensitive users.

My point is that all users should be considered the same, once again.

Since one can't decide between which one's data is more important than
another one's, we won't be able to define what's a "security-sensitive
user".

> To be clear:
> I want users and their data to be as safe as we can keep them.  And I
> want security disclosures to be transparent, well-communicated and
> fairly carried out, using a policy that -core produces.

I could suggest we can help in producing that policy, that -core would
approve. But OK. Let's wait.

> -selena

Thanks for your time.

> --
> http://chesnok.com


--
Jean-Paul Argudo
www.PostgreSQL.fr
www.Dalibo.com

>
> I agree it's better to forego open source doctrine in the interest of
> preventing a larger evil.  If we had not given Heroku early acces and they
> got hacked then the discussion would revolve around how said hack could have
> been prevented instead.  The is decidedly a worse discussion then making
> them a special class of user.
>

I see this argument coming in various forms but still I don't understand
why Heroku is so different from other large-scale users.

Sure Heroku was quite exposed to this vulnerability because :

a- they allow port access to their database servers from untrusted
networks (see [1])

b- they don't have tools to deploy a security fix quickly. (see [2])

But these are technical choices.

I'm not judging them and I'm not saying these two problems are easy to
solve for a large-scale company. But Heroku could have done things
different and they choose not to filter the access to their servers and
they didn't build the machinery for fast security fix deployment.
Moreover they're talking about it in public so I guess these are choices
they've made and they know the consequences behind them...

--

Let's take an example and imagine an hosting company called pgCloud,
with thousand of PostgreSQL servers in their public could. pgCloud is in
competition with Heroku.

pgCloud implemented a sophisticated IP filtering system to protect their
network and they built serious security scripts to update all their
servers in less than an hour. pgCloud did this because they what to
offer the best possible level of security to their customers. Even if a
nasty zero-day exploit is released someday (God forbid!)

Building these tools and procedures did cost a lot of time and money.
But that's fine because when a PostgreSQL vulnerability is discovered,
they can keep calm and wait for the security release.

But now pgCloud has a problem : Heroku was allowed to deploy the
security fix before it went public, while they had to wait for the
security release like everyone else....

So their question is : what shoud we do ? Should we spend time and money
to build/maintain a secure postgres cloud or should we "gain trust" from
the PostgreSQL community in order to be allowed to deploy earlier ?

And what would customer choose between the two ? pgCloud has a secured
network but Heroku has early access to the fix...

--

This example is simplistic of course but I think it shows clearly why we
should stay as "neutral" as possible (I didn't say "fair") and avoid any
special treatment that would disrupt a market. Otherwise we're just
opening a big shiny pandora box.


And yes I am aware that a company can have both : a secure network AND
trust from the community. But if I were sarcastic, I'd say that right
now the best security strategy for a large scale cloud provider is to
set unrestricted access to all their servers, don't bother with update
scripts and ask the community for early access because they're so exposed.



[1] https://news.ycombinator.com/item?id=5493353
[2]
https://blog.heroku.com/archives/2013/4/4/heroku_postgres_databases_patched

Jean-Paul,

* Jean-Paul Argudo (jean-paul@postgres.fr) wrote:
>   The fundamental question then, is how organizations qualify to become
>   "trusted organizations" ?
>
> On this point, AFAIK, there's still no answer.

It's been answered multiple times: -core (or some other committee which
they create, should they feel a need to) is responsible for reviewing
and approving such requests.

    Thanks,

        Stephen

> It's been answered multiple times: -core (or some other committee which
> they create, should they feel a need to) is responsible for reviewing
> and approving such requests.

Actually, at this point the question is *whether or not* to have a early
notification list at all.

Right now, the only people who get early information on not-yet-released
security updates are people who are directly involved in either (a)
patching the updates, or (b) packaging the updates, by policy.  The
definition of "packager" was extended to DBAAS vendors for the last
security release, but not necessarily on a permanent basis.

The security team and the packagers have to receive early information in
order for us to get a security update out the door.  Nobody else does.

There are a lot of pros and cons to having an early notification list at
all.  The pros are obvious to the prospective members of such a list,
but the cons are:

a) as the list grows, the probability of a leak approaches 100%

b) resentment by whomever doesn't make the cut to be on the list

c) effort to maintain the list.

That's the first question to answer.  Discussing who's on such a list
comes after deciding if we should have one at all.  Other open source
projects are split on the issue.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Wed, Apr 17, 2013 at 09:59:14AM -0700, Josh Berkus wrote:
>
> > It's been answered multiple times: -core (or some other committee which
> > they create, should they feel a need to) is responsible for reviewing
> > and approving such requests.
>
> Actually, at this point the question is *whether or not* to have a early
> notification list at all.
>
> Right now, the only people who get early information on not-yet-released
> security updates are people who are directly involved in either (a)
> patching the updates, or (b) packaging the updates, by policy.  The
> definition of "packager" was extended to DBAAS vendors for the last
> security release, but not necessarily on a permanent basis.
>
> The security team and the packagers have to receive early information in
> order for us to get a security update out the door.  Nobody else does.
>
> There are a lot of pros and cons to having an early notification list at
> all.  The pros are obvious to the prospective members of such a list,
> but the cons are:
>
> a) as the list grows, the probability of a leak approaches 100%
>
> b) resentment by whomever doesn't make the cut to be on the list
>
> c) effort to maintain the list.
>
> That's the first question to answer.  Discussing who's on such a list
> comes after deciding if we should have one at all.  Other open source
> projects are split on the issue.

These are all good points.  The vulnerability that got Heroku early
access was a network port vulnerability.  A different type of
vulnerability might _not_ have gotten them early access, and might have
gotten someone else early access.  This port vulnerability was of a
severity that historically we only see every five years, so it is hard
to come up with a policy that might not be exercised for another five
years.

Also, let me add that the value of Heroku testing was what swayed me to
support their early access.  We don't need dozens of users testing our
binaries pre-release, but helps to have a dedicated one who is capable
and reports their findings.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

* Josh Berkus (josh@agliodbs.com) wrote:
> Actually, at this point the question is *whether or not* to have a early
> notification list at all.

For my 2c, I don't view that as any real question at all.  There needs
to be multiple levels of early notifications, as I outlined in my prior
email.

Apologies to those who feel that everyone should have equal access, but
that's a horrible security policy and would get us shunned by the
security community for good reason.

    Thanks,

        Stephen

* Bruce Momjian (bruce@momjian.us) wrote:
> These are all good points.  The vulnerability that got Heroku early
> access was a network port vulnerability.  A different type of
> vulnerability might _not_ have gotten them early access, and might have
> gotten someone else early access.  This port vulnerability was of a
> severity that historically we only see every five years, so it is hard
> to come up with a policy that might not be exercised for another five
> years.

I'm not a fan of building some massive table of who has what exposures
that we need to go and consult every time we have a security fix.
There's either "ok, certain people should know about this ahead of time"
and "this is small-potatoes and doesn't really need early notice", which
mainly boils down into unauthenticated vs. authenticated
vulnerabilities, imv.

I do agree, however, that each security issue needs to be considered
independently on a case-by-case basis.

    Thanks,

        Stephen