Thread: Relative security of Community repos and packages

On 7/28/21 11:26 AM, pbj@cmicdo.com wrote:
> I hope this is the right group for this question:
> 
> Currently involved in a discussion about security of Postgres packages 
> from various sources.  I'm strongly advocating that we get our packages 
> directly from PGDG.
> 
> Would Postgres packages from Red Hat repos (and I guess we could include 
> EDB, 2nd Quadrant, Crunchy...) be considered more secure from being 
> hacked than those from the PGDG repos?

I would think the weak point would be:

https://www.postgresql.org/ftp/source/

as I am pretty sure that is where packagers pull the starting code from.


> 
> Thanks,
> PJ


-- 
Adrian Klaver
adrian.klaver@aklaver.com

> On Jul 28, 2021, at 11:26, pbj@cmicdo.com wrote:
> Currently involved in a discussion about security of Postgres packages from various sources.  I'm strongly advocating
thatwe get our packages directly from PGDG. 
>
> Would Postgres packages from Red Hat repos (and I guess we could include EDB, 2nd Quadrant, Crunchy...) be considered
moresecure from being hacked than those from the PGDG repos? 

While I have nothing bad to say about the other repo sources, every other repo (AFAIK) pulls from the community repos,
sothere's no reason that they would be *more* security than the community sources.  The Infra team takes build chain
andhosting security very seriously, and I would say that you are as safe with the community repos as you would be with
anyother source. 

> On Jul 28, 2021, at 14:02, Dave Page <dpage@pgadmin.org> wrote:
>
> No that is not the case, at least for community and EDB packages. It might be the case for upstream distributors
though(eg. OS vendors). 

They all pull from the community Git repo, though, correct?

Christophe Pettus <xof@thebuild.com> writes:
>> On Jul 28, 2021, at 14:02, Dave Page <dpage@pgadmin.org> wrote:
>> No that is not the case, at least for community and EDB packages. It might be the case for upstream distributors
though(eg. OS vendors). 

> They all pull from the community Git repo, though, correct?

I do not think Red Hat does that; they prefer identifiable released
tarballs.  I've not worked there in nigh ten years, but I still see
this in their PG specfile:

Source0: https://ftp.postgresql.org/pub/source/v%{version}/postgresql-%{version}.tar.bz2

and I clearly recall that there were cross-checks in their build process
that tarball components of an SRPM matched what could be fetched from
the stated URL.  Maybe now they have a process that works with direct git
pulls, but they're not using that method with us.

Can't speak to non-RH-based distros.

            regards, tom lane

Greetings,

* Christophe Pettus (xof@thebuild.com) wrote:
> > On Jul 28, 2021, at 11:26, pbj@cmicdo.com wrote:
> > Currently involved in a discussion about security of Postgres packages from various sources.  I'm strongly
advocatingthat we get our packages directly from PGDG. 
> >
> > Would Postgres packages from Red Hat repos (and I guess we could include EDB, 2nd Quadrant, Crunchy...) be
consideredmore secure from being hacked than those from the PGDG repos? 
>
> While I have nothing bad to say about the other repo sources, every other repo (AFAIK) pulls from the community
repos,so there's no reason that they would be *more* security than the community sources.  The Infra team takes build
chainand hosting security very seriously, and I would say that you are as safe with the community repos as you would be
withany other source. 

This strikes me as a rather confusing way of saying what is going on.

I'll try to clear it up a bit:

As far as I know, everyone pulls initially from the official source
repo, as Christophe says above, which is git.postgresql.org, which is
maintained by pginfra (a volunteer but trusted group of long time PG
contributors).  I'll note that there has been discussion about improving
the security of the git repo through the use of signed commits and such,
but that's clearly not done today as anyone can see.

There are organizations who further review every commit which is made to
that repo too and pull changes into their own git repos to do builds
from.

While the PGDG *binary/package* repos, which are hosted on
ftp.postgresql.org and friends, are maintained by the pginfra team, the
systems where the builds themselves are done are not maintained by
the pginfra team but by other PGDG volunteers.  If you're curious about
the security of those build systems, I'd suggest reaching out to the
appropriate mailing lists for the packages you're interested in and
asking there (or perhaps those volunteers will comment here).  Those
volunteers are also long time PostgreSQL contributors.

I do know that there are certainly organizations who perform their own
independent builds of PostgreSQL from the vetted and reviewed source
from their own trusted git mirror of the official repo on secured
hardware and then provided those trusted builds to their clients (in
fact, I suspect most of the organizations mentioned above do this..).

If you're curious about the security of packages provided by Red Hat, or
any other organization outside of PGDG, it would likely make sense to
ask them about their policies and approach.

Thanks,

Stephen (one of the pginfra team members, as is Christophe)

Stephen Frost <sfrost@snowman.net> writes:
> Indeed, that comment didn’t seem to help clear things up. I’m guessing Dave
> is referring to the fact that we have a separate “gitmaster” server, which
> is also maintained by pginfra and is where committers actually push changes
> to, and then that is mirrored to git.postgresql.org.  I didn’t check which
> repo the tarball building script pulls from (which is run on pginfra, in
> case anyone is wondering about that) and perhaps it pulls from gitmaster
> and not git.p.o.

It does pull from gitmaster.  There are multiple reasons for this design,
but one is that a compromise of our public git server wouldn't imperil
the contents of the official tarballs.

            regards, tom lane

Greetings,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > Indeed, that comment didn’t seem to help clear things up. I’m guessing Dave
> > is referring to the fact that we have a separate “gitmaster” server, which
> > is also maintained by pginfra and is where committers actually push changes
> > to, and then that is mirrored to git.postgresql.org.  I didn’t check which
> > repo the tarball building script pulls from (which is run on pginfra, in
> > case anyone is wondering about that) and perhaps it pulls from gitmaster
> > and not git.p.o.
>
> It does pull from gitmaster.  There are multiple reasons for this design,
> but one is that a compromise of our public git server wouldn't imperil
> the contents of the official tarballs.

That doesn't do much for the large number of folks who use
git.postgresql.org or the github mirror though, unfortunately.  Signed
commits, on the other hand, would help.

Thanks,

Stephen

Greetings,

* pbj@cmicdo.com (pbj@cmicdo.com) wrote:
>  On Thursday, July 29, 2021, 3:32:33 PM EDT, Dave Cramer <davecramer@gmail.com> wrote:   > On Thu, 29 Jul 2021 at
15:25,pbj@cmicdo.com <pbj@cmicdo.com> wrote: 
>  > On Thursday, July 29, 2021, 11:28:03 AM EDT, Stephen Frost <sfrost@snowman.net> wrote:
>  > > * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>  > > > Stephen Frost <sfrost@snowman.net> writes:
>  > > > > Indeed, that comment didn't seem to help clear things up. I'm guessing Dave
>  > > > > is referring to the fact that we have a separate "gitmaster" server, which
>  > > > > is also maintained by pginfra and is where committers actually push changes
>  > > > > to, and then that is mirrored to git.postgresql.org.  I didn't check which
>  > > > > repo the tarball building script pulls from (which is run on pginfra, in
>  > > > > case anyone is wondering about that) and perhaps it pulls from gitmaster
>  > > > > and not git.p.o.
>  > > >
>  > > > It does pull from gitmaster.  There are multiple reasons for this design,
>  > > > but one is that a compromise of our public git server wouldn't imperil
>  > > > the contents of the official tarballs.
>  > >
>  > > That doesn't do much for the large number of folks who use
>  > > git.postgresql.org or the github mirror though, unfortunately.  Signed
>  > > commits, on the other hand, would help.
>  >
>  > A slightly different tack on this question:  How quickly would you
>  > notice that a rogue RPM had been inserted into the repo and then be
>  > able to fix it?
>  >
>  > By someone other than the trusted RPM builder ?
> Yes.

No idea, it really depends on a lot of factors such as exactly how it
was put in place and when it ends up being reported (and quite possibly
where, for that matter..).  We do regularly re-sync from the primary FTP
server to the others, so it would also depend on which system was first
compromised- the build server, the ftp primary server, or one of the
other ftp servers.  Also, while the pginfra team has members from a few
different timezones, we certainly don't have anything like 24/7/365
coverage.  I'm sure there's things we could do to improve on this, but
we're also a volunteer group and there's only so many hours.  We'd be
happy to chat with folks who are interested in helping. :)

Thanks,

Stephen