Thread: [OT] Microsoft: Kubernetes clusters hacked in malware campaign via PostgreSQL

https://www.bleepingcomputer.com/news/security/microsoft-kubernetes-clusters-hacked-in-malware-campaign-via-postgresql/

On Tue, Jan 10, 2023 at 9:46 AM Magnus Hagander <magnus@hagander.net> wrote:
>
> On Tue, Jan 10, 2023, 15:42 Jeffrey Walton <noloader@gmail.com> wrote:
>>
>>
https://www.bleepingcomputer.com/news/security/microsoft-kubernetes-clusters-hacked-in-malware-campaign-via-postgresql/
>
> I think the most impressive part in that article is that they found and linked to the postgresql 7 documentation...

It looks like the article used an older version of the docs because
the link is broken for the newer version. When following the link to
the latest version of the docs, its results in a "Page not found".

Jeff

On Tue, 10 Jan 2023 at 17:54, Jeffrey Walton <noloader@gmail.com> wrote:
>
> On Tue, Jan 10, 2023 at 9:46 AM Magnus Hagander <magnus@hagander.net> wrote:
> >
> > On Tue, Jan 10, 2023, 15:42 Jeffrey Walton <noloader@gmail.com> wrote:
> >>
> >>
https://www.bleepingcomputer.com/news/security/microsoft-kubernetes-clusters-hacked-in-malware-campaign-via-postgresql/
> >
> > I think the most impressive part in that article is that they found and linked to the postgresql 7
documentation...
>
> It looks like the article used an older version of the docs because
> the link is broken for the newer version. When following the link to
> the latest version of the docs, its results in a "Page not found".

I wonder what was the vulnerability in Postgres that enabled "hackers"
to run malware? I've read the article and the linked ones and found no
causative link between Postgres and malware inside. Sorry, it seems
like baseless warnings, not a description of vulnerability. Maybe I
haven't got something?

Regards,
Pavel Borisov,
Supabase

On Tue, 10 Jan 2023 at 18:07, Magnus Hagander <magnus@hagander.net> wrote:
>
>
>
> On Tue, Jan 10, 2023 at 4:00 PM Pavel Borisov <pashkin.elfe@gmail.com> wrote:
>>
>> On Tue, 10 Jan 2023 at 17:54, Jeffrey Walton <noloader@gmail.com> wrote:
>> >
>> > On Tue, Jan 10, 2023 at 9:46 AM Magnus Hagander <magnus@hagander.net> wrote:
>> > >
>> > > On Tue, Jan 10, 2023, 15:42 Jeffrey Walton <noloader@gmail.com> wrote:
>> > >>
>> > >>
https://www.bleepingcomputer.com/news/security/microsoft-kubernetes-clusters-hacked-in-malware-campaign-via-postgresql/
>> > >
>> > > I think the most impressive part in that article is that they found and linked to the postgresql 7
documentation...
>> >
>> > It looks like the article used an older version of the docs because
>> > the link is broken for the newer version. When following the link to
>> > the latest version of the docs, its results in a "Page not found".
>
>
> The page simply doesn't exist, because the information is sperad out across multiple places. There is indeed a bug in
thata link is generated to /current/ even if that page does not exist. But the information that's on there is also
wildlyout of date. This page was removed from the documentation in 2001, over 20 years ago. Linking to such obsolete
pagesin an article from 2023 doesn't exactly inspire confidence. 
>
>
>
>> I wonder what was the vulnerability in Postgres that enabled "hackers"
>> to run malware? I've read the article and the linked ones and found no
>> causative link between Postgres and malware inside. Sorry, it seems
>> like baseless warnings, not a description of vulnerability. Maybe I
>> haven't got something?
>
>
> There is no vulnerability in postgres. They are exploiting incorrectly *configured* postgres instances that allow
unauthenticatedusers to log in as superuser, which by definition means the system is configured to allow arbitrary
usersto upload and run arbitrary code -- which they did. Similar to leaving the ssh port open to the world for a user
witha default name and no password. 
>
Oh, I see then. They edited pg_hba.conf (in the link
https://www.bigbinary.com/blog/how-my-server-got-infected-with-a-crypto-mining-malware-and-how-I-fixed-it
from the article by OP) but stopped short not describing how exactly.
That's the clue. Thanks!

Regards,
Pavel

On Tue, Jan 10, 2023 at 10:20 AM Pavel Borisov <pashkin.elfe@gmail.com> wrote:
> On Tue, 10 Jan 2023 at 18:07, Magnus Hagander <magnus@hagander.net> wrote:
> > [ ...]
> >> I wonder what was the vulnerability in Postgres that enabled "hackers"
> >> to run malware? I've read the article and the linked ones and found no
> >> causative link between Postgres and malware inside. Sorry, it seems
> >> like baseless warnings, not a description of vulnerability. Maybe I
> >> haven't got something?

From the article Pavel linked to (below), it looks like PostgreSQL may
suffer from CWE-521, Weak Password Requirements.

Well designed systems today reject weak and wounded passwords out of
the box. Users don't need to do something special to enjoy the
benefit.

Now if a user pulls out the foot gun and disables strong password
requirements, then the user created the misconfiguration and the user
is at fault. If the user did nothing out of the ordinary, then I would
look for a design flaw, like letting users use weak passwords in the
first place.

> > There is no vulnerability in postgres. They are exploiting incorrectly *configured* postgres instances that allow
unauthenticatedusers to log in as superuser, which by definition means the system is configured to allow arbitrary
usersto upload and run arbitrary code -- which they did. Similar to leaving the ssh port open to the world for a user
witha default name and no password. 
> >
> Oh, I see then. They edited pg_hba.conf (in the link
> https://www.bigbinary.com/blog/how-my-server-got-infected-with-a-crypto-mining-malware-and-how-I-fixed-it
> from the article by OP) but stopped short not describing how exactly.
> That's the clue. Thanks!

Jeff

On Tue, Jan 10, 2023 at 10:48 AM Magnus Hagander <magnus@hagander.net> wrote:
>
> On Tue, Jan 10, 2023 at 4:38 PM Jeffrey Walton <noloader@gmail.com> wrote:
>>
>> On Tue, Jan 10, 2023 at 10:20 AM Pavel Borisov <pashkin.elfe@gmail.com> wrote:
>> > On Tue, 10 Jan 2023 at 18:07, Magnus Hagander <magnus@hagander.net> wrote:
>> > > [ ...]
>> > >> I wonder what was the vulnerability in Postgres that enabled "hackers"
>> > >> to run malware? I've read the article and the linked ones and found no
>> > >> causative link between Postgres and malware inside. Sorry, it seems
>> > >> like baseless warnings, not a description of vulnerability. Maybe I
>> > >> haven't got something?
>>
>> From the article Pavel linked to (below), it looks like PostgreSQL may
>> suffer from CWE-521, Weak Password Requirements.
>>
>> Well designed systems today reject weak and wounded passwords out of
>> the box. Users don't need to do something special to enjoy the
>> benefit.
>
> The default PostgreSQL installation on most platforms doesn't even allow password based logins. And it doesn't allow
connectionsacross the network at all. And it most definitely doesn't assign any weak default passwords. 

CWE-521 does not require the system to generate a weak password. It is
triggered when a weak password is used for authentication.

>> Now if a user pulls out the foot gun and disables strong password
>> requirements, then the user created the misconfiguration and the user
>> is at fault. If the user did nothing out of the ordinary, then I would
>> look for a design flaw, like letting users use weak passwords in the
>> first place.
>
> The reference in the first article is to "trust" authentication, which is even worse than that -- it is explicitly
askingpostgres to "turn off all authentication". 
>
> The second article doesn't actually contain anything more than a guess that maybe the password was weak. But the core
problemthere more seems to be to expose the postgres port to the public with no restrictions at all - one should
*never*do that with the database port, regardless of database. It is correct that postgres does not itself have any
defenceagainst a brute force attack if you use the built-in password auth (if you use an integrated authentication
method,that of course depends on the method it's being integrated with, but for simple passwords it doesn't). 

Agreed.

But stepping back a bit, where is the PostgreSQL security guide or
hardening guide? I would expect the document to cover these topics
(and more). Earlier it was said, "The page simply doesn't exist,
because the information is spread out across multiple places". Sending
users on a scavenger hunt is not a good approach to the problem. Maybe
all of this points to a gap in documentation?

Jeff

On Tue, 10 Jan 2023 at 18:58, Jeffrey Walton <noloader@gmail.com> wrote:
>
> On Tue, Jan 10, 2023 at 10:48 AM Magnus Hagander <magnus@hagander.net> wrote:
> >
> > On Tue, Jan 10, 2023 at 4:38 PM Jeffrey Walton <noloader@gmail.com> wrote:
> >>
> >> On Tue, Jan 10, 2023 at 10:20 AM Pavel Borisov <pashkin.elfe@gmail.com> wrote:
> >> > On Tue, 10 Jan 2023 at 18:07, Magnus Hagander <magnus@hagander.net> wrote:
> >> > > [ ...]
> >> > >> I wonder what was the vulnerability in Postgres that enabled "hackers"
> >> > >> to run malware? I've read the article and the linked ones and found no
> >> > >> causative link between Postgres and malware inside. Sorry, it seems
> >> > >> like baseless warnings, not a description of vulnerability. Maybe I
> >> > >> haven't got something?
> >>
> >> From the article Pavel linked to (below), it looks like PostgreSQL may
> >> suffer from CWE-521, Weak Password Requirements.
> >>
> >> Well designed systems today reject weak and wounded passwords out of
> >> the box. Users don't need to do something special to enjoy the
> >> benefit.
> >
> > The default PostgreSQL installation on most platforms doesn't even allow password based logins. And it doesn't
allowconnections across the network at all. And it most definitely doesn't assign any weak default passwords. 
>
> CWE-521 does not require the system to generate a weak password. It is
> triggered when a weak password is used for authentication.
>
> >> Now if a user pulls out the foot gun and disables strong password
> >> requirements, then the user created the misconfiguration and the user
> >> is at fault. If the user did nothing out of the ordinary, then I would
> >> look for a design flaw, like letting users use weak passwords in the
> >> first place.
> >
> > The reference in the first article is to "trust" authentication, which is even worse than that -- it is explicitly
askingpostgres to "turn off all authentication". 
> >
> > The second article doesn't actually contain anything more than a guess that maybe the password was weak. But the
coreproblem there more seems to be to expose the postgres port to the public with no restrictions at all - one should
*never*do that with the database port, regardless of database. It is correct that postgres does not itself have any
defenceagainst a brute force attack if you use the built-in password auth (if you use an integrated authentication
method,that of course depends on the method it's being integrated with, but for simple passwords it doesn't). 
>
> Agreed.
>
> But stepping back a bit, where is the PostgreSQL security guide or
> hardening guide? I would expect the document to cover these topics
> (and more). Earlier it was said, "The page simply doesn't exist,
> because the information is spread out across multiple places". Sending
> users on a scavenger hunt is not a good approach to the problem. Maybe
> all of this points to a gap in documentation?

I don't think there is a gap in a documentation. The page
https://www.postgresql.org/docs/current/client-authentication.html
addresses auth options in a very detailed way. The only problem is
that 10+ years old page doesn't link to it.

Regarding the fact that using weak passwords isn't limited, I think
anyone can write an extension that limits (or advises against) weak
passwords based on the author's definition of weakness, like checking
by dictionary or so. That's the power of Postgres extensibility. I
don't like an idea that Postgres core should enforce some arbitrary
things like "weakness".

Regards,
Pavel Borisov.

On Tue, 10 Jan 2023 at 15:07, Magnus Hagander <magnus@hagander.net> wrote:
>
> The page simply doesn't exist, because the information is sperad out across multiple places. There is indeed a bug in
thata link is generated to /current/ even if that page does not exist. But the information that's on there is also
wildlyout of date. This page was removed from the documentation in 2001, over 20 years ago. Linking to such obsolete
pagesin an article from 2023 doesn't exactly inspire confidence. 
>

A Google search for "postgresql security" returns that page as the top
non-featured result.

Looking at the source of that page, it has <meta name="robots"
content="nofollow" />. Changing that to "noindex" might help.

Regards,
Dean

On Tue, Jan 10, 2023 at 12:17 PM Dean Rasheed <dean.a.rasheed@gmail.com> wrote:
>
> On Tue, 10 Jan 2023 at 15:07, Magnus Hagander <magnus@hagander.net> wrote:
> >
> > The page simply doesn't exist, because the information is sperad out across multiple places. There is indeed a bug
inthat a link is generated to /current/ even if that page does not exist. But the information that's on there is also
wildlyout of date. This page was removed from the documentation in 2001, over 20 years ago. Linking to such obsolete
pagesin an article from 2023 doesn't exactly inspire confidence. 
> >
>
> A Google search for "postgresql security" returns that page as the top
> non-featured result.
>
> Looking at the source of that page, it has <meta name="robots"
> content="nofollow" />. Changing that to "noindex" might help.

https://www.google.com/search?q=postgresql+security+hardening+guide
returns one result hosted on the PostgreSQL webserver. It is the old
article that has been condemned by the community.

Jeff

Pavel Borisov <pashkin.elfe@gmail.com> writes:
> Regarding the fact that using weak passwords isn't limited, I think
> anyone can write an extension that limits (or advises against) weak
> passwords based on the author's definition of weakness, like checking
> by dictionary or so. That's the power of Postgres extensibility. I
> don't like an idea that Postgres core should enforce some arbitrary
> things like "weakness".

It's less easy than it sounds, actually, because current best practice
is that the server will never see the cleartext password at all, so
you can't apply the usual litany of weak-password checks.  About all
you can do is apply a brute force rainbow attack, which is far from
cheap (although I guess the DBA could run such checks in the background).
Even that might be rendered infeasible if the password is stored as
a SCRAM verifier; I'm not sure about that.  Anyway, this is why
contrib/passwordcheck is a development backwater rather than something
people actually use.

            regards, tom lane

Greetings,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Pavel Borisov <pashkin.elfe@gmail.com> writes:
> > Regarding the fact that using weak passwords isn't limited, I think
> > anyone can write an extension that limits (or advises against) weak
> > passwords based on the author's definition of weakness, like checking
> > by dictionary or so. That's the power of Postgres extensibility. I
> > don't like an idea that Postgres core should enforce some arbitrary
> > things like "weakness".
>
> It's less easy than it sounds, actually, because current best practice
> is that the server will never see the cleartext password at all, so
> you can't apply the usual litany of weak-password checks.  About all
> you can do is apply a brute force rainbow attack, which is far from
> cheap (although I guess the DBA could run such checks in the background).
> Even that might be rendered infeasible if the password is stored as
> a SCRAM verifier; I'm not sure about that.  Anyway, this is why
> contrib/passwordcheck is a development backwater rather than something
> people actually use.

For better or worse, people do actually use it and it would be much
better if we'd actually allow some effort to be put into improving this
situation.  I'm all for better choices of authentication methods, but
that simply isn't always an option and there are cases where DBAs have
to be able to require passwords on select accounts be set via cleartext
to allow passwordcheck and similar tools to enforce password complexity
requirements.

Pushing back against that means fewer folks will use PostgreSQL because
it isn't able to reasonably address this requirement (unlike most of the
other databases out there..), and that's unfortunate.

Thanks,

Stephen