Thread: Document that server will start even if it's unable to open some TCP/IP ports

The attached patch clarifies that the server will start even if it is
unable to open the port on some of the TCP/IP addresses listed (or
implied by a value of '*' or localhost) in listen_addresses parameter.

I think it is important to call this out, because I was surprised to
see that the server started even though the port was occupied by
another process. Upon close inspection, I noticed that the other
process was using that port on 127.0.0.1, so Postgres complained about
that interface (a warning in server log), but it was able to open the
port on IPv6 ::1, so it started up as normal.

Upon further testing, I saw that server will not start only if it is
unable to open the port on _all_ the interfaces/addresses. It it's
able to open the port on at least one, the server will start.

If listen_addresses is empty, then server won't try to open any TCP/IP
ports. The patch does not change any language related to that.

Best regards,
Gurjeet
http://Gurje.et

On Sat, May 27, 2023 at 03:17:21PM -0700, Gurjeet Singh wrote:
> Upon further testing, I saw that server will not start only if it is
> unable to open the port on _all_ the interfaces/addresses. It it's
> able to open the port on at least one, the server will start.

This surprised me.  I would've expected the server to fail to start if it
failed for anything in listen_addresses.  After some digging, I found what
I believe is the original justification [0] as well as a follow-up thread
[1] that seems to call out kernel support for IPv6 as the main objection.
Perhaps it is time to reevaluate this decision.

> If listen_addresses is empty, then server won't try to open any TCP/IP
> ports. The patch does not change any language related to that.

Your proposed change notes that the server only starts if it can listen on
at least one TCP/IP address, which I worry might lead folks to think that
the server won't start if listen_addresses is empty.

[0] https://postgr.es/m/6739.1079384078%40sss.pgh.pa.us
[1] https://postgr.es/m/200506281149.51696.peter_e%40gmx.net

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

On Mon, Jun 12, 2023 at 10:59 PM Nathan Bossart
<nathandbossart@gmail.com> wrote:
> On Sat, May 27, 2023 at 03:17:21PM -0700, Gurjeet Singh wrote:
> > If listen_addresses is empty, then server won't try to open any TCP/IP
> > ports. The patch does not change any language related to that.
>
> Your proposed change notes that the server only starts if it can listen on
> at least one TCP/IP address, which I worry might lead folks to think that
> the server won't start if listen_addresses is empty.

Perhaps we can prefix that statement with "If listen_addresses is not
empty", like so:

--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -661,3 +661,9 @@ include_dir 'conf.d'
          which allows only local TCP/IP <quote>loopback</quote>
connections to be
-         made.  While client authentication (<xref
+         made.  If <varname>listen_addresses</varname> is not empty, the server
+         will start only if it can open the <varname>port</varname>
+         on at least one TCP/IP address.  If server is unable to open
+         <varname>port</varname> on a TCP/IP address, it emits a warning.
+       <para>
+       </para>
+         While client authentication (<xref
          linkend="client-authentication"/>) allows fine-grained control




Best regards,
Gurjeet
http://Gurje.et

On Mon, Jun 12, 2023 at 11:57:45PM -0700, Gurjeet Singh wrote:
> Perhaps we can prefix that statement with "If listen_addresses is not
> empty", like so:

Before we spend too much time trying to document the current behavior, I
think we should see if we can change it to something less surprising (i.e.,
failing to start if the server fails for any address).  The original
objections around kernel support for IPv6 might no longer stand.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

Nathan Bossart <nathandbossart@gmail.com> writes:
> Before we spend too much time trying to document the current behavior, I
> think we should see if we can change it to something less surprising (i.e.,
> failing to start if the server fails for any address).  The original
> objections around kernel support for IPv6 might no longer stand.

I think that'd be more surprising not less.

The systemd guys certainly believe that daemons ought to auto-adapt
to changes in the machine's internet connectivity.  We aren't there
yet, but I can imagine somebody trying to fix that someday soon.
If the postmaster is able to dynamically acquire and drop ports then
it would certainly not make sense to behave as you suggest.

            regards, tom lane

On Tue, Jun 13, 2023 at 04:28:31PM -0400, Tom Lane wrote:
> Nathan Bossart <nathandbossart@gmail.com> writes:
>> Before we spend too much time trying to document the current behavior, I
>> think we should see if we can change it to something less surprising (i.e.,
>> failing to start if the server fails for any address).  The original
>> objections around kernel support for IPv6 might no longer stand.
> 
> I think that'd be more surprising not less.

The reason it surprises me is because it creates uncertainty about the
server configuration.  Granted, I could look in the logs for any warnings,
but I'm not sure that's the best experience.  I would expect this to work
more like huge_pages.  If I set huge_pages to "on", I know that the server
is using huge pages if it starts up.

> The systemd guys certainly believe that daemons ought to auto-adapt
> to changes in the machine's internet connectivity.  We aren't there
> yet, but I can imagine somebody trying to fix that someday soon.
> If the postmaster is able to dynamically acquire and drop ports then
> it would certainly not make sense to behave as you suggest.

Agreed, if listen_addresses became a PGC_SIGHUP parameter, it would make
sense to avoid shutting down the server if it was dynamically
misconfigured, as is done for the configuration files.  I think that
argument applies for changes in connectivity, too.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

At Tue, 13 Jun 2023 14:38:14 -0700, Nathan Bossart <nathandbossart@gmail.com> wrote in 
> On Tue, Jun 13, 2023 at 04:28:31PM -0400, Tom Lane wrote:
> > Nathan Bossart <nathandbossart@gmail.com> writes:
> >> Before we spend too much time trying to document the current behavior, I
> >> think we should see if we can change it to something less surprising (i.e.,
> >> failing to start if the server fails for any address).  The original
> >> objections around kernel support for IPv6 might no longer stand.
> > 
> > I think that'd be more surprising not less.
> 
> The reason it surprises me is because it creates uncertainty about the
> server configuration.  Granted, I could look in the logs for any warnings,
> but I'm not sure that's the best experience.  I would expect this to work
> more like huge_pages.  If I set huge_pages to "on", I know that the server
> is using huge pages if it starts up.
> 
> > The systemd guys certainly believe that daemons ought to auto-adapt
> > to changes in the machine's internet connectivity.  We aren't there
> > yet, but I can imagine somebody trying to fix that someday soon.
> > If the postmaster is able to dynamically acquire and drop ports then
> > it would certainly not make sense to behave as you suggest.
> 
> Agreed, if listen_addresses became a PGC_SIGHUP parameter, it would make
> sense to avoid shutting down the server if it was dynamically
> misconfigured, as is done for the configuration files.  I think that
> argument applies for changes in connectivity, too.

If I had to say, I would feel it rather surprising if server
successfully starts even when any explicitly-specified port can't be
opened (which is the current case). The current auto-adaption is fine
iff I use '*' for listen_addresses.  IMHO, for "reliable"
auto-adaption, we might want '[+-]?xxx.xxx.xxx.xxx/nn' (and the same
for v6), or '[+-]?interface-name' notation to require, allow, or
disallow to use specific networks.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Kyotaro Horiguchi <horikyota.ntt@gmail.com> writes:
> If I had to say, I would feel it rather surprising if server
> successfully starts even when any explicitly-specified port can't be
> opened (which is the current case).

There is certainly an argument that such a condition indicates that
something's very broken in our configuration and we should complain.
But I'm not sure how exciting the case is in practice.  The systemd
guys would really like us to be willing to come up before any network
interfaces are up, and then auto-listen to those interfaces when they
do come up.  On the other hand, the situation with Unix sockets is
much more static: if you can't make a socket in /tmp or /var/run at
the instant of postmaster start, it's unlikely you will be able to do
so later.

Maybe we need different rules for TCP versus Unix-domain sockets?
I'm not sure what exactly, but lumping those cases together for
a discussion like this feels wrong.

            regards, tom lane

On Tue, Jun 13, 2023 at 11:11:04PM -0400, Tom Lane wrote:
> Kyotaro Horiguchi <horikyota.ntt@gmail.com> writes:
> > If I had to say, I would feel it rather surprising if server
> > successfully starts even when any explicitly-specified port can't be
> > opened (which is the current case).
> 
> There is certainly an argument that such a condition indicates that
> something's very broken in our configuration and we should complain.
> But I'm not sure how exciting the case is in practice.  The systemd
> guys would really like us to be willing to come up before any network
> interfaces are up, and then auto-listen to those interfaces when they
> do come up.  On the other hand, the situation with Unix sockets is
> much more static: if you can't make a socket in /tmp or /var/run at
> the instant of postmaster start, it's unlikely you will be able to do
> so later.
> 
> Maybe we need different rules for TCP versus Unix-domain sockets?
> I'm not sure what exactly, but lumping those cases together for
> a discussion like this feels wrong.

If we are going to retry for network configuration changes, it seems we
would also retry Unix domain sockets for cases like when the permissions
are wrong, and then fixed.

However, it seem hard to figure out exactly what _is_ working if we take
the approach of dynamically retrying listen methods.  Do we report
anything helpful in the server logs when we start and can't listen on
anything?

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.

On Mon, Jun 19, 2023 at 5:48 PM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Tue, Jun 13, 2023 at 11:11:04PM -0400, Tom Lane wrote:
> >
> > There is certainly an argument that such a condition indicates that
> > something's very broken in our configuration and we should complain.
> > But I'm not sure how exciting the case is in practice.  The systemd
> > guys would really like us to be willing to come up before any network
> > interfaces are up, and then auto-listen to those interfaces when they
> > do come up.

That sounds like a reasonable expectation, as the network conditions
can change without any explicit changes made by someone.

> > On the other hand, the situation with Unix sockets is
> > much more static: if you can't make a socket in /tmp or /var/run at
> > the instant of postmaster start, it's unlikely you will be able to do
> > so later.

I think you're describing a setup where Postgres startup is automatic,
as part of server/OS startup. That is the most common case.

In cases where someone is performing a Postgres startup manually, they
are very likely to have the permissions to fix the problem preventing
the startup.

> > Maybe we need different rules for TCP versus Unix-domain sockets?
> > I'm not sure what exactly, but lumping those cases together for
> > a discussion like this feels wrong.

+1.

> If we are going to retry for network configuration changes, it seems we
> would also retry Unix domain sockets for cases like when the permissions
> are wrong, and then fixed.

The network managers (systemd, etc.) are expected to respond to
dynamic conditions, and hence they may perform network config changes
in response to things like network outages, and hardware failures,
time of day, etc.

On the other hand, the permissions required to create files for Unix
domain sockets are only expected to change if someone decides to make
that change. I wouldn't expect these permissions to be changed
dynamically.

On those grounds, keeping the treatment of Unix domain sockets out of
this discussion for this patch seems reasonable.

> However, it seem hard to figure out exactly what _is_ working if we take
> the approach of dynamically retrying listen methods.  Do we report
> anything helpful in the server logs when we start and can't listen on
> anything?

Yes. For every host listed in listen_addresses, if Postgres fails to
open the port on that address, we get a WARNING message in the server
log. After the end of processing of a non-empty listen_addresses, if
there are zero open TCP/IP connections, the server exits (with a FATAL
message, IIRC).

Best regards,
Gurjeet
http://Gurje.et

On Mon, Jun 12, 2023 at 11:57:45PM -0700, Gurjeet Singh wrote:
> On Mon, Jun 12, 2023 at 10:59 PM Nathan Bossart
> <nathandbossart@gmail.com> wrote:
> > On Sat, May 27, 2023 at 03:17:21PM -0700, Gurjeet Singh wrote:
> > > If listen_addresses is empty, then server won't try to open any TCP/IP
> > > ports. The patch does not change any language related to that.
> >
> > Your proposed change notes that the server only starts if it can listen on
> > at least one TCP/IP address, which I worry might lead folks to think that
> > the server won't start if listen_addresses is empty.
> 
> Perhaps we can prefix that statement with "If listen_addresses is not
> empty", like so:

I came up with a slightly modified doc patch, attached.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.

Thanks for picking this up.

On Thu, Sep 07, 2023 at 03:33:57PM -0400, Bruce Momjian wrote:
>           The default value is <systemitem class="systemname">localhost</systemitem>,
>           which allows only local TCP/IP <quote>loopback</quote> connections to be
> -         made.  While client authentication (<xref
> +         made.  If <varname>listen_addresses</varname> is not empty,
> +         the server will start if it can open a <varname>port</varname>
> +         on at least one TCP/IP address.  A warning will be emitted for
> +         any TCP/IP address which cannot be opened.

I think we should move this sentence to before the ѕentence about the
default value.  That way, "If the list is empty, ..." is immediately
followed by "If the list is not empty, ..."

IMO the phrase "open a port" is kind of nonstandard.  I think we should say
something along the lines of

    If listen_addresses is not empty, the server will start only if it can
    listen on at least one of the specified addresses.  A warning will be
    emitted for any addresses that the server cannot listen on.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

On Thu, Sep  7, 2023 at 02:54:13PM -0700, Nathan Bossart wrote:
> Thanks for picking this up.
> 
> On Thu, Sep 07, 2023 at 03:33:57PM -0400, Bruce Momjian wrote:
> >           The default value is <systemitem class="systemname">localhost</systemitem>,
> >           which allows only local TCP/IP <quote>loopback</quote> connections to be
> > -         made.  While client authentication (<xref
> > +         made.  If <varname>listen_addresses</varname> is not empty,
> > +         the server will start if it can open a <varname>port</varname>
> > +         on at least one TCP/IP address.  A warning will be emitted for
> > +         any TCP/IP address which cannot be opened.
> 
> I think we should move this sentence to before the ѕentence about the
> default value.  That way, "If the list is empty, ..." is immediately
> followed by "If the list is not empty, ..."
> 
> IMO the phrase "open a port" is kind of nonstandard.  I think we should say
> something along the lines of
> 
>     If listen_addresses is not empty, the server will start only if it can
>     listen on at least one of the specified addresses.  A warning will be
>     emitted for any addresses that the server cannot listen on.

Good idea, updated patch attached.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.

On Thu, Sep 07, 2023 at 07:13:44PM -0400, Bruce Momjian wrote:
> On Thu, Sep  7, 2023 at 02:54:13PM -0700, Nathan Bossart wrote:
>> IMO the phrase "open a port" is kind of nonstandard.  I think we should say
>> something along the lines of
>> 
>>     If listen_addresses is not empty, the server will start only if it can
>>     listen on at least one of the specified addresses.  A warning will be
>>     emitted for any addresses that the server cannot listen on.
> 
> Good idea, updated patch attached.

I still think we should say "listen on an address" instead of "open a
port," but otherwise it LGTM.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

On Thu, Sep  7, 2023 at 09:21:07PM -0700, Nathan Bossart wrote:
> On Thu, Sep 07, 2023 at 07:13:44PM -0400, Bruce Momjian wrote:
> > On Thu, Sep  7, 2023 at 02:54:13PM -0700, Nathan Bossart wrote:
> >> IMO the phrase "open a port" is kind of nonstandard.  I think we should say
> >> something along the lines of
> >> 
> >>     If listen_addresses is not empty, the server will start only if it can
> >>     listen on at least one of the specified addresses.  A warning will be
> >>     emitted for any addresses that the server cannot listen on.
> > 
> > Good idea, updated patch attached.
> 
> I still think we should say "listen on an address" instead of "open a
> port," but otherwise it LGTM.

Agreed, I never liked the "port" mention.  I couldn't figure how to get
"open" out of the warning sentence though.  Updated patch attached.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.

On Fri, Sep 08, 2023 at 10:52:10AM -0400, Bruce Momjian wrote:
> On Thu, Sep  7, 2023 at 09:21:07PM -0700, Nathan Bossart wrote:
>> On Thu, Sep 07, 2023 at 07:13:44PM -0400, Bruce Momjian wrote:
>> > On Thu, Sep  7, 2023 at 02:54:13PM -0700, Nathan Bossart wrote:
>> >> IMO the phrase "open a port" is kind of nonstandard.  I think we should say
>> >> something along the lines of
>> >> 
>> >>     If listen_addresses is not empty, the server will start only if it can
>> >>     listen on at least one of the specified addresses.  A warning will be
>> >>     emitted for any addresses that the server cannot listen on.
>> > 
>> > Good idea, updated patch attached.
>> 
>> I still think we should say "listen on an address" instead of "open a
>> port," but otherwise it LGTM.
> 
> Agreed, I never liked the "port" mention.  I couldn't figure how to get
> "open" out of the warning sentence though.  Updated patch attached.

WFM

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

On Fri, Sep 8, 2023 at 7:52 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Thu, Sep  7, 2023 at 09:21:07PM -0700, Nathan Bossart wrote:
> > On Thu, Sep 07, 2023 at 07:13:44PM -0400, Bruce Momjian wrote:
> > > On Thu, Sep  7, 2023 at 02:54:13PM -0700, Nathan Bossart wrote:
> > >> IMO the phrase "open a port" is kind of nonstandard.  I think we should say
> > >> something along the lines of
> > >>
> > >>    If listen_addresses is not empty, the server will start only if it can
> > >>    listen on at least one of the specified addresses.  A warning will be
> > >>    emitted for any addresses that the server cannot listen on.
> > >
> > > Good idea, updated patch attached.
> >
> > I still think we should say "listen on an address" instead of "open a
> > port," but otherwise it LGTM.
>
> Agreed, I never liked the "port" mention.  I couldn't figure how to get
> "open" out of the warning sentence though.  Updated patch attached.

LGTM.

Best regards,
Gurjeet
http://Gurje.et

On Tue, Sep 12, 2023 at 05:25:44PM -0700, Gurjeet Singh wrote:
> On Fri, Sep 8, 2023 at 7:52 AM Bruce Momjian <bruce@momjian.us> wrote:
> >
> > On Thu, Sep  7, 2023 at 09:21:07PM -0700, Nathan Bossart wrote:
> > > On Thu, Sep 07, 2023 at 07:13:44PM -0400, Bruce Momjian wrote:
> > > > On Thu, Sep  7, 2023 at 02:54:13PM -0700, Nathan Bossart wrote:
> > > >> IMO the phrase "open a port" is kind of nonstandard.  I think we should say
> > > >> something along the lines of
> > > >>
> > > >>    If listen_addresses is not empty, the server will start only if it can
> > > >>    listen on at least one of the specified addresses.  A warning will be
> > > >>    emitted for any addresses that the server cannot listen on.
> > > >
> > > > Good idea, updated patch attached.
> > >
> > > I still think we should say "listen on an address" instead of "open a
> > > port," but otherwise it LGTM.
> >
> > Agreed, I never liked the "port" mention.  I couldn't figure how to get
> > "open" out of the warning sentence though.  Updated patch attached.
> 
> LGTM.

Patch applied back to PG 11.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Only you can decide what is important to you.

On Tue, Sep 26, 2023 at 4:02 PM Bruce Momjian <bruce@momjian.us> wrote:
>
> Patch applied back to PG 11.

+Peter E. since I received the following automated note:

> Closed in commitfest 2023-09 with status: Moved to next CF (petere)

Just a note that this patch has been committed (3fea854691), so I have
marked the CF item [1] as 'Committed', and specified Bruce as the
committer.

[1]: https://commitfest.postgresql.org/45/4333/


Best regards,
Gurjeet
http://Gurje.et