Thread: Ability to listen on two unix sockets
Hi, before I ask the main question, just a little background for one issue we're currently having in Fedora 17: PrivateTmp is a systemd's feature, which allows to have private /tmp directory for services, which in turn means that such services aren't able to access systems's /tmp directory. It's been enabled by some services already, including Apache, while PostgreSQL uses system's /tmp directory, where its unix socket is located. Naturally, it resulted in a state, where Apache or other services with PrivateTmp enabled are not able to communicate with PostgreSQL using the socket. Since we don't want just to move socket for compatibility reasons, I'm going to prepare a draft patch to allow PostgreSQL to use a second unix socket at a time. A question I'd like to ask now is: Do we need a new configuration variable for this or it's enough to have the location hard-coded? What are your opinions? Regards, Honza
On Jun6, 2012, at 15:50 , Honza Horak wrote: > before I ask the main question, just a little background for one issue we're currently having in Fedora 17: > > PrivateTmp is a systemd's feature, which allows to have private /tmp directory for services, which in turn means that suchservices aren't able to access systems's /tmp directory. It's been enabled by some services already, including Apache,while PostgreSQL uses system's /tmp directory, where its unix socket is located. Naturally, it resulted in a state,where Apache or other services with PrivateTmp enabled are not able to communicate with PostgreSQL using the socket. Couldn't you simply tell postgres to put it's socket in, say, /var/run, and create a symlink to that socket in the global/tmp directory? > Since we don't want just to move socket for compatibility reasons, I'm going to prepare a draft patch to allow PostgreSQLto use a second unix socket at a time. A question I'd like to ask now is: Do we need a new configuration variablefor this or it's enough to have the location hard-coded? What are your opinions? If we're going to have this at all, we should go all the way and support an arbitrary number of sockets. But then, is thereany advantage in providing this feature natively compare to simply creating symlinks? best regards, Florian Pflug
Florian Pflug <fgp@phlo.org> writes: > Couldn't you simply tell postgres to put it's socket in, say, /var/run, and create a symlink to that socket in the global/tmp directory? FYI, this proposal emerged out of a discussion between Honza and myself. "Use a symlink" was my first idea too, but on reflection it seems like it will take less new code to support two sockets. We already support multiple TCP sockets, so multiple Unix sockets shouldn't be that much extra trouble. The reasons a symlink doesn't seem attractive are: 1. The code to create/delete it has to be in the postmaster. If we tried to make the Fedora-specific startup script manage it, we would first have to teach that script how to know which port number the postmaster will select, which means parsing config files. Ugh. 2. What if two postmasters try to create a symlink in the same place? Or we're just trying to decide if the previous creator crashed without removing it? So we need a lockfile beside it. So at this point we are building a whole bunch of new infrastructure to create symlinks, whereas we can probably just call the same subroutine twice if we go with the two-socket design. > If we're going to have this at all, we should go all the way and > support an arbitrary number of sockets. Well, that's what I wanted to discuss before Honza starts coding. It's not obvious that there are any use-cases for more than two. It's also not clear whether there is any value in supporting run-time rather than build-time configuration of the socket locations. The Fedora use-case has no need of that, but if people can point to other cases where it would be sensible, we can write the patch that way. You might think we should design this exactly like the TCP-socket multiple-listen-addresses case, ie just have a config variable containing a list of directory names. The sticking point there is that the directories aren't really interchangeable. In particular, there is still going to be one directory that is the one hard-wired into libpq. So whereas multiple TCP sockets really are pretty much interchangeable, I think in the Unix-socket case we are going to have to think of it as being a primary socket and one or more alternate sockets. Is there a reason to have more than one alternate, and if so what is the use-case? (BTW, we would probably just adopt the Debian solution if we were sure there were no non-libpq clients out there; but we aren't.) regards, tom lane
On Wednesday, June 06, 2012 04:38:42 PM Tom Lane wrote: > Florian Pflug <fgp@phlo.org> writes: > > If we're going to have this at all, we should go all the way and > > support an arbitrary number of sockets. > > Well, that's what I wanted to discuss before Honza starts coding. > It's not obvious that there are any use-cases for more than two. > It's also not clear whether there is any value in supporting run-time > rather than build-time configuration of the socket locations. The > Fedora use-case has no need of that, but if people can point to other > cases where it would be sensible, we can write the patch that way. I had the need to make pg available from multiple chroots via unix sockets. The same might come up more frequently with the availability of filesystem namespaces... > You might think we should design this exactly like the TCP-socket > multiple-listen-addresses case, ie just have a config variable > containing a list of directory names. The sticking point there is > that the directories aren't really interchangeable. In particular, > there is still going to be one directory that is the one hard-wired > into libpq. I wonder if the whole issue doesn't require libpq to also try multiple hardcoded socket locations. Andres -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Excerpts from Tom Lane's message of mié jun 06 10:38:42 -0400 2012: > Florian Pflug <fgp@phlo.org> writes: > > Couldn't you simply tell postgres to put it's socket in, say, /var/run, and create a symlink to that socket in the global/tmp directory? > > FYI, this proposal emerged out of a discussion between Honza and > myself. "Use a symlink" was my first idea too, but on reflection > it seems like it will take less new code to support two sockets. > We already support multiple TCP sockets, so multiple Unix sockets > shouldn't be that much extra trouble. > > The reasons a symlink doesn't seem attractive are: > > 1. The code to create/delete it has to be in the postmaster. If we > tried to make the Fedora-specific startup script manage it, we would > first have to teach that script how to know which port number the > postmaster will select, which means parsing config files. Ugh. Well, you could use postmaster -C port The other reason seems compelling enough, though ... particularly, handling a lockfile sounds messy; if it's a symlink and it's created by the script, then it would need a separate lockfile, and filling its data wouldn't be exactly trivial. > (BTW, we would probably just adopt the Debian solution if we were > sure there were no non-libpq clients out there; but we aren't.) Maybe this is a good time to make the /var/run socket location (Debian's choice) the primary one, and /tmp be the alternate. -- Álvaro Herrera <alvherre@commandprompt.com> The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On 06/06/2012 04:50 PM, Andres Freund wrote: > On Wednesday, June 06, 2012 04:38:42 PM Tom Lane wrote: >> Florian Pflug<fgp@phlo.org> writes: > >>> If we're going to have this at all, we should go all the way and >>> support an arbitrary number of sockets. >> >> Well, that's what I wanted to discuss before Honza starts coding. >> It's not obvious that there are any use-cases for more than two. >> It's also not clear whether there is any value in supporting run-time >> rather than build-time configuration of the socket locations. The >> Fedora use-case has no need of that, but if people can point to other >> cases where it would be sensible, we can write the patch that way. > I had the need to make pg available from multiple chroots via unix sockets. > The same might come up more frequently with the availability of filesystem > namespaces... It seems you were not alone with such need: http://archives.postgresql.org/pgsql-novice/2006-09/msg00172.php Honza
Alvaro Herrera <alvherre@commandprompt.com> writes: > Excerpts from Tom Lane's message of mié jun 06 10:38:42 -0400 2012: >> (BTW, we would probably just adopt the Debian solution if we were >> sure there were no non-libpq clients out there; but we aren't.) > Maybe this is a good time to make the /var/run socket location (Debian's > choice) the primary one, and /tmp be the alternate. I'm not really in favor of making /var/run be the out-of-the-box default, because it would discriminate against personal/testing installations (ie, you couldn't set it up without root privileges). It's a reasonable choice for distro-provided packages, but not so much for one-off builds --- think about the buildfarm if nothing else. Having said that, if we made it easier to configure things that way than by patching the source, I bet Martin Pitt isn't going to object. regards, tom lane
Honza Horak <hhorak@redhat.com> writes: > On 06/06/2012 04:50 PM, Andres Freund wrote: >> On Wednesday, June 06, 2012 04:38:42 PM Tom Lane wrote: >>> Florian Pflug<fgp@phlo.org> writes: >>>> If we're going to have this at all, we should go all the way and >>>> support an arbitrary number of sockets. >>> Well, that's what I wanted to discuss before Honza starts coding. >>> It's not obvious that there are any use-cases for more than two. >>> It's also not clear whether there is any value in supporting run-time >>> rather than build-time configuration of the socket locations. The >>> Fedora use-case has no need of that, but if people can point to other >>> cases where it would be sensible, we can write the patch that way. >> I had the need to make pg available from multiple chroots via unix sockets. >> The same might come up more frequently with the availability of filesystem >> namespaces... > It seems you were not alone with such need: > http://archives.postgresql.org/pgsql-novice/2006-09/msg00172.php I had forgotten that conversation, but it does seem like there is interest in this type of configuration. Can anybody confirm that dropping a socket into a chroot or jail would actually work, ie make it possible to connect from inside the chroot to a postmaster running outside? If that's real and not just wishful thinking, it seems like enough of an argument to justify supporting N sockets. regards, tom lane
On Wed, Jun 06, 2012 at 11:32:45AM -0400, Tom Lane wrote: > I had forgotten that conversation, but it does seem like there is > interest in this type of configuration. Can anybody confirm that > dropping a socket into a chroot or jail would actually work, ie > make it possible to connect from inside the chroot to a postmaster > running outside? If that's real and not just wishful thinking, > it seems like enough of an argument to justify supporting N sockets. We need to deal with exactly this sort of issue with schroot, where we may want to provide programs in the chroot with access to facilities outside the chroot. We generally just bind mount in the minimal set of stuff needed. This might mean binding just the socket, or it could be /var/run/postgresql. We do this for the X11 socket for our desktop configuration profile to permit X11 programs to run in a chroot, though we currently bind mount all of /tmp rather than just the socket, since we want that as well in any case. (http://people.debian.org/~rleigh/schroot.pdf) While not exactly what was proposed (multiple sockets), this allows one to re-use a single socket without the daemon requiring any special support for it. Regards, Roger -- .''`. Roger Leigh: :' : Debian GNU/Linux http://people.debian.org/~rleigh/`. `' schroot and sbuild http://alioth.debian.org/projects/buildd-tools `- GPG Public Key F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800
On Wed, Jun 6, 2012 at 10:38 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Well, that's what I wanted to discuss before Honza starts coding. > It's not obvious that there are any use-cases for more than two. > It's also not clear whether there is any value in supporting run-time > rather than build-time configuration of the socket locations. The > Fedora use-case has no need of that, but if people can point to other > cases where it would be sensible, we can write the patch that way. > > You might think we should design this exactly like the TCP-socket > multiple-listen-addresses case, ie just have a config variable > containing a list of directory names. The sticking point there is > that the directories aren't really interchangeable. In particular, > there is still going to be one directory that is the one hard-wired > into libpq. So whereas multiple TCP sockets really are pretty much > interchangeable, I think in the Unix-socket case we are going to have > to think of it as being a primary socket and one or more alternate > sockets. Is there a reason to have more than one alternate, and if > so what is the use-case? > > (BTW, we would probably just adopt the Debian solution if we were > sure there were no non-libpq clients out there; but we aren't.) I recently had an urge to make it possible for the postmaster to listen on multiple ports and even went so far as to code up a patch to allow that. It still applies, with offsets, so I'll attach it here. So I guess I'm +1 on the idea of allowing N UNIX sockets rather than limiting it to N=2, and really I'd like to do one better and allow listening on multiple TCP ports as well. Since the PID file contains the port number, multiple TCP sockets stop being interchangeable as soon as you allow multiple ports, but that's not very difficult to handle. Now, you might ask whether this has any real-world value, and obviously I'm going to say yes or I wouldn't be proposing it. The reason for wanting multiple UNIX sockets is because those sockets might be in different places that are not all equally accessible to everyone, because of things like chroot. But of course the same thing is possible in the network space using iptables and similar tools. For example, you might want to have users connect to application A using port 5432, and to application B using port 15432. Now you can use network monitoring tools to see how much data each application is sending and receiving, without needing deep packet inspection. You can firewall those ports differently to provide access to different groups of users. And you can even decide, if the database gets overloaded, to cut off access to one of those ports, so that the application causing the problem becomes inaccessible but the rest of the database ceases being overloaded and you can still operate. Of course, you could also do that by changing pg_hba.conf, but for some people it might be more convenient (or feel more bullet-proof) to do it using network management tools. There are probably other use cases, as well. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Attachment
On Wed, Jun 6, 2012 at 1:55 PM, Robert Haas <robertmhaas@gmail.com> wrote:
+1 for multiple TCP port numbers.
A few days ago I started working on enabling Postgres to communicate using WebSockets protocol (acting as a wrapper around FE/BE), and I found it difficult (not impossible) to use the same port for communicating FE/BE protocol and for https+WebSockets too. It would have been a lot simpler if I could say that WebSockets is enabled on 5431 and FE/BE on 5432.
On Wed, Jun 6, 2012 at 10:38 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:I recently had an urge to make it possible for the postmaster to
> Well, that's what I wanted to discuss before Honza starts coding.
> It's not obvious that there are any use-cases for more than two.
> It's also not clear whether there is any value in supporting run-time
> rather than build-time configuration of the socket locations. The
> Fedora use-case has no need of that, but if people can point to other
> cases where it would be sensible, we can write the patch that way.
>
> You might think we should design this exactly like the TCP-socket
> multiple-listen-addresses case, ie just have a config variable
> containing a list of directory names. The sticking point there is
> that the directories aren't really interchangeable. In particular,
> there is still going to be one directory that is the one hard-wired
> into libpq. So whereas multiple TCP sockets really are pretty much
> interchangeable, I think in the Unix-socket case we are going to have
> to think of it as being a primary socket and one or more alternate
> sockets. Is there a reason to have more than one alternate, and if
> so what is the use-case?
>
> (BTW, we would probably just adopt the Debian solution if we were
> sure there were no non-libpq clients out there; but we aren't.)
listen on multiple ports and even went so far as to code up a patch to
allow that. It still applies, with offsets, so I'll attach it here.
So I guess I'm +1 on the idea of allowing N UNIX sockets rather than
limiting it to N=2, and really I'd like to do one better and allow
listening on multiple TCP ports as well. Since the PID file contains
the port number, multiple TCP sockets stop being interchangeable as
soon as you allow multiple ports, but that's not very difficult to
handle. Now, you might ask whether this has any real-world value, and
obviously I'm going to say yes or I wouldn't be proposing it. The
reason for wanting multiple UNIX sockets is because those sockets
might be in different places that are not all equally accessible to
everyone, because of things like chroot. But of course the same thing
is possible in the network space using iptables and similar tools.
For example, you might want to have users connect to application A
using port 5432, and to application B using port 15432. Now you can
use network monitoring tools to see how much data each application is
sending and receiving, without needing deep packet inspection. You
can firewall those ports differently to provide access to different
groups of users. And you can even decide, if the database gets
overloaded, to cut off access to one of those ports, so that the
application causing the problem becomes inaccessible but the rest of
the database ceases being overloaded and you can still operate. Of
course, you could also do that by changing pg_hba.conf, but for some
people it might be more convenient (or feel more bullet-proof) to do
it using network management tools. There are probably other use
cases, as well.
+1 for multiple TCP port numbers.
A few days ago I started working on enabling Postgres to communicate using WebSockets protocol (acting as a wrapper around FE/BE), and I found it difficult (not impossible) to use the same port for communicating FE/BE protocol and for https+WebSockets too. It would have been a lot simpler if I could say that WebSockets is enabled on 5431 and FE/BE on 5432.
Regards,
--
Gurjeet Singh
EnterpriseDB Corporation
The Enterprise PostgreSQL Company
EnterpriseDB Corporation
The Enterprise PostgreSQL Company
On 06/06/2012 04:50 PM, Andres Freund wrote: > On Wednesday, June 06, 2012 04:38:42 PM Tom Lane wrote: >> You might think we should design this exactly like the TCP-socket >> multiple-listen-addresses case, ie just have a config variable >> containing a list of directory names. The sticking point there is >> that the directories aren't really interchangeable. In particular, >> there is still going to be one directory that is the one hard-wired >> into libpq. > I wonder if the whole issue doesn't require libpq to also try multiple > hardcoded socket locations. I guess so. Let's say we add additional socket support and some server uses one e.g. at /var/run/postgresql. Then a client can either (1) specify the same path explicitly and then we don't need to specify any additional sockets on the client side or (2) stick to the default path, which is hard-coded, currently to /tmp. Going back to the original problem (inaccessible /tmp directory), it is the case (2) -- a client uses the default path. So any additional client-side socket option won't probably help here, but we would probably need a second hard-coded path e.g. at /var/run/postgresql. Regards, Honza
Honza Horak <hhorak@redhat.com> writes: > On 06/06/2012 04:50 PM, Andres Freund wrote: >> I wonder if the whole issue doesn't require libpq to also try multiple >> hardcoded socket locations. > I guess so. I don't really want to go there. Some use cases have been shown in this thread for having a server listen in multiple places, but that does not translate to saying that clients need to support automatically looking in multiple places. I think that mainly introduces questions we could do without, like which server did you actually end up contacting. regards, tom lane
On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: > Honza Horak <hhorak@redhat.com> writes: > > On 06/06/2012 04:50 PM, Andres Freund wrote: > >> I wonder if the whole issue doesn't require libpq to also try multiple > >> hardcoded socket locations. > > > > I guess so. > > I don't really want to go there. Some use cases have been shown in > this thread for having a server listen in multiple places, but that does > not translate to saying that clients need to support automatically > looking in multiple places. I think that mainly introduces questions we > could do without, like which server did you actually end up contacting. It would be really nice to have a development psql connect to a distro installed psql and vice versa without having to specify -h /var/run/psql and - h /tmp all the time... Andres -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Thu, Jun 7, 2012 at 11:57 AM, Andres Freund <andres@2ndquadrant.com> wrote: > On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: >> Honza Horak <hhorak@redhat.com> writes: >> > On 06/06/2012 04:50 PM, Andres Freund wrote: >> >> I wonder if the whole issue doesn't require libpq to also try multiple >> >> hardcoded socket locations. >> > >> > I guess so. >> >> I don't really want to go there. Some use cases have been shown in >> this thread for having a server listen in multiple places, but that does >> not translate to saying that clients need to support automatically >> looking in multiple places. I think that mainly introduces questions we >> could do without, like which server did you actually end up contacting. > It would be really nice to have a development psql connect to a distro > installed psql and vice versa without having to specify -h /var/run/psql and - > h /tmp all the time... This is true, but you have this problem already. It might be worth fixing, but it seems like a separate issue from the topic of this thread, which is where the server listens. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Andres Freund <andres@2ndquadrant.com> writes: > On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: >> Honza Horak <hhorak@redhat.com> writes: > On 06/06/2012 04:50 PM, Andres Freund wrote: >>> I wonder if the whole issue doesn't require libpq to also try multiple >>> hardcoded socket locations. >> I don't really want to go there. > It would be really nice to have a development psql connect to a distro > installed psql and vice versa without having to specify -h /var/run/psql and - > h /tmp all the time... I don't find that "nice" at all. Which server did you actually connect to? How do you control it? You're going to end up needing the -h switch anyway. regards, tom lane
On Thursday, June 07, 2012 06:20:32 PM Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: > >> Honza Horak <hhorak@redhat.com> writes: > > On 06/06/2012 04:50 PM, Andres Freund wrote: > >>> I wonder if the whole issue doesn't require libpq to also try multiple > >>> hardcoded socket locations. > >> > >> I don't really want to go there. > > > > It would be really nice to have a development psql connect to a distro > > installed psql and vice versa without having to specify -h /var/run/psql > > and - h /tmp all the time... > > I don't find that "nice" at all. Which server did you actually connect > to? How do you control it? You're going to end up needing the -h > switch anyway. They can't run on the same port anyway unless you disable listening on localhost. Changing a single port number is far less effort than typing -h /var/run/postgresql ;) Anyway, I am not wed to this, and I don't plan to put work into it so I better shut up ;) Andres -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On ons, 2012-06-06 at 11:21 -0400, Tom Lane wrote: > I'm not really in favor of making /var/run be the out-of-the-box > default, because it would discriminate against personal/testing > installations (ie, you couldn't set it up without root privileges). > It's a reasonable choice for distro-provided packages, but not so > much for one-off builds --- think about the buildfarm if nothing else. > > Having said that, if we made it easier to configure things that way > than by patching the source, I bet Martin Pitt isn't going to object. > We already have the ability to configure the Unix socket directory in postgresql.conf. All you need to do is turn that into a list. >
On Saturday, June 09, 2012 11:43:53 PM Peter Eisentraut wrote: > On ons, 2012-06-06 at 11:21 -0400, Tom Lane wrote: > > I'm not really in favor of making /var/run be the out-of-the-box > > default, because it would discriminate against personal/testing > > installations (ie, you couldn't set it up without root privileges). > > It's a reasonable choice for distro-provided packages, but not so > > much for one-off builds --- think about the buildfarm if nothing else. > > > > Having said that, if we made it easier to configure things that way > > than by patching the source, I bet Martin Pitt isn't going to object. > > We already have the ability to configure the Unix socket directory in > postgresql.conf. All you need to do is turn that into a list. That doesn't help libpq using clients. Andres -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Peter Eisentraut <peter_e@gmx.net> writes: > We already have the ability to configure the Unix socket directory in > postgresql.conf. All you need to do is turn that into a list. Um, no, it's not quite that simple. In particular, what do you think should happen on the client side? I'm inclined to think that we should (continue to) have a hardwired "primary" directory, which is the one that libpq is also configured to look in by default. But we could add a run-time-configured list of secondary directories to establish sockets in. regards, tom lane
On lör, 2012-06-09 at 23:45 +0200, Andres Freund wrote: > On Saturday, June 09, 2012 11:43:53 PM Peter Eisentraut wrote: > > On ons, 2012-06-06 at 11:21 -0400, Tom Lane wrote: > > > I'm not really in favor of making /var/run be the out-of-the-box > > > default, because it would discriminate against personal/testing > > > installations (ie, you couldn't set it up without root privileges). > > > It's a reasonable choice for distro-provided packages, but not so > > > much for one-off builds --- think about the buildfarm if nothing else. > > > > > > Having said that, if we made it easier to configure things that way > > > than by patching the source, I bet Martin Pitt isn't going to object. > > > > We already have the ability to configure the Unix socket directory in > > postgresql.conf. All you need to do is turn that into a list. > That doesn't help libpq using clients. There is no mandate here to do anything about that.
On lör, 2012-06-09 at 18:02 -0400, Tom Lane wrote: > Peter Eisentraut <peter_e@gmx.net> writes: > > We already have the ability to configure the Unix socket directory in > > postgresql.conf. All you need to do is turn that into a list. > > Um, no, it's not quite that simple. In particular, what do you think > should happen on the client side? Nothing changes there. > I'm inclined to think that we should (continue to) have a hardwired > "primary" directory, which is the one that libpq is also configured > to look in by default. Yes. > But we could add a run-time-configured list of secondary directories to establish sockets in. Yes, I'm just pointing out that we already have that list (unix_socket_directory in postgresql.conf), except it's currently limited to length 1, because no one has needed a longer one until now.
On Sunday, June 10, 2012 12:09:30 AM Peter Eisentraut wrote: > On lör, 2012-06-09 at 23:45 +0200, Andres Freund wrote: > > On Saturday, June 09, 2012 11:43:53 PM Peter Eisentraut wrote: > > > On ons, 2012-06-06 at 11:21 -0400, Tom Lane wrote: > > > > I'm not really in favor of making /var/run be the out-of-the-box > > > > default, because it would discriminate against personal/testing > > > > installations (ie, you couldn't set it up without root privileges). > > > > It's a reasonable choice for distro-provided packages, but not so > > > > much for one-off builds --- think about the buildfarm if nothing > > > > else. > > > > > > > > Having said that, if we made it easier to configure things that way > > > > than by patching the source, I bet Martin Pitt isn't going to object. > > > > > > We already have the ability to configure the Unix socket directory in > > > postgresql.conf. All you need to do is turn that into a list. > > That doesn't help libpq using clients. > There is no mandate here to do anything about that. Well, Martin Pitt/the debian package is patching exactly that. Youre saying that everything that needs to be done to make that easier is to make unix_socket_dir a list. So there seems to be some disparity there ;) Andres -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Peter Eisentraut <peter_e@gmx.net> writes: > On lör, 2012-06-09 at 18:02 -0400, Tom Lane wrote: >> I'm inclined to think that we should (continue to) have a hardwired >> "primary" directory, which is the one that libpq is also configured >> to look in by default. > Yes. >> But we could add a run-time-configured list of secondary directories to establish sockets in. > Yes, I'm just pointing out that we already have that list > (unix_socket_directory in postgresql.conf), except it's currently > limited to length 1, because no one has needed a longer one until now. That's not actually quite the same thing as what I suggest above. Currently, unix_socket_directory *overrides* the compiled-in choice. I'm suggesting that it would be better to invent a list that is *added to* the compiled-in choice. If we think it would be best to still be able to override that, then I'd vote for keeping unix_socket_directory as is, and then adding a list named something like "secondary_socket_directories". But if we just turn unix_socket_directory into a list, I think the lack of separation between primary and secondary directories will be confusing. Or maybe I'm wrong and it's better doing it as you suggest, but I think this needs consideration. regards, tom lane
On sön, 2012-06-10 at 00:25 +0200, Andres Freund wrote: > > > > We already have the ability to configure the Unix socket > directory in > > > > postgresql.conf. All you need to do is turn that into a list. > > > That doesn't help libpq using clients. > > There is no mandate here to do anything about that. > Well, Martin Pitt/the debian package is patching exactly that. Youre > saying > that everything that needs to be done to make that easier is to make > unix_socket_dir a list. So there seems to be some disparity there ;) > The Debian package doesn't need any change or assistance, really. You can change the default location of the socket by patching pg_config_manual.h, and that's a one-liner. The only thing that would make that slightly easier or better is making it a configure option, but that was intentionally decided against in the old days (not by me).
On lör, 2012-06-09 at 18:26 -0400, Tom Lane wrote: > > Yes, I'm just pointing out that we already have that list > > (unix_socket_directory in postgresql.conf), except it's currently > > limited to length 1, because no one has needed a longer one until > now. > > That's not actually quite the same thing as what I suggest above. > Currently, unix_socket_directory *overrides* the compiled-in choice. > I'm suggesting that it would be better to invent a list that is *added > to* the compiled-in choice. If we think it would be best to still be > able to override that, then I'd vote for keeping unix_socket_directory > as is, and then adding a list named something like > "secondary_socket_directories". But if we just turn > unix_socket_directory into a list, I think the lack of separation > between primary and secondary directories will be confusing. By that logic, any list-valued parameter should be split into a primary and secondary setting. That could actually be moderately useful in some cases (think search_path, or if we get there, multiple port settings), but then we should put this into the grammar or processing logic of postgresql.conf, not invent a bunch of new settings. (E.g., unix_socket_directory += ...).
Peter Eisentraut <peter_e@gmx.net> writes: > On lör, 2012-06-09 at 18:26 -0400, Tom Lane wrote: >> That's not actually quite the same thing as what I suggest above. >> Currently, unix_socket_directory *overrides* the compiled-in choice. >> I'm suggesting that it would be better to invent a list that is *added >> to* the compiled-in choice. If we think it would be best to still be >> able to override that, then I'd vote for keeping unix_socket_directory >> as is, and then adding a list named something like >> "secondary_socket_directories". But if we just turn >> unix_socket_directory into a list, I think the lack of separation >> between primary and secondary directories will be confusing. > By that logic, any list-valued parameter should be split into a primary > and secondary setting. Well, no: the key point here is that there will be one directory that is special because it's the one baked into libpq. I agree that for the purposes of the backend in isolation, we might as well just have a list. What's less clear is whether, when considering the backend+client ecosystem as a whole, the special status of the configure-time socket directory ought to be reflected in the way we set up the GUCs. I have to admit that I'm not totally sold on either approach. regards, tom lane
On Sun, Jun 10, 2012 at 8:36 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Peter Eisentraut <peter_e@gmx.net> writes: >> On lör, 2012-06-09 at 18:26 -0400, Tom Lane wrote: >>> That's not actually quite the same thing as what I suggest above. >>> Currently, unix_socket_directory *overrides* the compiled-in choice. >>> I'm suggesting that it would be better to invent a list that is *added >>> to* the compiled-in choice. If we think it would be best to still be >>> able to override that, then I'd vote for keeping unix_socket_directory >>> as is, and then adding a list named something like >>> "secondary_socket_directories". But if we just turn >>> unix_socket_directory into a list, I think the lack of separation >>> between primary and secondary directories will be confusing. > >> By that logic, any list-valued parameter should be split into a primary >> and secondary setting. > > Well, no: the key point here is that there will be one directory that is > special because it's the one baked into libpq. I agree that for the > purposes of the backend in isolation, we might as well just have a list. > What's less clear is whether, when considering the backend+client > ecosystem as a whole, the special status of the configure-time socket > directory ought to be reflected in the way we set up the GUCs. I have > to admit that I'm not totally sold on either approach. I think we should consider this in the context of allowing both additional UNIX sockets and additional TCP ports. In the case of TCP ports, it's clearly no good to turn "port" into a list, because one port number has to be primary, since it goes into the PID file and also affects the naming of any UNIX sockets created. If we add secondary_socket_dirs, I think we will also need secondary_ports. One idea might be to have one new GUC that serves both purposes: additional_sockets = '12345, /foo' I'm sure there are other ways to skin the cat as well. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On sön, 2012-06-10 at 09:41 -0400, Robert Haas wrote: > I think we should consider this in the context of allowing both > additional UNIX sockets and additional TCP ports. In the case of TCP > ports, it's clearly no good to turn "port" into a list, because one > port number has to be primary, since it goes into the PID file Not necessarily. The port number in the PID file is only used for pg_ctl to contact the server, and for that it only needs some port, not the primary one. Also, we write the first listen_address into the PID file for the same reason. So if you think there should be a primary port, then there should also be a primary listen_addresses. > and also affects the naming of any UNIX sockets created. Why would that matter? If you configure M ports and N Unix socket locations, you get M*N actual sockets created. > If we add > secondary_socket_dirs, I think we will also need secondary_ports. One > idea might be to have one new GUC that serves both purposes: > > additional_sockets = '12345, /foo' I was getting around to that, although I don't follow the specific syntax you have chosen here. I would like something where you set host and port together, so you can listen on port 5432 on localhost, and port 5433 on *, for example. So maybe listen_addresses = localhost:5432,*:5433 Web servers usually allow that sort of thing, but if you dig deep there, the configuration settings and their interactions can get pretty complicated. For example, you can usually set a default port and then override it in the listen_addresses equivalent.
On Sun, Jun 10, 2012 at 5:06 PM, Peter Eisentraut <peter_e@gmx.net> wrote: > On sön, 2012-06-10 at 09:41 -0400, Robert Haas wrote: >> I think we should consider this in the context of allowing both >> additional UNIX sockets and additional TCP ports. In the case of TCP >> ports, it's clearly no good to turn "port" into a list, because one >> port number has to be primary, since it goes into the PID file > > Not necessarily. The port number in the PID file is only used for > pg_ctl to contact the server, and for that it only needs some port, not > the primary one. Also, we write the first listen_address into the PID > file for the same reason. So if you think there should be a primary > port, then there should also be a primary listen_addresses. Fair enough, as far as this part goes, but... >> and also affects the naming of any UNIX sockets created. > > Why would that matter? If you configure M ports and N Unix socket > locations, you get M*N actual sockets created. ...I *seriously* doubt that this is the behavior anyone wants. Creating M sockets per directory seems patently silly. >> If we add >> secondary_socket_dirs, I think we will also need secondary_ports. One >> idea might be to have one new GUC that serves both purposes: >> >> additional_sockets = '12345, /foo' > > I was getting around to that, although I don't follow the specific > syntax you have chosen here. I was thinking that each element could be of the form /path or port. But I guess it should really be /path or host:port. > I would like something where you set host and port together, so you can > listen on port 5432 on localhost, and port 5433 on *, for example. So > maybe > > listen_addresses = localhost:5432,*:5433 > > Web servers usually allow that sort of thing, but if you dig deep there, > the configuration settings and their interactions can get pretty > complicated. For example, you can usually set a default port and then > override it in the listen_addresses equivalent. That seems like the obvious syntax, but I'm fuzzy on the details. We could let 'port' continue to mean the default port, and then listen_addresses can contain either unadorned addresses (in which case we bind to that address using the default port) or address:port designators (in which case we bind to the given address and port). But then support port = 1234 and listen_addresses = '5.5.5.5:6789'. Presumably the UNIX socket is still /tmp/.s.PGSQL.1234, but then what ends up in the lock file? PostmasterMain's current algorithm for figuring that out would write 5.5.5.5 for the host and 1234 for the port, which seems like nonsense. Such confusion is why I thought we might fall back on listing all the additional listen locations in a new, separate GUC. But perhaps there is a way to make it work. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes: > On Sun, Jun 10, 2012 at 5:06 PM, Peter Eisentraut <peter_e@gmx.net> wrote: >> On s�n, 2012-06-10 at 09:41 -0400, Robert Haas wrote: >>> If we add >>> secondary_socket_dirs, I think we will also need secondary_ports. �One >>> idea might be to have one new GUC that serves both purposes: >>> >>> additional_sockets = '12345, /foo' >> I was getting around to that, although I don't follow the specific >> syntax you have chosen here. > I was thinking that each element could be of the form /path or port. > But I guess it should really be /path or host:port. I'm uncomfortable with the potential for ambiguity there. I think we'd be much better off having two lists, one for TCP addresses and one for Unix socket directories. I'm unconvinced that allowing multiple port numbers is worth the amount of confusion it will cause. In particular, we've traditionally used "the port number" as part of the key for resources such as shared memory. I think we'd want the number used for that purpose to be what is written into the lock file ... but then what if the postmaster is not actually listening on *any* actual socket with that number? pg_ctl will not be happy. regards, tom lane
On Sun, Jun 10, 2012 at 5:35 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Sun, Jun 10, 2012 at 5:06 PM, Peter Eisentraut <peter_e@gmx.net> wrote: >>> On sön, 2012-06-10 at 09:41 -0400, Robert Haas wrote: >>>> If we add >>>> secondary_socket_dirs, I think we will also need secondary_ports. One >>>> idea might be to have one new GUC that serves both purposes: >>>> >>>> additional_sockets = '12345, /foo' > >>> I was getting around to that, although I don't follow the specific >>> syntax you have chosen here. > >> I was thinking that each element could be of the form /path or port. >> But I guess it should really be /path or host:port. > > I'm uncomfortable with the potential for ambiguity there. I think we'd > be much better off having two lists, one for TCP addresses and one for > Unix socket directories. I suggested it mostly because we already use that convention in libpq: leading slash = pathname. > I'm unconvinced that allowing multiple port numbers is worth the > amount of confusion it will cause. In particular, we've traditionally > used "the port number" as part of the key for resources such as shared > memory. I think we'd want the number used for that purpose to be what > is written into the lock file ... but then what if the postmaster is not > actually listening on *any* actual socket with that number? pg_ctl will > not be happy. Well, that's why I shied away from the syntax Peter is proposing. I think if we leave the semantics of listen_addresses and port alone, and just add a new GUC for extra places to listen, there's no problem.If you look at the patch I posted upthread, you'llsee that I set things up so that we'll still fail if the primary port can't be listened on; once we've established that we can listen there, we'll try to set up the other sockets as well. I think that's a pretty sane way to allow this (which a number of people, not only me, seem to support) without creating surprising semantics. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On sön, 2012-06-10 at 17:24 -0400, Robert Haas wrote: > >> and also affects the naming of any UNIX sockets created. > > > > Why would that matter? If you configure M ports and N Unix socket > > locations, you get M*N actual sockets created. > > ...I *seriously* doubt that this is the behavior anyone wants. > Creating M sockets per directory seems patently silly. How else would it work? If I say, syntax aside, listen on "ports" 5432 and 5433, and use socket directories /tmp and /var/run/postgresql, then a libpq-using client would expect to be able to connect using -h /tmp -p 5432 -h /tmp -p 5433 -h /var/run/postgresql -p 5432 -h /var/run/postgresql -p 5433 So you do need to create M*N sockets. I don't really see a problem with that.
On sön, 2012-06-10 at 17:35 -0400, Tom Lane wrote: > I'm unconvinced that allowing multiple port numbers is worth the > amount of confusion it will cause. Well, it's a feature that people have asked for. I would love to have it. Much more than multiple Unix-domain socket locations. > In particular, we've traditionally > used "the port number" as part of the key for resources such as shared > memory. But it hasn't been a requirement for a long time that those match up exactly. It's already possible that they don't, if you configure postmasters with the same port and non-conflicting IP addresses or Unix-socket locations. > I think we'd want the number used for that purpose to be what > is written into the lock file ... but then what if the postmaster is > not actually listening on *any* actual socket with that number? > pg_ctl will not be happy. > I'm not sure why pg_ctl needs to know about the shared memory business. We write the shared memory key into the lock file, so the port number in the lock file should just be a port number for pg_ctl to use. Of course you can configure things so that pg_ctl cannot contact the postmaster, but this problem already exists in a more likely fashion with listen_addresses. Adding an extra port doesn't make it more likely.
Peter Eisentraut <peter_e@gmx.net> writes: > On sön, 2012-06-10 at 17:24 -0400, Robert Haas wrote: >>> Why would that matter? If you configure M ports and N Unix socket >>> locations, you get M*N actual sockets created. >> ...I *seriously* doubt that this is the behavior anyone wants. >> Creating M sockets per directory seems patently silly. > How else would it work? > If I say, syntax aside, listen on "ports" 5432 and 5433, and use socket > directories /tmp and /var/run/postgresql, then a libpq-using client > would expect to be able to connect using This argument seems quite circular to me: you are assuming that we will adopt exactly the behavior that Robert is questioning. What would make more sense to me is (1) there is still a *single* "port" parameter, which is what we use for things like shared memory keys; (2) listen_addresses (and the hypothetical socket_directories list) grows the ability to specify a port number in any list element. The primary port number parameter sets the default. So for instance port = 5432listen_addresses = '*, 127.0.0.1:5433' results in listening on *:5432 and 127.0.0.1:5433. > So you do need to create M*N sockets. > I don't really see a problem with that. I do: first, it's a lotta sockets, and second, it's not real hard to foresee cases where somebody actively doesn't want that cross-product. regards, tom lane
On Mon, Jun 11, 2012 at 4:47 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
What about entries in pg_hba.conf? Will they need to be able to specify both the directory and the port number?
--
Mike Nolan
On sön, 2012-06-10 at 17:24 -0400, Robert Haas wrote:How else would it work?
> >> and also affects the naming of any UNIX sockets created.
> >
> > Why would that matter? If you configure M ports and N Unix socket
> > locations, you get M*N actual sockets created.
>
> ...I *seriously* doubt that this is the behavior anyone wants.
> Creating M sockets per directory seems patently silly.
If I say, syntax aside, listen on "ports" 5432 and 5433, and use socket
directories /tmp and /var/run/postgresql, then a libpq-using client
would expect to be able to connect using
-h /tmp -p 5432
-h /tmp -p 5433
-h /var/run/postgresql -p 5432
-h /var/run/postgresql -p 5433
So you do need to create M*N sockets.
I don't really see a problem with that.
What about entries in pg_hba.conf? Will they need to be able to specify both the directory and the port number?
--
Mike Nolan
On 06/11/2012 11:47 PM, Peter Eisentraut wrote: > On sön, 2012-06-10 at 17:24 -0400, Robert Haas wrote: >>>> and also affects the naming of any UNIX sockets created. >>> >>> Why would that matter? If you configure M ports and N Unix socket >>> locations, you get M*N actual sockets created. >> >> ...I *seriously* doubt that this is the behavior anyone wants. >> Creating M sockets per directory seems patently silly. > > How else would it work? > > If I say, syntax aside, listen on "ports" 5432 and 5433, and use socket > directories /tmp and /var/run/postgresql, then a libpq-using client > would expect to be able to connect using > > -h /tmp -p 5432 > -h /tmp -p 5433 > -h /var/run/postgresql -p 5432 > -h /var/run/postgresql -p 5433 This could be true in case all listening ports are equal, which I guess isn't a good idea, because IIUIC the port number as a part of the socket name is used for distinguish sockets of various postmasters in the same directory. In that scenario every client should know which port to connect and also which one is primary. Regards, Honza
On Wed, Jun 06, 2012 at 10:38:42AM -0400, Tom Lane wrote: > Florian Pflug <fgp@phlo.org> writes: > > Couldn't you simply tell postgres to put it's socket in, say, /var/run, and create a symlink to that socket in the global/tmp directory? > > FYI, this proposal emerged out of a discussion between Honza and > myself. "Use a symlink" was my first idea too, but on reflection > it seems like it will take less new code to support two sockets. > We already support multiple TCP sockets, so multiple Unix sockets --------------------------------------- > shouldn't be that much extra trouble. We do? I didn't think listening on multiple interfaces meant we listened on multiple sockets. Is there something else? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Bruce Momjian <bruce@momjian.us> writes: > On Wed, Jun 06, 2012 at 10:38:42AM -0400, Tom Lane wrote: >> We already support multiple TCP sockets, so multiple Unix sockets >> shouldn't be that much extra trouble. > We do? I didn't think listening on multiple interfaces meant we > listened on multiple sockets. Is there something else? There's one socket for each entry in the listen_addresses list, plus one for the Unix socket. regards, tom lane
On Tue, Jun 12, 2012 at 05:48:58PM -0400, Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > On Wed, Jun 06, 2012 at 10:38:42AM -0400, Tom Lane wrote: > >> We already support multiple TCP sockets, so multiple Unix sockets > >> shouldn't be that much extra trouble. > > > We do? I didn't think listening on multiple interfaces meant we > > listened on multiple sockets. Is there something else? > > There's one socket for each entry in the listen_addresses list, > plus one for the Unix socket. Oh, how do we handle '*'? We pass that to the kernel, I assume. Shame there is "wildcard" ability for unix domain sockets, which would use any directory --- guess that wouldn't work out well. ;-) -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 06/10/2012 12:37 AM, Peter Eisentraut wrote: > On sön, 2012-06-10 at 00:25 +0200, Andres Freund wrote: >>>>> We already have the ability to configure the Unix socket >> directory in >>>>> postgresql.conf. All you need to do is turn that into a list. >>>> That doesn't help libpq using clients. >>> There is no mandate here to do anything about that. >> Well, Martin Pitt/the debian package is patching exactly that. Youre >> saying >> that everything that needs to be done to make that easier is to make >> unix_socket_dir a list. So there seems to be some disparity there ;) >> > The Debian package doesn't need any change or assistance, really. You > can change the default location of the socket by patching > pg_config_manual.h, and that's a one-liner. The only thing that would > make that slightly easier or better is making it a configure option, but > that was intentionally decided against in the old days (not by me). Since systemd with it's PrivateTmp feature is going to be used in more and more distros, there will probably be a bigger need to solve in-accessible default unix socket directory /tmp in the future. Thus, I'd like to ask if anybody is aware of any issue connected with simply patching pg_config_manual.h, same as Debian does it already? For example, is there any piece of software, that simply rely on /tmp location of the socket and doesn't use libpg for the communication? Regards, Honza
On 06/10/2012 03:41 PM, Robert Haas wrote: > On Sun, Jun 10, 2012 at 8:36 AM, Tom Lane<tgl@sss.pgh.pa.us> wrote: >> Peter Eisentraut<peter_e@gmx.net> writes: >>> On lör, 2012-06-09 at 18:26 -0400, Tom Lane wrote: >>>> That's not actually quite the same thing as what I suggest above. >>>> Currently, unix_socket_directory *overrides* the compiled-in choice. >>>> I'm suggesting that it would be better to invent a list that is *added >>>> to* the compiled-in choice. If we think it would be best to still be >>>> able to override that, then I'd vote for keeping unix_socket_directory >>>> as is, and then adding a list named something like >>>> "secondary_socket_directories". But if we just turn >>>> unix_socket_directory into a list, I think the lack of separation >>>> between primary and secondary directories will be confusing. >> >>> By that logic, any list-valued parameter should be split into a primary >>> and secondary setting. >> >> Well, no: the key point here is that there will be one directory that is >> special because it's the one baked into libpq. I agree that for the >> purposes of the backend in isolation, we might as well just have a list. >> What's less clear is whether, when considering the backend+client >> ecosystem as a whole, the special status of the configure-time socket >> directory ought to be reflected in the way we set up the GUCs. I have >> to admit that I'm not totally sold on either approach. > > I think we should consider this in the context of allowing both > additional UNIX sockets and additional TCP ports. In the case of TCP > ports, it's clearly no good to turn "port" into a list, because one > port number has to be primary, since it goes into the PID file and > also affects the naming of any UNIX sockets created. If we add > secondary_socket_dirs, I think we will also need secondary_ports. One > idea might be to have one new GUC that serves both purposes: > > additional_sockets = '12345, /foo' > > I'm sure there are other ways to skin the cat as well. Going through the thread, I'd like to sum it up choosing approach with less potential issues and would like to find a consensus if possible. It seems unix_socket_directory could be turned into list and probably renamed to unix_socket_directories, since it would be confusing if a list value is in singular. On the other hand, we probably don't want to specify listening ports together with additional unix sockets in one configuration option, so it seems better to add a new configuration option to distinguish the primary listening port from additional ports. Regards, Honza
On Jun13, 2012, at 15:14 , Honza Horak wrote: > Since systemd with it's PrivateTmp feature is going to be used in more and more distros, there will probably be a biggerneed to solve in-accessible default unix socket directory /tmp in the future. > > Thus, I'd like to ask if anybody is aware of any issue connected with simply patching pg_config_manual.h, same as Debiandoes it already? For example, is there any piece of software, that simply rely on /tmp location of the socket and doesn'tuse libpg for the communication? I've used self-compiled postgres version on debian for years, and debian's way of doing things is major PITA in that situation.You end up having to specify that full path to the socket directory *everywhere*, because otherwise things startto break if you recompile something and it suddenly happens to use the debian-supplied libpq instead of your own. Supporting sockets in multiple directories would solve that, once and for all. best regards, Florian Pflug
On ons, 2012-06-13 at 15:14 +0200, Honza Horak wrote: > Thus, I'd like to ask if anybody is aware of any issue connected with > simply patching pg_config_manual.h, same as Debian does it already? > For example, is there any piece of software, that simply rely on /tmp > location of the socket and doesn't use libpg for the communication? If you're asking, should Red Hat/Fedora simply fix the issue locally by patching pg_config_manual.h, then yes, that would work, see Debian, but it has its annoyances, especially with additional software compiled from source, or third-party binary installers.
On mån, 2012-06-11 at 18:07 -0400, Tom Lane wrote: > Peter Eisentraut <peter_e@gmx.net> writes: > > On sön, 2012-06-10 at 17:24 -0400, Robert Haas wrote: > >>> Why would that matter? If you configure M ports and N Unix socket > >>> locations, you get M*N actual sockets created. > > >> ...I *seriously* doubt that this is the behavior anyone wants. > >> Creating M sockets per directory seems patently silly. > > > How else would it work? > > > If I say, syntax aside, listen on "ports" 5432 and 5433, and use socket > > directories /tmp and /var/run/postgresql, then a libpq-using client > > would expect to be able to connect using > > This argument seems quite circular to me: you are assuming that we will > adopt exactly the behavior that Robert is questioning. > > What would make more sense to me is > > (1) there is still a *single* "port" parameter, which is what we use for > things like shared memory keys; > > (2) listen_addresses (and the hypothetical socket_directories list) > grows the ability to specify a port number in any list element. The > primary port number parameter sets the default. > > So for instance > > port = 5432 > listen_addresses = '*, 127.0.0.1:5433' > > results in listening on *:5432 and 127.0.0.1:5433. > > > So you do need to create M*N sockets. > > I don't really see a problem with that. > > I do: first, it's a lotta sockets, and second, it's not real hard to > foresee cases where somebody actively doesn't want that cross-product. Well, it's fine if we provide ways not to have the cross-product, but there should also be an easy way to get it. I can easily see cases in systems I have administered where I would have liked to use two unix sockets, two IP sockets, and two ports. Maybe I actually would have needed only 7 out of those 8 sockets, but it's far easier to configure, document, and explain if I just set up all 8 of them.
On mån, 2012-06-11 at 18:45 -0500, Michael Nolan wrote: > What about entries in pg_hba.conf? Will they need to be able to specify > both the directory and the port number? I think the philosophy behind pg_hba.conf is that you distinguish client *systems*. So one client might be Kerberos-capable, or one client might be Windows and use SSPI. For that, it doesn't matter which of multiple ports or local sockets it uses. So I don't think there are any changes needed in this area.
On tis, 2012-06-12 at 14:47 +0200, Honza Horak wrote: > This could be true in case all listening ports are equal, which I > guess isn't a good idea, because IIUIC the port number as a part of > the socket name is used for distinguish sockets of various postmasters > in the same directory. In that scenario every client should know which > port to connect and also which one is primary. Why? The client won't care which port is primary or secondary or whatever as long as it reaches the server.
On ons, 2012-06-13 at 15:25 +0200, Honza Horak wrote: > It seems unix_socket_directory could be turned into list and probably > renamed to unix_socket_directories, since it would be confusing if a > list value is in singular. Well, it would also be annoying to rename the parameter name for a marginal change in functionality. > On the other hand, we probably don't want to specify listening ports > together with additional unix sockets in one > configuration option, so it seems better to add a new configuration > option to distinguish the primary listening port from additional > ports. I think it would be wiser if you left the port business out of this. There are more issues hidden in there than you need to deal with.
On Thu, Jun 14, 2012 at 01:31:31AM +0300, Peter Eisentraut wrote: > On ons, 2012-06-13 at 15:25 +0200, Honza Horak wrote: > > It seems unix_socket_directory could be turned into list and probably > > renamed to unix_socket_directories, since it would be confusing if a > > list value is in singular. > > Well, it would also be annoying to rename the parameter name for a > marginal change in functionality. We often rename admin-only config variables for clarity, and this seems to be such a case. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Peter Eisentraut <peter_e@gmx.net> writes: > On mån, 2012-06-11 at 18:07 -0400, Tom Lane wrote: >> Peter Eisentraut <peter_e@gmx.net> writes: >>> So you do need to create M*N sockets. >>> I don't really see a problem with that. >> I do: first, it's a lotta sockets, and second, it's not real hard to >> foresee cases where somebody actively doesn't want that cross-product. > Well, it's fine if we provide ways not to have the cross-product, but > there should also be an easy way to get it. I can easily see cases in > systems I have administered where I would have liked to use two unix > sockets, two IP sockets, and two ports. Maybe I actually would have > needed only 7 out of those 8 sockets, but it's far easier to configure, > document, and explain if I just set up all 8 of them. Allow me to doubt that people are going to need cross-product socket sets that are so large that it's painful to enumerate all the cases. I can believe your 4x2 example, but not ones that are much bigger than that. regards, tom lane
On Thu, Jun 14, 2012 at 12:18 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Peter Eisentraut <peter_e@gmx.net> writes: >> On mån, 2012-06-11 at 18:07 -0400, Tom Lane wrote: >>> Peter Eisentraut <peter_e@gmx.net> writes: >>>> So you do need to create M*N sockets. >>>> I don't really see a problem with that. > >>> I do: first, it's a lotta sockets, and second, it's not real hard to >>> foresee cases where somebody actively doesn't want that cross-product. > >> Well, it's fine if we provide ways not to have the cross-product, but >> there should also be an easy way to get it. I can easily see cases in >> systems I have administered where I would have liked to use two unix >> sockets, two IP sockets, and two ports. Maybe I actually would have >> needed only 7 out of those 8 sockets, but it's far easier to configure, >> document, and explain if I just set up all 8 of them. > > Allow me to doubt that people are going to need cross-product socket > sets that are so large that it's painful to enumerate all the cases. > I can believe your 4x2 example, but not ones that are much bigger than > that. Same here. I can't really understand why someone would want to have, say, six socket directories with four completely interchangeable sockets in each one. At any rate I have no problem with allowing it, but I think it's marginal enough that we can sanely require that a system admin who needs that has to list out all 24 sockets explicitly.Maybe: listen_addresses = { host | host:port | * | *:port } [, ...] unix_socket_directory = { directory | directory:port ] [,...] ...except that colon is a valid character in a directory name. Not sure what to do about that. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes: > Maybe: > listen_addresses = { host | host:port | * | *:port } [, ...] > unix_socket_directory = { directory | directory:port ] [,...] > ...except that colon is a valid character in a directory name. Not > sure what to do about that. Do we need to do anything? There are no standard scenarios in which a colon would appear in such paths, and I don't see why we need to cater for it. (Remember that Windows doesn't enter into this ...) regards, tom lane
On Thu, Jun 14, 2012 at 1:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> Maybe: > >> listen_addresses = { host | host:port | * | *:port } [, ...] >> unix_socket_directory = { directory | directory:port ] [,...] > >> ...except that colon is a valid character in a directory name. Not >> sure what to do about that. > > Do we need to do anything? There are no standard scenarios in which a > colon would appear in such paths, and I don't see why we need to cater > for it. (Remember that Windows doesn't enter into this ...) True. And, we should be able to design the parsing algorithm so that they can fix it by adding an otherwise-redundant port specification - i.e. if you want to put the socket in a directory called /me:1, then write /me:1:5432 -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 06/13/2012 03:25 PM, Honza Horak wrote: > Going through the thread, I'd like to sum it up choosing approach with > less potential issues and would like to find a consensus if possible. > > It seems unix_socket_directory could be turned into list and probably > renamed to unix_socket_directories, since it would be confusing if a > list value is in singular. On the other hand, we probably don't want to > specify listening ports together with additional unix sockets in one > configuration option, so it seems better to add a new configuration > option to distinguish the primary listening port from additional ports. > > Regards, > Honza A draft patch is attached. It renames unix_socket_directory to unix_socket_directories and allows to use directory:port to be able to create more sockets in one directory with different port number in the socket name. Regards, Honza
Attachment
> On 06/13/2012 03:25 PM, Honza Horak wrote: > A draft patch is attached. It renames unix_socket_directory to > unix_socket_directories and allows to use directory:port to be able > to > create more sockets in one directory with different port number in > the > socket name. I realized the patch has some difficulties -- namely the socket path in the data dir lock file, which currently uses oneport for socket and the same for interface. So to allow users to use arbitrary port for all unix sockets, we'd need toadd another line only for unix socket, which doesn't apply for other platforms. Or we could just say that the first socketwill allways use the default port (PostPortNumber), which is a solution I prefer currently, but will be glad for anyother opinion. This is also why there is still un-necesary string splitting in pg_ctl.c, which will be removed after theissue above is solved. Regards, Honza
On 06/15/2012 05:40 PM, Honza Horak wrote: > I realized the patch has some difficulties -- namely the socket path in the data dir lock file, which currently uses oneport for socket and the same for interface. So to allow users to use arbitrary port for all unix sockets, we'd need toadd another line only for unix socket, which doesn't apply for other platforms. Or we could just say that the first socketwill allways use the default port (PostPortNumber), which is a solution I prefer currently, but will be glad for anyother opinion. This is also why there is still un-necesary string splitting in pg_ctl.c, which will be removed after theissue above is solved. > This is an enhanced patch, which forbids using a port number in the first socket directory entry. Honza
Attachment
Honza Horak <hhorak@redhat.com> writes: > On 06/15/2012 05:40 PM, Honza Horak wrote: >> I realized the patch has some difficulties -- namely the socket path in the data dir lock file, which currently uses oneport for socket and the same for interface. So to allow users to use arbitrary port for all unix sockets, we'd need toadd another line only for unix socket, which doesn't apply for other platforms. Or we could just say that the first socketwill allways use the default port (PostPortNumber), which is a solution I prefer currently, but will be glad for anyother opinion. This is also why there is still un-necesary string splitting in pg_ctl.c, which will be removed after theissue above is solved. I did a review pass over this patch. > This is an enhanced patch, which forbids using a port number in the > first socket directory entry. Well, not so much "forbids" as "silently ignores", which doesn't seem like great user-interface design to me. If we're going to adopt this solution I think we need it to throw an error instead of just ignoring the port specification. On the whole I prefer the solution you mention above: let's generalize the postmaster.pid format (and pg_ctl) so that we don't need to assume anything about port numbers matching up. The nearby discussion about allowing listen_addresses to specify port number would break this assumption anyway. If we just add two port numbers into postmaster.pid, one for the Unix socket and one for the TCP port, we could get rid of the problem entirely. I read through the patch for awhile and noticed some other smaller issues: * the patch does not compile warning-free: postgres.c: In function 'PostgresMain': postgres.c:3669:3: warning: implicit declaration of function 'SplitUnixDirectories' [-Wimplicit-function-declaration] varlena.c: In function 'SplitUnixDirectories': varlena.c:2577:3: warning: suggest parentheses around assignment used as truth value [-Wparentheses] * the changes in bootstrap.c and postgres.c won't compile at all when not HAVE_UNIX_SOCKETS, since mainSocket is referred to even though it won't be defined in such a case. * I'm not especially thrilled with propagating SplitUnixDirectories calls into those two places anyway, nor with the weird decision for SplitUnixDirectories to return a separate "mainSocket" value. Perhaps what would be most sensible is to attach an assign hook to the unix_socket_directories GUC parameter that would automatically split the string and store the components into a globally-visible List variable (which could replace the globally-visible string value we have now). Then the places that need to reference the "main socket" could just look at the first list entry if any. Also, the hook could enforce valid parameter syntax. * Also, I'm inclined to think it will work better if SplitUnixDirectories takes care of separating out the port number data, which it could return in an integer list parallel to the string list of directory names. For one thing, if you don't then the places that currently are looking at "mainSocket" are going to need to duplicate the port-splitting logic. * Also, CreateDataDirLockFile generally gets all its information about GUC settings by looking directly at the GUCs (eg, DataDir). It seems inconsistent to pass it just this one value rather than having it find the info for itself. So on the whole I'd drop the bootstrap.c and postgres.c changes altogether and do whatever's needful inside CreateLockFile, or perhaps even better, not write the Unix socket info at file creation but add it with AddToDataDirLockFile later. * It might be a good idea to s/unixSocketName/unixSocketDir/ throughout pqcomm.c, since AFAICS none of those variables are actually the full path name of the socket file. Also, I'm a bit disturbed that StreamServerPort is now being called with UnixSocketDirs in a lot of places; that's either wrong or useless. Maybe we could pass NULL in the places where it's not meant to be a sensible value? * Having Lock_AF_UNIX pass back a socket path seems rather grotty. Possibly better to move the UNIXSOCK_PATH call to its caller and just pass it a sock_path, similar to Setup_AF_UNIX? The placement of the addition to the sock_paths list seems a bit random (or at least undercommented), too. * In postmaster.c, is it really possible for UnixSocketDirs to be null? I'm inclined to think that code path is unreachable, since guc.c initializes the value to empty-string not null. But in any case, unix_socket_directories being an empty string should specify creating zero sockets, I think. We need to change the default value to be DEFAULT_PGSOCKET_DIR, or empty on Windows. (Likewise, if we're going to pass a socket file name to CreateLockFile, it should be a socket file name, not something that might need to be replaced by DEFAULT_PGSOCKET_DIR.) * Not sure about adding an is_absolute_path() insistence as you have done in postmaster.c. We never required that before. I'm also inclined to think that the canonicalize_path work should be done in SplitDirectoriesString not here. regards, tom lane
I wrote: > On the whole I prefer the solution you mention above: let's generalize > the postmaster.pid format (and pg_ctl) so that we don't need to assume > anything about port numbers matching up. The nearby discussion about > allowing listen_addresses to specify port number would break this > assumption anyway. If we just add two port numbers into postmaster.pid, > one for the Unix socket and one for the TCP port, we could get rid of > the problem entirely. After further thought, I think that this approach would make it a good idea to drop support for alternate port numbers from the present patch. Let's just deal with alternate socket directories for now. There could be a follow-on patch that adds support for nondefault port numbers in both listen_addresses and unix_socket_directories, and fixes up the postmaster.pid format to support that. I will admit that part of my desire to do it this way is a narrow Fedora rationale: in the Fedora package, we are going to want to back-patch the alternate-directory feature into 9.2 (and maybe 9.1) so as to fix our problems with systemd's PrivateTmp feature. The alternate-port-number feature is not necessary for that, and leaving it out would make for a significantly smaller back-patch. But in any case, it seems like adding alternate-port-number support for Unix sockets and not doing it for TCP ports at the same time is just weird. So I think it's a separate feature and should be a separate patch. regards, tom lane
<br /><br /><div class="gmail_quote">On Tue, Jul 3, 2012 at 11:51 AM, Tom Lane <span dir="ltr"><<a href="mailto:tgl@sss.pgh.pa.us"target="_blank">tgl@sss.pgh.pa.us</a>></span> wrote:<br /><blockquote class="gmail_quote"style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">I wrote:<br /> >On the whole I prefer the solution you mention above: let's generalize<br /> > the postmaster.pid format (and pg_ctl)so that we don't need to assume<br /> > anything about port numbers matching up. The nearby discussion about<br/> > allowing listen_addresses to specify port number would break this<br /> > assumption anyway. If we justadd two port numbers into postmaster.pid,<br /> > one for the Unix socket and one for the TCP port, we could get ridof<br /> > the problem entirely.<br /><br /></div>After further thought, I think that this approach would make it agood<br /> idea to drop support for alternate port numbers from the present patch.<br /> Let's just deal with alternatesocket directories for now. There could<br /> be a follow-on patch that adds support for nondefault port numbersin<br /> both listen_addresses and unix_socket_directories, and fixes up the<br /> postmaster.pid format to supportthat.<br /><br /> I will admit that part of my desire to do it this way is a narrow Fedora<br /> rationale: in theFedora package, we are going to want to back-patch the<br /> alternate-directory feature into 9.2 (and maybe 9.1) so asto fix our<br /> problems with systemd's PrivateTmp feature. The alternate-port-number<br /> feature is not necessaryfor that, and leaving it out would make for a<br /> significantly smaller back-patch. But in any case, it seemslike adding<br /> alternate-port-number support for Unix sockets and not doing it for TCP<br /> ports at the same timeis just weird. So I think it's a separate<br /> feature and should be a separate patch.<br /><div class="HOEnZb"><divclass="h5"> <br /></div></div></blockquote></div><br />+1<br /><br />I still findit difficult to think of a good use case for multiple ports.<br /><br />cheers<br /><br />andrew<br />
On 07/02/2012 09:45 PM, Tom Lane wrote: > Honza Horak <hhorak@redhat.com> writes: >> On 06/15/2012 05:40 PM, Honza Horak wrote: >>> I realized the patch has some difficulties -- namely the socket path in the data dir lock file, which currently usesone port for socket and the same for interface. So to allow users to use arbitrary port for all unix sockets, we'd needto add another line only for unix socket, which doesn't apply for other platforms. Or we could just say that the firstsocket will allways use the default port (PostPortNumber), which is a solution I prefer currently, but will be gladfor any other opinion. This is also why there is still un-necesary string splitting in pg_ctl.c, which will be removedafter the issue above is solved. > > I did a review pass over this patch. > I have finally an enhanced patch, see the attachment and feel free to comment. > Well, not so much "forbids" as "silently ignores", which doesn't seem like > great user-interface design to me. If we're going to adopt this solution > I think we need it to throw an error instead of just ignoring the port > specification. Alternate-port-number support has been removed from the patch, as per Tom's e-mail from 07/03/12. It can be add in the future, if we really need it. > * I'm not especially thrilled with propagating SplitUnixDirectories calls > into those two places anyway, nor with the weird decision for > SplitUnixDirectories to return a separate "mainSocket" value. Perhaps > what would be most sensible is to attach an assign hook to the > unix_socket_directories GUC parameter that would automatically split the > string and store the components into a globally-visible List variable > (which could replace the globally-visible string value we have now). Replacing the old global string value would probably need a new configuration type "List" to be added, since otherwise guc works with it as with a string. Adding that seems like too big overhead to me and thus it seems better to add a new global (List *) variable and let the original value of type (char *) to store non-parsed value. Except that I believe all other Tom's comments have been involved. Regards, Honza
Attachment
Honza Horak <hhorak@redhat.com> writes: > Alternate-port-number support has been removed from the patch, as per > Tom's e-mail from 07/03/12. It can be add in the future, if we really > need it. I've reviewed and committed this. There were some cosmetic things I adjusted, as well as a couple of fairly large non-cosmetic things: * I did not like rearranging the order in which TCP and Unix sockets get opened. It's possible that this comment in postmaster.c * For the same reason, it's best to grab the TCP socket(s) before the * Unix socket. is no longer relevant, but I'm doubtful of that. The reason you had switched them appeared to be to update the SOCKET_DIR line in postmaster.pid before updating the LISTEN_ADDR line, but the only reason to do that is the implementation restriction in AddToDataDirLockFile that it can't update a non-last line in the pidfile. That's not that hard to get rid of, and it's something we'd probably want someday anyway, so I fixed that function and put the socket opening order back as it had been. * The code in pqcomm.c queued another on_proc_exit function for each socket. There was no purpose in that since the first one would do all the work, but the postmaster's on_proc_exit array isn't very large, and it's not hard to foresee the useless entries causing a failure by overflowing the array. Similarly, miscinit.c queued an on_proc_exit function for each lock file it had to get rid of, which was fine as long as there was an upper bound of 2, but now not so much. I fixed them to use similar logic of keeping a list of file names and queueing the on_proc_exit function when adding the first list entry. * You'd fixed TouchSocketFiles to touch all the sockets, but missed making TouchSocketLockFiles touch all their lock files. That would be problematic if /tmp was a non-first entry in the list. I also simplified the GUC interactions by leaving the GUC variable as a simple string and splitting it at the point of use, so that the code is more parallel to what we do with listen_addresses. regards, tom lane