Thread: Re: [PATCHES] Avahi support for Postgresql
Hello, Congratulations to the 8.3 release. No, that pre-release stress is over I'd really like to put attention on my Avahi patches again. Avahi/Bonjour/DNS-SD support[1] is very important, for integrating Postgresql with modern desktop environments like OSX, GNOME, KDE: It's very convenient to choose active DBMS servers in your local network from a list, instead of memorizing "cryptic" connection parameters. In Glom[2] we use Postgresql as DBMS of choice. This application would greatly benefit from the possibility to choose central Postgresql servers from an automatically updated list. For OSX builds Postgresql already has DNS-SD support via Apple's Bonjour library. For Linux Avahi[3] is the library of choice. Modern distribu- tions ship it as main package, since both GNOME and KDE make use of it. Since DNS-SD is a published IETF standard [4] client applications choose whatever DNS-SD implementation they like. Applying this patche series would bring Postgresql's Linux port on par with the OSX port in terms of desktop integration. People not wanting DNS-SD support for their server can easily control that feature via the "--with-avahi" configure scripts. Thank your for attention, Mathias [1] http://www.dns-sd.org/ [2] http://www.glom.org/ [3] http://www.avahi.org/ [4] http://www.ietf.org/rfc/rfc2782.txt Am Dienstag, den 27.11.2007, 11:19 +0100 schrieb Mathias Hasselmann: > Postmaster already has code to announce its services via DNS-SD > (ZeroConf) by using Apple's Bonjour API. This series of patches > implements that capability on top of the Avahi library[1] which > is free software, available for a wider variety of platforms. > > I've separated the change set into smaller pieces for easy review. Also > I don't know, if the 4th patch adding subtypes describing Postgresql's > capabilities is wanted by the Postgresql community. Being a desktop guy, > using Postgresql as embedded database, I'd really like to see all of the > patches merged, but I could imagine some opposition against the verbose > nature of the subtypes patch. > > Each of the patches as a short description of its purpose. > > I couldn't find any ChangeLog files in the repository, and also I didn't > invest much time in investigating how the HISTORY file of distribution > tarballs is assembled. So if the repository contains file I should have > modified for documenting my changes, I apologize for not updating them. > > [1] http://www.avahi.org/ > ---------------------------(end of broadcast)--------------------------- > TIP 1: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly -- Mathias Hasselmann <mathias@openismus.com> http://www.openismus.com/ - We can get it done.
Mathias Hasselmann wrote: > Congratulations to the 8.3 release. No, that pre-release stress is over > I'd really like to put attention on my Avahi patches again. I've added your item to the patch list (http://developer.postgresql.org/index.php/Todo:PatchStatus). It would help if you could include or point to your actual patch. -- Peter Eisentraut http://developer.postgresql.org/~petere/
Am Samstag, den 23.02.2008, 21:08 +0100 schrieb Peter Eisentraut: > Mathias Hasselmann wrote: > > Congratulations to the 8.3 release. Now, that pre-release stress is over > > I'd really like to put attention on my Avahi patches again. > > I've added your item to the patch list > (http://developer.postgresql.org/index.php/Todo:PatchStatus). It would help > if you could include or point to your actual patch. Thanks a lot. The patches were in my initial mail, but now I've also uploaded them to my personal site for convenience: http://taschenorakel.de/files/pgsql-avahi-support/ I just failed to update the Wiki page: This page has been locked to prevent editing. You can view and copy the source of this page: Ciao, Mathias -- Mathias Hasselmann <mathias@openismus.com> http://www.openismus.com/ - We can get it done.
Mathias Hasselmann wrote: > The patches were in my initial mail, but now I've also uploaded them to > my personal site for convenience: > > http://taschenorakel.de/files/pgsql-avahi-support/ Hmm, a quick look at the third patch reveals that it is using the "threaded" Avahi client. That's a showstopper. Please consider using some other approach -- writing our own handlers for AvahiPoll would seem apropos. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
Am Montag, 25. Februar 2008 schrieb Alvaro Herrera: > Hmm, a quick look at the third patch reveals that it is using the > "threaded" Avahi client. That's a showstopper. Could you elaborate why that is? -- Peter Eisentraut http://developer.postgresql.org/~petere/
Peter Eisentraut wrote: > Am Montag, 25. Februar 2008 schrieb Alvaro Herrera: > > Hmm, a quick look at the third patch reveals that it is using the > > "threaded" Avahi client. That's a showstopper. > > Could you elaborate why that is? Because it creates a new thread under the Postmaster to handle Avahi events, if I'm reading the Avahi docs right. This is verboten. We have an event loop in the postmaster -- see ServerLoop. Is there a reason the Avahi events could not be hooked in there? -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Am Montag, den 25.02.2008, 14:32 -0300 schrieb Alvaro Herrera: > Peter Eisentraut wrote: > > Am Montag, 25. Februar 2008 schrieb Alvaro Herrera: > > > Hmm, a quick look at the third patch reveals that it is using the > > > "threaded" Avahi client. That's a showstopper. > > > > Could you elaborate why that is? > > Because it creates a new thread under the Postmaster to handle Avahi > events, if I'm reading the Avahi docs right. This is verboten. Just to be sure we talk about the same topic: I assume the prohibition you talk about is something like "no use of threads in Postmaster"? If that's the case: Are there some docs, mails, ... explaining the rationale behind this restriction? I could imagine your do not want random locking in the postmaster code? See, interaction points with the main thread are very small: 1) Lock-free creation of the threaded Avahi client in PostmasterMain() 2) Locked shutdown of the Avahi client in ExitAvahiClient(), which only is called from ExitPostmaster(). So IMHO usage of the threaded poll API has much smaller impact on the behavior of the postmaster process, than any attempt to integrate Avahi with postmaster's main loop. > We have an event loop in the postmaster -- see ServerLoop. Is there a > reason the Avahi events could not be hooked in there? Currently there are four ___well tested___ implementations of Avahi's poll API: AvahiSimplePoll, which really just works for simple command line tools and demonstration purposes. The single threaded APIs that integrate with the main loops of glib and Qt, and the threaded poll API. Avahi's requirements for a poll API aren't exactly trivial: You don't only have to care about file descriptors, you also have to implement some kind of timeout scheduling. So in favor of reinventing the wheel and delivering an untested custom poll API, I've chosen the threaded poll API: It's the only well-tested poll API that fits into Postgresql, and its interaction points with the Postmaster process are minimal. >From looking at ServerLoop() I do not see any facilities for registering timeout callbacks. Select timeouts are static. So for implementing Avahi's poll API in ServerLoop() some radical code changes would be needed. I don't believe such changes would be justified, unless other portions of postmaster also need timeout callbacks. Ciao, Mathias -- Mathias Hasselmann <mathias@openismus.com> http://www.openismus.com/ - We can get it done.
Mathias Hasselmann <mathias@openismus.com> writes: > Just to be sure we talk about the same topic: I assume the prohibition > you talk about is something like "no use of threads in Postmaster"? Correct. > If that's the case: Are there some docs, mails, ... explaining the > rationale behind this restriction? I could imagine your do not want > random locking in the postmaster code? Portability, irreproducible misbehavior, etc. Some trawling in the pgsql-hackers archives should turn up previous discussions. For a recent demonstration that wanting to avoid threads is not just idle paranoia on our part, see http://archives.postgresql.org/pgsql-patches/2007-09/msg00194.php regards, tom lane
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sat, Feb 23, 2008 at 01:13:38PM +0100, Mathias Hasselmann wrote: [...] > Avahi/Bonjour/DNS-SD support[1] is very important, for integrating > Postgresql with modern desktop environments like OSX, GNOME, KDE: It's > very convenient to choose active DBMS servers in your local network from > a list, instead of memorizing "cryptic" connection parameters. [...] > People not wanting DNS-SD support for their server can easily control > that feature via the "--with-avahi" configure scripts. Sorry for a dumb question, but I couldn't figure that out from your references [1]..[4]: does that mean that the PostgreSQL server would "advertise itself" on the local net? Or what is the purpose of liking-in libavahi into the postmaster? Surely one wouldn't want this in a data center? Is there a possiblity to disable that at run time? Thanks for any insights - -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFH7jUxBcgs9XrR2kYRAsfOAJ0Ulfydo3G1kzQo3HTOdjtswA1A2gCfYYyL VHprJ63unId85/Iht4ha2RE= =get0 -----END PGP SIGNATURE-----
Am Samstag, den 29.03.2008, 12:25 +0000 schrieb tomas@tuxteam.de: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Sat, Feb 23, 2008 at 01:13:38PM +0100, Mathias Hasselmann wrote: > > [...] > > > Avahi/Bonjour/DNS-SD support[1] is very important, for integrating > > Postgresql with modern desktop environments like OSX, GNOME, KDE: It's > > very convenient to choose active DBMS servers in your local network from > > a list, instead of memorizing "cryptic" connection parameters. > > [...] > > > People not wanting DNS-SD support for their server can easily control > > that feature via the "--with-avahi" configure scripts. > > Sorry for a dumb question, but I couldn't figure that out from your > references [1]..[4]: does that mean that the PostgreSQL server would > "advertise itself" on the local net? Or what is the purpose of liking-in > libavahi into the postmaster? Yes, that's the purpose. > Surely one wouldn't want this in a data center? Yes, this feature definitely targets small-office use, personal use, DB developers. Don't know enough about data centers to judge the impact there, but since Avahi - as used in the patch - announces to the local network only, the impact sould be small. Still you can tell Avahi to explicitly announce at a certain, non-local domain, but this feature is not implemented by the patch. Maybe database developers in large network environments could make use of such announcements. It would be trivial to add. > Is there a possiblity to disable that at run time? The feature is disabled by default. As long as you do not specify a zeroconf_name in your configuration file, nothing happens. This is the same behavior as established by the Bonjour code. Ciao, Mathias -- Mathias Hasselmann <mathias@openismus.com> http://www.openismus.com/ - We can get it done.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, Apr 01, 2008 at 09:35:56AM +0200, Mathias Hasselmann wrote: > Am Samstag, den 29.03.2008, 12:25 +0000 schrieb tomas@tuxteam.de: > [...] > > Sorry for a dumb question, but I couldn't figure that out from your > > references [1]..[4]: does that mean that the PostgreSQL server would > > "advertise itself" on the local net? Or what is the purpose of liking-in > > libavahi into the postmaster? > > Yes, that's the purpose. > > > Surely one wouldn't want this in a data center? > > Yes, this feature definitely targets small-office use, personal use, DB > developers [...] > Still you can tell Avahi to explicitly announce at a certain, non-local > domain, but this feature is not implemented by the patch. Maybe database > developers in large network environments could make use of such > announcements. It would be trivial to add. Personally, I'be rather scared than delighted ;-) > > Is there a possiblity to disable that at run time? > > The feature is disabled by default. As long as you do not specify a > zeroconf_name in your configuration file, nothing happens. This is the > same behavior as established by the Bonjour code. Thanks, good to know. Isn't there a less-intrusive option to linking a lib into each and every possible server, like a config file in which to put what is to be announced? Regards - -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFH8iRFBcgs9XrR2kYRAmJ0AJkB7MkxfYI0nVa4RqHVEV1HYjz41gCdEgWz YQ2T4Y/xfoLRF4D6hMLbpEk= =Goho -----END PGP SIGNATURE-----
Am Dienstag, den 01.04.2008, 12:02 +0000 schrieb tomas@tuxteam.de: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Tue, Apr 01, 2008 at 09:35:56AM +0200, Mathias Hasselmann wrote: > > Am Samstag, den 29.03.2008, 12:25 +0000 schrieb tomas@tuxteam.de: > > [...] > > > Sorry for a dumb question, but I couldn't figure that out from your > > > references [1]..[4]: does that mean that the PostgreSQL server would > > > "advertise itself" on the local net? Or what is the purpose of liking-in > > > libavahi into the postmaster? > > > > Yes, that's the purpose. > > > > > Surely one wouldn't want this in a data center? > > > > Yes, this feature definitely targets small-office use, personal use, DB > > developers [...] > > Still you can tell Avahi to explicitly announce at a certain, non-local > > domain, but this feature is not implemented by the patch. Maybe database > > developers in large network environments could make use of such > > announcements. It would be trivial to add. > > Personally, I'be rather scared than delighted ;-) So in data centers you don't even trust the machines in your broadcast domain? > > > Is there a possiblity to disable that at run time? > > > > The feature is disabled by default. As long as you do not specify a > > zeroconf_name in your configuration file, nothing happens. This is the > > same behavior as established by the Bonjour code. > > Thanks, good to know. > > Isn't there a less-intrusive option to linking a lib into each and every > possible server, like a config file in which to put what is to be announced? You could directly talk to the D-Bus interface of Avahi. libavahi-client just is a convenience wrapper. Well, but this route will be much more cumbersome. One other route is calling avahi-publish-service on startup and killing it on shutdown, but: avahi-publish-service really only exists for demonstration purposes and doesn't handle service name collisions for instance. I don't believe that a high-profile application like Postgresql should rely on low-quality hacks, like invoking educational demo programs. Ciao, Mathias -- Mathias Hasselmann <mathias@openismus.com> http://www.openismus.com/ - We can get it done.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, Apr 01, 2008 at 05:07:31PM +0200, Mathias Hasselmann wrote: [...] > > Personally, I'be rather scared than delighted ;-) > > So in data centers you don't even trust the machines in your broadcast > domain? Kind of. Put it another way: never have services running you don't use. [...] > > Isn't there a less-intrusive option to linking a lib into each and every > > possible server, like a config file in which to put what is to be announced? > > You could directly talk to the D-Bus interface of Avahi. libavahi-client > just is a convenience wrapper. Well, but this route will be much more > cumbersome. So this goes through the D-Bus. Makes kind of sense. Thanks for the enlightenment. > One other route is calling avahi-publish-service on startup and killing > it on shutdown, but: avahi-publish-service really only exists for > demonstration purposes and doesn't handle service name collisions for > instance. I don't believe that a high-profile application like > Postgresql should rely on low-quality hacks, like invoking educational > demo programs. Unelegant as it might seem -- this solution still affords a lot more when it comes to "separation of concerns". I'm still a bit wary at the prospect that each and every daemon evolves into a huge fuzzball linked to all conceivable service-lets with a multitude of funny side-effects (remember tcpwrappers?). Of course, "you can always disable this at compile time", but let's face it: with the predominance of binary distribs, the path of least resistance will be to put up with whatever strange side-effects. I would really prefer a more loosely coupled system. Regards - -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFH8lYJBcgs9XrR2kYRAmDJAJ4jWKYkhUfKEAIaZVnIbAAEqJF2AwCfS/6D 4rH9OoY7wjia7h1cuk5CjZI= =AF1W -----END PGP SIGNATURE-----
On Tue, 2008-04-01 at 15:34 +0000, tomas@tuxteam.de wrote: > I would really prefer a more loosely coupled system. The functionality will be much the same. The implementation would be more difficult and obscure and there would be more points of failure and more things to configure, but it wouldn't remove much risk, I think. Anyway, this feature is already in Postgres when it's built for MacOS X. So this decision seems to have been made already, at least for that platform. -- murrayc@murrayc.com www.murrayc.com www.openismus.com
Alvaro Herrera wrote: > Mathias Hasselmann wrote: > > > The patches were in my initial mail, but now I've also uploaded them to > > my personal site for convenience: > > > > http://taschenorakel.de/files/pgsql-avahi-support/ > > Hmm, a quick look at the third patch reveals that it is using the > "threaded" Avahi client. That's a showstopper. Please consider using > some other approach -- writing our own handlers for AvahiPoll would seem > apropos. Based on the threading usage in this patch, I assume it is rejected, and that we don't want a TODO item for this. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce Momjian wrote: > Alvaro Herrera wrote: > > Mathias Hasselmann wrote: > > > > > The patches were in my initial mail, but now I've also uploaded them to > > > my personal site for convenience: > > > > > > http://taschenorakel.de/files/pgsql-avahi-support/ > > > > Hmm, a quick look at the third patch reveals that it is using the > > "threaded" Avahi client. That's a showstopper. Please consider using > > some other approach -- writing our own handlers for AvahiPoll would seem > > apropos. > > Based on the threading usage in this patch, I assume it is rejected, and > that we don't want a TODO item for this. This particular patch should be rejected, but we certainly do need a TODO item IMHO. -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Added to TODO: * Implement the non-threaded Avahi service discovery protocol http://archives.postgresql.org/pgsql-hackers/2008-02/msg00939.php http://archives.postgresql.org/pgsql-patches/2008-02/msg00097.php http://archives.postgresql.org/pgsql-hackers/2008-03/msg01211.php http://archives.postgresql.org/pgsql-patches/2008-04/msg00001.php --------------------------------------------------------------------------- Alvaro Herrera wrote: > Bruce Momjian wrote: > > Alvaro Herrera wrote: > > > Mathias Hasselmann wrote: > > > > > > > The patches were in my initial mail, but now I've also uploaded them to > > > > my personal site for convenience: > > > > > > > > http://taschenorakel.de/files/pgsql-avahi-support/ > > > > > > Hmm, a quick look at the third patch reveals that it is using the > > > "threaded" Avahi client. That's a showstopper. Please consider using > > > some other approach -- writing our own handlers for AvahiPoll would seem > > > apropos. > > > > Based on the threading usage in this patch, I assume it is rejected, and > > that we don't want a TODO item for this. > > This particular patch should be rejected, but we certainly do need a > TODO item IMHO. > > -- > Alvaro Herrera http://www.CommandPrompt.com/ > PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +