Thread: Plans for 8.4
I'm looking at shortlisting things that I'll be working on for 8.4. I've thrown out a few small patches I had, but now am looking at these projects: * Hot Standby * Logical replication via transaction log * Integration with synchronous replication * Join removal * stats hooks * DDL locking reductions Initial design work on them is mostly done, just need to start posting designs and patches. I've got a early days prototype of Hot Standby that allows connection, plus months of thinking on the other topics, so I'm further ahead than it may previously have appeared. All of this is dependent upon sponsorship... THANK YOU to current sponsors. I've had to break off work on MERGE now. For lack of time, plus the presence of some unresolved details in the design that make it questionable whether it would be fully accepted. I've also punted on a few other items that look like more time than value to the project, in my opinion. But it might be possible to pick up later, depending what happens in next few months. -- Simon Riggs www.2ndQuadrant.comPostgreSQL Training, Services and Support
Hi, Simon Riggs wrote: > * Logical replication via transaction log > * Integration with synchronous replication I'm curious on what you mean by these two points. AFAIK you are speaking of "logical" replication for methods which don't rely on the internal storage format of the database, but instead replicate closer to the SQL level. But the transaction log (the XLOG for Postres, REDO log for Oracle, etc..) is certainly bound to the storage format, so I'd classify the log shipping approaches as "physical" replication. What is it you are talking about? What do you plan to integrate with synchronous replication? Did you consider having a look at the Postgres-R project? Regards Markus Wanner
On Tue, 2008-07-22 at 10:23 -0700, David E. Wheeler wrote: > On Jul 22, 2008, at 08:35, Simon Riggs wrote: > > > * Hot Standby > > * Logical replication via transaction log > > * Integration with synchronous replication > > Getting these in 8.4 would so rock. Agreed. Thank me when its done, 'cos it ain't done yet by a long way! -- Simon Riggs www.2ndQuadrant.comPostgreSQL Training, Services and Support
On Tue, 2008-07-22 at 18:37 +0200, Markus Wanner wrote: > Simon Riggs wrote: > > * Logical replication via transaction log > > * Integration with synchronous replication > > I'm curious on what you mean by these two points. AFAIK you are speaking > of "logical" replication for methods which don't rely on the internal > storage format of the database, but instead replicate closer to the SQL > level. But the transaction log (the XLOG for Postres, REDO log for > Oracle, etc..) is certainly bound to the storage format, so I'd classify > the log shipping approaches as "physical" replication. What is it you > are talking about? Reconstructing SQL from WAL, allowing logical apply. So yes, you're right, the actual replication of the data from one node to another is physical - its the final apply of the changes that is logical. So this fits neatly with our commitment to put synchronous replication into server. It allows WAL to be used as the replication transport, which reduces the impact of tracking changes via triggers. The presence of commit records in the sequence can be used to parallelize the apply phase if required. I'm proposing to put the mechanisms in place to allow WAL to be used by other replication projects. Which turn out to be relatively small changes to the main server. -- Simon Riggs www.2ndQuadrant.comPostgreSQL Training, Services and Support
On Jul 22, 2008, at 08:35, Simon Riggs wrote: > * Hot Standby > * Logical replication via transaction log > * Integration with synchronous replication Getting these in 8.4 would so rock. Thank you! David
Simon Riggs <simon@2ndquadrant.com> writes:
> On Tue, 2008-07-22 at 18:37 +0200, Markus Wanner wrote:
>> What is it you
>> are talking about?
> Reconstructing SQL from WAL, allowing logical apply.
> ...
> I'm proposing to put the mechanisms in place to allow WAL to be used by
> other replication projects. Which turn out to be relatively small
> changes to the main server.
[ retrieves eyebrows from ceiling... ] I doubt that's even possible,
let alone a "relatively small change".
regards, tom lane
Hi, Simon Riggs wrote: > Reconstructing SQL from WAL, allowing logical apply. So yes, you're > right, the actual replication of the data from one node to another is > physical - its the final apply of the changes that is logical. So this > fits neatly with our commitment to put synchronous replication into > server. Aha, thanks for your explanation. > It allows WAL to be used as the replication transport, which reduces the > impact of tracking changes via triggers. The presence of commit records > in the sequence can be used to parallelize the apply phase if required. I note you are comparing against trigger based replication solutions. > I'm proposing to put the mechanisms in place to allow WAL to be used by > other replication projects. Which turn out to be relatively small > changes to the main server. Comparing with the Postgres-R way to do it, there's a difference in the serialized change set format. Where Postgres-R uses it's a specialized format, you are proposing to use the existing WAL format to transfer change set data, which is architecture dependent and quite tightly bound to the physical format on disc. It cannot be changed to also carry other information required for replication, without having an impact on WAL for single node installations. I think the CPU load to serialize and deserialize is pretty similar and most probably negligible. Regards Markus
On Tue, 2008-07-22 at 16:54 -0400, Tom Lane wrote: > Simon Riggs <simon@2ndquadrant.com> writes: > > On Tue, 2008-07-22 at 18:37 +0200, Markus Wanner wrote: > >> What is it you > >> are talking about? > > > Reconstructing SQL from WAL, allowing logical apply. > > ... > > I'm proposing to put the mechanisms in place to allow WAL to be used by > > other replication projects. Which turn out to be relatively small > > changes to the main server. > > [ retrieves eyebrows from ceiling... ] I doubt that's even possible, > let alone a "relatively small change". I'm talking about building the transport layer into Postgres core only. SMOP outside of core, with the right changes. There's always potential blockers to any problem. We'll see when I publish the design. -- Simon Riggs www.2ndQuadrant.comPostgreSQL Training, Services and Support
What's the time frame for 8.4? I'm making no promises, but what would people think of a hostgss hba option? Using it would imply the gssapi/sspi authentication option. It would be mutually exclusive of the ssl link-encryption option. It would support strong encryption of the whole connection without the need to get X509 certs deployed (which would be a big win if you're using gssapi/sspi authentication anyway). The thing that prevented me from including it in the gssapi patches I did for 8.3 was that I couldn't disentangle the program logic to the point of inserting the gssapi security layer code above the SSL code and below everything else. I'm thinking that doing both is pretty much an edge case, so I propose to do gssapi security layers instead of SSL. The mods are a lot more obvious. I'm *NOT* proposing to make build support of gssapi security layers exclusive of SSL. You might, for example, configure a server to support username/password over SSL for intra-net addresses, but support gssapi for Internet addresses. ------------------------------------------------------ The opinions expressed in this message are mine, not those of Caltech, JPL, NASA, or the US Government. Henry.B.Hotz@jpl.nasa.gov, or hbhotz@oxy.edu
* Henry B. Hotz (hbhotz@oxy.edu) wrote:
> I'm making no promises, but what would people think of a hostgss hba
> option?
As described, sounds like a win to me. It'd be very nice to be able to
just use GSSAPI encryption on the link. That, combined w/ Magnus' work
on username/princ mappings, would really bring PostgreSQL up to date wrt
GSSAPI support.
It'd really be great to have this support in the ODBC and JDBC drivers
too.. I think in JDBC it might 'just work', I'm less sure about ODBC.
As a practical question- would you really need a seperate explicit
pg_hba option for it? It'd be nice to be able to require it, if
desired, but that strikes me as more sensible as an option to the 'gss'
auth mechanism?
Thanks!
Stephen
"Henry B. Hotz" <hbhotz@oxy.edu> writes: > What's the time frame for 8.4? http://wiki.postgresql.org/wiki/PostgreSQL_8.4_Development_Plan Executive summary: new-feature patches had better be submitted before 1 November. > I'm making no promises, but what would people think of a hostgss hba > option? Not qualified to comment on the usefulness of this ... regards, tom lane
Stephen Frost wrote: > * Henry B. Hotz (hbhotz@oxy.edu) wrote: >> I'm making no promises, but what would people think of a hostgss hba >> option? > > As described, sounds like a win to me. It'd be very nice to be able to > just use GSSAPI encryption on the link. That, combined w/ Magnus' work > on username/princ mappings, would really bring PostgreSQL up to date wrt > GSSAPI support. Yeah, +1 on this feature, it would be quite useful. > It'd really be great to have this support in the ODBC and JDBC drivers > too.. I think in JDBC it might 'just work', I'm less sure about ODBC. ODBC will need hackery I think. They use libpq for authentication only, but have their own SSL code and such. I do think ODBC would be a fairly major point to it being a success, though, so it'd be good if a plan could be secured for it. But it's not a showstopper, of course. > As a practical question- would you really need a seperate explicit > pg_hba option for it? It'd be nice to be able to require it, if > desired, but that strikes me as more sensible as an option to the 'gss' > auth mechanism? Yeah, if we can get rid of that, that'd be good. The stuff I'm working on will allow us to have multiple parameters for each row in name/value pairs, so if we could use that, it'd be better. (I've been considering changing how host/hostssl work that way as well - by having a parameter similar to what we have on the client side with sslmode=...) A thought that I came across - is it even possible to use GSSAPI encryption *without* using GSSAPI authentication? If not, it really seems like it should belong more in the parameter part of the field. Since in that case it is also not possible to enable encryption *before* authentication, or is it? //Magnus
On Jul 31, 2008, at 7:58 AM, Magnus Hagander wrote: > Stephen Frost wrote: >> * Henry B. Hotz (hbhotz@oxy.edu) wrote: >>> I'm making no promises, but what would people think of a hostgss hba >>> option? >> >> As described, sounds like a win to me. It'd be very nice to be >> able to >> just use GSSAPI encryption on the link. That, combined w/ Magnus' >> work >> on username/princ mappings, would really bring PostgreSQL up to >> date wrt >> GSSAPI support. > > Yeah, +1 on this feature, it would be quite useful. > > >> It'd really be great to have this support in the ODBC and JDBC >> drivers >> too.. I think in JDBC it might 'just work', I'm less sure about >> ODBC. > > ODBC will need hackery I think. They use libpq for authentication > only, > but have their own SSL code and such. I do think ODBC would be a > fairly > major point to it being a success, though, so it'd be good if a plan > could be secured for it. But it's not a showstopper, of course. I don't know enough about ODBC. If ODBC does SSL independently of PG then it requires thought by someone who understands ODBC. > >> As a practical question- would you really need a seperate explicit >> pg_hba option for it? It'd be nice to be able to require it, if >> desired, but that strikes me as more sensible as an option to the >> 'gss' >> auth mechanism? > > Yeah, if we can get rid of that, that'd be good. The stuff I'm working > on will allow us to have multiple parameters for each row in name/ > value > pairs, so if we could use that, it'd be better. (I've been considering > changing how host/hostssl work that way as well - by having a > parameter > similar to what we have on the client side with sslmode=...) > > A thought that I came across - is it even possible to use GSSAPI > encryption *without* using GSSAPI authentication? If not, it really > seems like it should belong more in the parameter part of the field. > Since in that case it is also not possible to enable encryption > *before* > authentication, or is it? You're on the right track. My problem isn't the hba file parsing at all. My problem is the interaction between the buffering logic and the encrypted I/O routines. The technical issue is that to make a GSSAPI security layer independent of SSL you need to invent a whole new buffering layer. That's a lot of work, and it only buys you the ability to do both SSL and GSSAPI at the same time. That doesn't seem worth it. The code being affected is what's currently configured in column 1 of hba. The ability to use the new capability requires that SSL *NOT* be configured in column 1 for the relevant client addresses. In short, no, it doesn't make sense to make it an option to the gss authentication method, even though it requires it. If we make it an option to the gss authentication method it would still need to act like it was specified in column 1, which would be confusing. GSSAPI security layers are negotiated after the authentication (or at least after the start of authentication). There are GSSAPI status flags that indicate if the security layer is available yet. The GSSAPI security layer code would check those flags and gss_wrap() or not accordingly. (-: There's a flush() or two from my original patch that will need to be added back in, otherwise we'll encrypt a message that tells the other end how to decrypt messages. Not a big deal. ;-) ------------------------------------------------------ The opinions expressed in this message are mine, not those of Caltech, JPL, NASA, or the US Government. Henry.B.Hotz@jpl.nasa.gov, or hbhotz@oxy.edu
Henry B. Hotz wrote: > > On Jul 31, 2008, at 7:58 AM, Magnus Hagander wrote: > >> Stephen Frost wrote: >>> * Henry B. Hotz (hbhotz@oxy.edu) wrote: >>>> I'm making no promises, but what would people think of a hostgss hba >>>> option? >>> >>> As described, sounds like a win to me. It'd be very nice to be able to >>> just use GSSAPI encryption on the link. That, combined w/ Magnus' work >>> on username/princ mappings, would really bring PostgreSQL up to date wrt >>> GSSAPI support. >> >> Yeah, +1 on this feature, it would be quite useful. >> >> >>> It'd really be great to have this support in the ODBC and JDBC drivers >>> too.. I think in JDBC it might 'just work', I'm less sure about ODBC. >> >> ODBC will need hackery I think. They use libpq for authentication only, >> but have their own SSL code and such. I do think ODBC would be a fairly >> major point to it being a success, though, so it'd be good if a plan >> could be secured for it. But it's not a showstopper, of course. > > I don't know enough about ODBC. If ODBC does SSL independently of PG > then it requires thought by someone who understands ODBC. I just meant please consider coordinating with the ODBC folks to make sure it gets in there as well - and in time for the same release. >>> As a practical question- would you really need a seperate explicit >>> pg_hba option for it? It'd be nice to be able to require it, if >>> desired, but that strikes me as more sensible as an option to the 'gss' >>> auth mechanism? >> >> Yeah, if we can get rid of that, that'd be good. The stuff I'm working >> on will allow us to have multiple parameters for each row in name/value >> pairs, so if we could use that, it'd be better. (I've been considering >> changing how host/hostssl work that way as well - by having a parameter >> similar to what we have on the client side with sslmode=...) >> >> A thought that I came across - is it even possible to use GSSAPI >> encryption *without* using GSSAPI authentication? If not, it really >> seems like it should belong more in the parameter part of the field. >> Since in that case it is also not possible to enable encryption *before* >> authentication, or is it? > > You're on the right track. My problem isn't the hba file parsing at all. > > My problem is the interaction between the buffering logic and the > encrypted I/O routines. The technical issue is that to make a GSSAPI > security layer independent of SSL you need to invent a whole new > buffering layer. That's a lot of work, and it only buys you the ability > to do both SSL and GSSAPI at the same time. That doesn't seem worth it. Yeah, there seems to be no general point in that. However, implementing a layer there might have other benefits. Such as being able to use other SSL implementations (right now we only do OpenSSL. There has been talk about GnuTLS, and it would be good to be able to do schannel) > The code being affected is what's currently configured in column 1 of > hba. The ability to use the new capability requires that SSL *NOT* be > configured in column 1 for the relevant client addresses. In short, no, > it doesn't make sense to make it an option to the gss authentication > method, even though it requires it. If we make it an option to the gss > authentication method it would still need to act like it was specified > in column 1, which would be confusing. Does this hold even if we move the "hostssl" stuff into a parameter "at the end"? I was thinking maybe something like: host all all 0.0.0.0/0 md5 ssl=require host all all 0.0.0.0/0 gss ssl=forbid gssencrypt=require (you get the idea) > GSSAPI security layers are negotiated after the authentication (or at > least after the start of authentication). There are GSSAPI status flags > that indicate if the security layer is available yet. The GSSAPI > security layer code would check those flags and gss_wrap() or not > accordingly. (-: There's a flush() or two from my original patch that > will need to be added back in, otherwise we'll encrypt a message that > tells the other end how to decrypt messages. Not a big deal. ;-) Ok, makes sense. //Magnus
Added to TODO: * Add 'hostgss' pg_hba.conf option to allow GSS link-level encryption http://archives.postgresql.org/pgsql-hackers/2008-07/msg01454.php --------------------------------------------------------------------------- Henry B. Hotz wrote: > What's the time frame for 8.4? > > I'm making no promises, but what would people think of a hostgss hba > option? > > Using it would imply the gssapi/sspi authentication option. It would > be mutually exclusive of the ssl link-encryption option. It would > support strong encryption of the whole connection without the need to > get X509 certs deployed (which would be a big win if you're using > gssapi/sspi authentication anyway). > > The thing that prevented me from including it in the gssapi patches I > did for 8.3 was that I couldn't disentangle the program logic to the > point of inserting the gssapi security layer code above the SSL code > and below everything else. I'm thinking that doing both is pretty > much an edge case, so I propose to do gssapi security layers instead > of SSL. The mods are a lot more obvious. > > I'm *NOT* proposing to make build support of gssapi security layers > exclusive of SSL. You might, for example, configure a server to > support username/password over SSL for intra-net addresses, but > support gssapi for Internet addresses. > > ------------------------------------------------------ > The opinions expressed in this message are mine, > not those of Caltech, JPL, NASA, or the US Government. > Henry.B.Hotz@jpl.nasa.gov, or hbhotz@oxy.edu > > > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +