Thread: [pgsql-advocacy] Assembling "top features" list for beta announcement
Folks: While 10 isn't 100% baked yet, I'd like to get started sorting out what our "top 2-4" features for the beta announcement are going to be. Traditionally, we've mentioned two to four features in detail, together with a bullet list of other significant features. With the help of PVH, here's what I have so far: Banner features: - pglogical logical replication - table partitions in core Significant features: - XMLTABLE query expressions - hash indexes in wal - PGFDW: push down aggregates It looks like quorum sync rep won't make it, correct? Notable note: pg_xlog => pg_wal So .., please add to the above lists. Thanks! -- Josh Berkus Containers & Databases Oh My!
On 7 Apr 2017, at 17:39, Josh Berkus <josh@berkus.org> wrote: > Folks: > > While 10 isn't 100% baked yet, I'd like to get started sorting out what > our "top 2-4" features for the beta announcement are going to be. > Traditionally, we've mentioned two to four features in detail, together > with a bullet list of other significant features. With the help of PVH, > here's what I have so far: > > Banner features: > > - pglogical logical replication > - table partitions in core > > Significant features: > > - XMLTABLE query expressions > - hash indexes in wal > - PGFDW: push down aggregates > > It looks like quorum sync rep won't make it, correct? > > Notable note: pg_xlog => pg_wal > > So .., please add to the above lists. Thanks! Would any of these be suitable for using "pretty graphics" with? eg for illustrating potential change in query time/database load/etc Purely for making things easier for readers to grok that is. :) + Justin -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi
On 04/07/2017 09:47 AM, Justin Clift wrote: > On 7 Apr 2017, at 17:39, Josh Berkus <josh@berkus.org> wrote: >> Folks: >> >> While 10 isn't 100% baked yet, I'd like to get started sorting out what >> our "top 2-4" features for the beta announcement are going to be. >> Traditionally, we've mentioned two to four features in detail, together >> with a bullet list of other significant features. With the help of PVH, >> here's what I have so far: >> >> Banner features: >> >> - pglogical logical replication >> - table partitions in core >> >> Significant features: >> >> - XMLTABLE query expressions >> - hash indexes in wal >> - PGFDW: push down aggregates >> >> It looks like quorum sync rep won't make it, correct? >> >> Notable note: pg_xlog => pg_wal >> >> So .., please add to the above lists. Thanks! > > Would any of these be suitable for using "pretty graphics" with? > > eg for illustrating potential change in query time/database load/etc > > Purely for making things easier for readers to grok that is. :) > I'd save that for the final release announcement, but if you're prepared to do graphics, the new partitioning might be a good candidate. -- Josh Berkus Containers & Databases Oh My!
On 7 Apr 2017, at 17:52, Josh Berkus <josh@berkus.org> wrote: > On 04/07/2017 09:47 AM, Justin Clift wrote: >> On 7 Apr 2017, at 17:39, Josh Berkus <josh@berkus.org> wrote: >>> Folks: >>> >>> While 10 isn't 100% baked yet, I'd like to get started sorting out what >>> our "top 2-4" features for the beta announcement are going to be. >>> Traditionally, we've mentioned two to four features in detail, together >>> with a bullet list of other significant features. With the help of PVH, >>> here's what I have so far: >>> >>> Banner features: >>> >>> - pglogical logical replication >>> - table partitions in core >>> >>> Significant features: >>> >>> - XMLTABLE query expressions >>> - hash indexes in wal >>> - PGFDW: push down aggregates >>> >>> It looks like quorum sync rep won't make it, correct? >>> >>> Notable note: pg_xlog => pg_wal >>> >>> So .., please add to the above lists. Thanks! >> >> Would any of these be suitable for using "pretty graphics" with? >> >> eg for illustrating potential change in query time/database load/etc >> >> Purely for making things easier for readers to grok that is. :) > > I'd save that for the final release announcement, but if you're prepared > to do graphics, the new partitioning might be a good candidate. Yep, that should be something I have time for. :) + Justin -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi
On Sat, Apr 8, 2017 at 1:39 AM, Josh Berkus <josh@berkus.org> wrote: > > It looks like quorum sync rep won't make it, correct? > Quorum sync rep will be there. See: https://www.postgresql.org/docs/devel/static/runtime-config-replication.html#runtime-config-replication-master "The keyword ANY, coupled with num_sync, specifies a quorum-based synchronous replication and makes transaction commits wait until their WAL records are replicated to at least num_sync listed standbys. For example,..." Thanks, Amit
On 04/07/2017 12:30 PM, Amit Langote wrote: > On Sat, Apr 8, 2017 at 1:39 AM, Josh Berkus <josh@berkus.org> wrote: >> >> It looks like quorum sync rep won't make it, correct? >> > > Quorum sync rep will be there. See: > > https://www.postgresql.org/docs/devel/static/runtime-config-replication.html#runtime-config-replication-master > > "The keyword ANY, coupled with num_sync, specifies a quorum-based > synchronous replication and makes transaction commits wait until their > WAL records are replicated to at least num_sync listed standbys. For > example,..." Ah, I misunderstood some of the comments on the patch! That's good news, that's a 3rd banner feature. -- Josh Berkus Containers & Databases Oh My!
Re: [pgsql-advocacy] Assembling "top features" list for betaannouncement
From
Peter Eisentraut
Date:
On 4/7/17 12:39, Josh Berkus wrote: > - pglogical logical replication Note that "pglogical" != "built-in logical replication" (or whatever you want to call the latter). -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Fri, Apr 7, 2017 at 11:39 AM, Josh Berkus <josh@berkus.org> wrote: > While 10 isn't 100% baked yet, I'd like to get started sorting out what > our "top 2-4" features for the beta announcement are going to be. > Traditionally, we've mentioned two to four features in detail, together > with a bullet list of other significant features. With the help of PVH, > here's what I have so far: > > Banner features: > > - pglogical logical replication > - table partitions in core > > Significant features: > > - XMLTABLE query expressions > - hash indexes in wal > - PGFDW: push down aggregates > Notable note: pg_xlog => pg_wal > > So .., please add to the above lists. Thanks! I don't think it merits top 2-4 features, but transition tables for AFTER triggers is a SQL standard feature that I think most products have had and we have not ... until v10. -- Kevin Grittner
Re: [pgsql-advocacy] Assembling "top features" list for beta announcement
From
Peter Geoghegan
Date:
On Fri, Apr 7, 2017 at 9:39 AM, Josh Berkus <josh@berkus.org> wrote: > Banner features: > > - pglogical logical replication > - table partitions in core IMV, this also includes: - More extensive parallelism. Not really one feature, but a good bullet point for marketing purposes. - Expression evaluation optimization work, which will eventually lead to PostgreSQL using an LLVM based JIT. Both talks on these new features were absolutely packed at pgConf US, so it's clear to me that the user interest is certainly there. -- Peter Geoghegan VMware vCenter Server https://www.vmware.com/
Re: [pgsql-advocacy] Assembling "top features" list for beta announcement
From
Michael Paquier
Date:
On Sat, Apr 8, 2017 at 7:01 AM, Peter Geoghegan <pg@bowt.ie> wrote: > On Fri, Apr 7, 2017 at 9:39 AM, Josh Berkus <josh@berkus.org> wrote: >> Banner features: >> >> - pglogical logical replication >> - table partitions in core > > IMV, this also includes: > > - More extensive parallelism. Not really one feature, but a good > bullet point for marketing purposes. > - Expression evaluation optimization work, which will eventually lead > to PostgreSQL using an LLVM based JIT. > > Both talks on these new features were absolutely packed at pgConf US, > so it's clear to me that the user interest is certainly there. SCRAM perhaps? Users complain a good deal about MD5. -- Michael
On Sat, Apr 08, 2017 at 07:23:35AM +0900, Michael Paquier wrote: > On Sat, Apr 8, 2017 at 7:01 AM, Peter Geoghegan <pg@bowt.ie> wrote: > > On Fri, Apr 7, 2017 at 9:39 AM, Josh Berkus <josh@berkus.org> wrote: > >> Banner features: > >> > >> - pglogical logical replication > >> - table partitions in core > > > > IMV, this also includes: > > > > - More extensive parallelism. Not really one feature, but a good > > bullet point for marketing purposes. > > - Expression evaluation optimization work, which will eventually lead > > to PostgreSQL using an LLVM based JIT. > > > > Both talks on these new features were absolutely packed at pgConf US, > > so it's clear to me that the user interest is certainly there. > > SCRAM perhaps? Users complain a good deal about MD5. +1 to each of these. Best, David. -- David Fetter <david(at)fetter(dot)org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
Re: [pgsql-advocacy] Assembling "top features" list for beta announcement
From
Thomas Kellerer
Date:
Kevin Grittner schrieb am 08.04.2017 um 00:01: > I don't think it merits top 2-4 features, but transition tables for > AFTER triggers is a SQL standard feature that I think most products > have had and we have not ... until v10. I think only SQL Server and DB2 have that. Oracle has no such thing for their statement level trigger (it can be implemented using a compound trigger and a temporarytable) MySQL has no statement level triggers
On 04/07/2017 03:01 PM, Peter Geoghegan wrote: > On Fri, Apr 7, 2017 at 9:39 AM, Josh Berkus <josh@berkus.org> wrote: >> Banner features: >> >> - pglogical logical replication >> - table partitions in core > > IMV, this also includes: > > - More extensive parallelism. Not really one feature, but a good > bullet point for marketing purposes. Anyone got a list of the additional things which are parallelizable now? I know index scans, what else? -- Josh Berkus Containers & Databases Oh My!
Re: [pgsql-advocacy] Assembling "top features" list for beta announcement
From
Peter Geoghegan
Date:
On Fri, Apr 7, 2017 at 4:43 PM, Josh Berkus <josh@berkus.org> wrote: > Anyone got a list of the additional things which are parallelizable now? > I know index scans, what else? This is a good summary: https://rhaas.blogspot.com/2017/03/parallel-query-v2.html Of the "not yet committed" items listed there, only "Improve Access to Parallel Query from Procedural Languages" ended up making it into Postgres 10. -- Peter Geoghegan VMware vCenter Server https://www.vmware.com/
On 04/07/2017 02:42 PM, Peter Eisentraut wrote: >> - pglogical logical replication > Note that "pglogical" != "built-in logical replication" (or whatever you > want to call the latter). Clarify? -- Josh Berkus Containers & Databases Oh My!
So, lemme update the running list: Banner Features: - pglogical - new partitioning - quorum sync rep - parallel index scan, bitmap scan, and merge join Other Features: - XMLTABLE query expressions - hash indexes in wal - PGFDW: push down aggregates - SCRAM authentication ... seems like the "other features" list should be longer. -- Josh Berkus Containers & Databases Oh My!
Re: [pgsql-advocacy] Assembling "top features" list for betaannouncement
From
Peter Eisentraut
Date:
On 4/7/17 20:36, Josh Berkus wrote: > On 04/07/2017 02:42 PM, Peter Eisentraut wrote: >>> - pglogical logical replication >> Note that "pglogical" != "built-in logical replication" (or whatever you >> want to call the latter). > > Clarify? They are two different implementations that do similar things. - https://www.postgresql.org/docs/devel/static/logical-replication.html - https://2ndquadrant.com/en/resources/pglogical/ Only the first one is a feature of PostgreSQL 10. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: [pgsql-advocacy] Assembling "top features" list for betaannouncement
From
Peter Eisentraut
Date:
On 4/7/17 20:40, Josh Berkus wrote: > So, lemme update the running list: > > Banner Features: > > - pglogical > - new partitioning > - quorum sync rep > - parallel index scan, bitmap scan, and merge join > > Other Features: > > - XMLTABLE query expressions > - hash indexes in wal > - PGFDW: push down aggregates > - SCRAM authentication > > ... seems like the "other features" list should be longer. Some more possibilities: - better defaults for replication/online backups out of the box - ICU support - extended statistics (better performance for certain queries?) - executor performance improvements Also note that parallel query is now on by default, whereas it was disabled by default in 9.6. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Fri, Apr 7, 2017 at 5:35 PM, Thomas Kellerer <spam_eater@gmx.net> wrote: > Kevin Grittner schrieb am 08.04.2017 um 00:01: >> >> I don't think it merits top 2-4 features, but transition tables for >> AFTER triggers is a SQL standard feature that I think most products >> have had and we have not ... until v10. > > I think only SQL Server and DB2 have that. SAP SQL Anywhere also supports it and Sybase ASE supports the capability, but with fixed names for the transition tables (inserted and deleted). > Oracle has no such thing for their statement level trigger (it can be > implemented using a compound trigger and a temporary table) > > MySQL has no statement level triggers SAP MaxDB and SQLite are in the same camp as MySQL in this regard -- no support for it. -- Kevin Grittner
Re: [pgsql-advocacy] Assembling "top features" list for betaannouncement
From
Andreas Karlsson
Date:
On 04/08/2017 12:01 AM, Kevin Grittner wrote: > I don't think it merits top 2-4 features, but transition tables for > AFTER triggers is a SQL standard feature that I think most products > have had and we have not ... until v10. I think this will be a pretty huge feature for some users, so probably worth including among the significant features. Andreas
Hi, I think besides assembling the list, it is also important to come up with clear, correct and catchy names for each feature. Am Freitag, den 07.04.2017, 17:40 -0700 schrieb Josh Berkus: > So, lemme update the running list: > > Banner Features: > > - pglogical Again, pglogical is a specific out-of-core extension, which is not in V10. "Logical Replication" is the one we should promote. > - new partitioning I'd call it "Native Partitioning" or "Declarative Partitioning" > - quorum sync rep "Qurom-Based Syncronous Replication" > - parallel index scan, bitmap scan, and merge join "Improved and Extended Parallel Query" IMO the three headline features are Logical Replication, Partitioning and Parallel Query - not sure Quorum Commit stands out compared to the others below. > Other Features: > > - XMLTABLE query expressions > - hash indexes in wal "Improved and Crash-Proof Hash Indexes", or "Improved and Production-Ready Hash indexes" - there have been several performance optimizations as well. > - PGFDW: push down aggregates As the list is getting rather large, I am not sure this one merits inclusion, but I don't how much it improves things, really. > - SCRAM authentication I'm afraid this doesn't tell a lot of people what's going on, but I haven't come up with a better 3-5 word explanation, yet. > > ... seems like the "other features" list should be longer. There is also: "Client-Side Connection-Failover" "Improved and Easier Replication Setup" (Peter mentioned that as well) Plus what else Peter mentioned. From a marketing standpoint, I guess it is good that some features like WARM did not make it or else we'd have even more trouble coming up with a concise list. Michael -- Michael Banck Projektleiter / Senior Berater Tel.: +49 2166 9901-171 Fax: +49 2166 9901-100 Email: michael.banck@credativ.de credativ GmbH, HRB Mönchengladbach 12080 USt-ID-Nummer: DE204566209 Trompeterallee 108, 41189 Mönchengladbach Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer
On 8 Apr 2017, at 09:50, Michael Banck <michael.banck@credativ.de> wrote: <snip> > "Client-Side Connection-Failover" Is that for making HA/replicated servers easier to work with? If so, that sounds pretty important for application developers. :) + Justin -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi
On Fri, Apr 7, 2017 at 12:39 PM, Josh Berkus <josh@berkus.org> wrote: > While 10 isn't 100% baked yet, I'd like to get started sorting out what > our "top 2-4" features for the beta announcement are going to be. I have been thinking about this topic as well, and wrote a blog post about it. http://rhaas.blogspot.com/2017/04/new-features-coming-in-postgresql-10.html I wasn't able to pare the major features list down to the 2-4 range; I found it difficult enough to get it down to 7, and there's still a lot of other stuff that is also *really* good. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 8 Apr 2017, at 14:39, Robert Haas <robertmhaas@gmail.com> wrote: > On Fri, Apr 7, 2017 at 12:39 PM, Josh Berkus <josh@berkus.org> wrote: >> While 10 isn't 100% baked yet, I'd like to get started sorting out what >> our "top 2-4" features for the beta announcement are going to be. > > I have been thinking about this topic as well, and wrote a blog post about it. > > http://rhaas.blogspot.com/2017/04/new-features-coming-in-postgresql-10.html > > I wasn't able to pare the major features list down to the 2-4 range; I > found it difficult enough to get it down to 7, and there's still a lot > of other stuff that is also *really* good. Excellent post, that's well put together. :) Maybe we should steal content from it for the beta announcement? + Justin -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi
Michael, Robert (at bottom): On 04/08/2017 01:50 AM, Michael Banck wrote: > Hi, > > I think besides assembling the list, it is also important to come up > with clear, correct and catchy names for each feature. Thank you so much for the marketing-friendly terminology! > Am Freitag, den 07.04.2017, 17:40 -0700 schrieb Josh Berkus: >> So, lemme update the running list: >> >> Banner Features: >> >> - pglogical > > Again, pglogical is a specific out-of-core extension, which is not in > V10. "Logical Replication" is the one we should promote. Looks like we can use a more concise version of Robert's explanation for this. > >> - new partitioning > > I'd call it "Native Partitioning" or "Declarative Partitioning" "Declarative", I think > >> - quorum sync rep > > "Qurom-Based Syncronous Replication" > >> - parallel index scan, bitmap scan, and merge join > > "Improved and Extended Parallel Query" > > IMO the three headline features are Logical Replication, Partitioning > and Parallel Query - not sure Quorum Commit stands out compared to the > others below. I'd argue that Quorum Sync Rep combined with the client connection failover means that you can easily build synchronous PostgreSQL clusters out of the box, something which has previously required a lot of 3rd-party tooling. So it makes my list for major, especially if we combine the two features. Given that scale-out has been a weak story for Postgres in the past, I'm eager to trumpet any and all features we add in this direction. > >> Other Features: >> >> - XMLTABLE query expressions >> - hash indexes in wal > > "Improved and Crash-Proof Hash Indexes", or "Improved and > Production-Ready Hash indexes" - there have been several performance > optimizations as well. > >> - PGFDW: push down aggregates > > As the list is getting rather large, I am not sure this one merits > inclusion, but I don't how much it improves things, really. It's really "more progress in DB federation". Not momentous, but more significant than XMLTABLE, which personally I can't even explain. >> - SCRAM authentication > > I'm afraid this doesn't tell a lot of people what's going on, but I > haven't come up with a better 3-5 word explanation, yet. "More secure password authentication using SCRAM" > >> >> ... seems like the "other features" list should be longer. > > There is also: > > "Client-Side Connection-Failover" See above. > "Improved and Easier Replication Setup" (Peter mentioned that as well) Oooh, anyone have a link to details? > > Plus what else Peter mentioned. > > From a marketing standpoint, I guess it is good that some features like > WARM did not make it or else we'd have even more trouble coming up with > a concise list. Well, it's fine for the beta announcement to have an extended feature list, although at a certain point we just need to link in a What's New page on the wiki. On 04/08/2017 06:39 AM, Robert Haas wrote: > On Fri, Apr 7, 2017 at 12:39 PM, Josh Berkus <josh@berkus.org> wrote: >> While 10 isn't 100% baked yet, I'd like to get started sorting out what >> our "top 2-4" features for the beta announcement are going to be. > > I have been thinking about this topic as well, and wrote a blog post about it. > > http://rhaas.blogspot.com/2017/04/new-features-coming-in-postgresql-10.html > > I wasn't able to pare the major features list down to the 2-4 range; I > found it difficult enough to get it down to 7, and there's still a lot > of other stuff that is also *really* good. It's always nice to see that the reason the reviewer pipeline is so clogged is that we are getting so many contributions. From a PR standpoint, 5 is the absolute max, and 3 is better. Really, the ideal situation is a set of features which can be closely tied together around a simple theme. -- Josh Berkus Containers & Databases Oh My!
On 8 Apr 2017, at 20:48, Josh Berkus <josh@berkus.org> wrote: <snip> > From a PR standpoint, 5 is the absolute max, and 3 is better. Really, > the ideal situation is a set of features which can be closely tied > together around a simple theme. * Replication * Performance * Reliability ? + Justin -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi
On 04/08/2017 12:53 PM, Justin Clift wrote: > On 8 Apr 2017, at 20:48, Josh Berkus <josh@berkus.org> wrote: > <snip> >> From a PR standpoint, 5 is the absolute max, and 3 is better. Really, >> the ideal situation is a set of features which can be closely tied >> together around a simple theme. > > * Replication > * Performance > * Reliability Well, if we take logical replication, quorum commit, connection failover and partitioning, that together is a serious scale-out story. -- Josh Berkus Containers & Databases Oh My!
On 2017-04-08 1:50 AM, Michael Banck wrote: >> - PGFDW: push down aggregates > > As the list is getting rather large, I am not sure this one merits > inclusion, but I don't how much it improves things, really. I believe this absolutely merits inclusion in the announcement, though not as a top 3 item. Pushing down aggregates is huge for the scalability and performance of federated queries. If one is for example summarizing a billion remote rows into a dozen rows, this change means the GROUP BY and aggregation happens remotely and then only a dozen rows are passed over the wire instead of a billion rows. Next to pushing down projections or restrictions, the aggregate pushdown means the difference for a large fraction of common queries to be feasible versus not. It means that say a billion row aggregate takes effectively zero time of the client Postgres' resources. -- Darren Duncan
On Sat, Apr 8, 2017 at 3:48 PM, Josh Berkus <josh@berkus.org> wrote: > From a PR standpoint, 5 is the absolute max, and 3 is better. Really, > the ideal situation is a set of features which can be closely tied > together around a simple theme. I kinda don't buy this, but I suppose if I had to pick three, I'd pick partitioning, logical replication, and improved parallel query. If I could add a fourth I'd add SCRAM, particularly because that really needs testing and driver support. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Am Samstag, den 08.04.2017, 12:48 -0700 schrieb Josh Berkus: > > "Improved and Easier Replication Setup" (Peter mentioned that as well) > > Oooh, anyone have a link to details? Basically, the number of configuration changes required to setup replication has gone down; the only thing still needed is to enable hot_standby (and optionally setting up a replication slot). The following parameters were changed to make streaming replication possible at their default values: wal_level, max_wal_senders and max_replication_slots. Also, pg_hba.conf now allows replication from localhost by default. Relevant commits are "Change default values for backup and replication parameters" (f6d6d292) and "Enable replication connections by default in pg_hba.conf" (be37c212). The recovery.conf folding into postgresql.conf didn't make it, so not sure how big a deal this is. Michael -- Michael Banck Projektleiter / Senior Berater Tel.: +49 2166 9901-171 Fax: +49 2166 9901-100 Email: michael.banck@credativ.de credativ GmbH, HRB Mönchengladbach 12080 USt-ID-Nummer: DE204566209 Trompeterallee 108, 41189 Mönchengladbach Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer
Am Samstag, den 08.04.2017, 13:26 -0700 schrieb Josh Berkus: > On 04/08/2017 12:53 PM, Justin Clift wrote: > > On 8 Apr 2017, at 20:48, Josh Berkus <josh@berkus.org> wrote: > > <snip> > >> From a PR standpoint, 5 is the absolute max, and 3 is better. Really, > >> the ideal situation is a set of features which can be closely tied > >> together around a simple theme. > > > > * Replication > > * Performance > > * Reliability > > Well, if we take logical replication, quorum commit, connection failover > and partitioning, that together is a serious scale-out story. I'm cautious about how to tie logical replication, partitioning and connection failover into the (same) scale-out story. Also, 9.6 had remote_apply, which is arguably at least as important as quorum commit, as it means querying the standby right afterwards will result in the expected behavior; if we just had quorum_commit with synchronous_commit merely to 'on', querying the sync standbys immediately after a commit might still return the old rows (the new rows are written to WAL so are crash-safe, but are not necessarily replayed yet), to the best of my knowledge. So quorum commit extends remote_apply to multiple standbys. Finally, note that even though libpq now can handle mutiple hosts in the connection string, it won't load-balance between those (in contrast to pgJDBC BTW, which can do load balancing), but just use the next one if the current connection goes away (i.e. failover). Load-Balancing still needs external tooling for now. So we need to be rather careful about not over-selling things here, IMO (while not under-selling either, of course). Others more familiar with the new feature will hopefully weigh in. Michael -- Michael Banck Projektleiter / Senior Berater Tel.: +49 2166 9901-171 Fax: +49 2166 9901-100 Email: michael.banck@credativ.de credativ GmbH, HRB Mönchengladbach 12080 USt-ID-Nummer: DE204566209 Trompeterallee 108, 41189 Mönchengladbach Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer
One more thing: Am Sonntag, den 09.04.2017, 11:00 +0200 schrieb Michael Banck: > Also, 9.6 had remote_apply, which is arguably at least as important as > quorum commit, as it means querying the standby right afterwards will > result in the expected behavior; if we just had quorum_commit with > synchronous_commit merely to 'on', querying the sync standbys > immediately after a commit might still return the old rows (the new rows > are written to WAL so are crash-safe, but are not necessarily replayed > yet), to the best of my knowledge. So quorum commit extends remote_apply > to multiple standbys. > > Finally, note that even though libpq now can handle mutiple hosts in the > connection string, it won't load-balance between those (in contrast to > pgJDBC BTW, which can do load balancing), but just use the next one if > the current connection goes away (i.e. failover). Load-Balancing still > needs external tooling for now. Assuming we will get (connection-based) load balancing for 11 (educated guess - it should be pretty easy to add this now that we support multiple hosts) and also causal reads[1,2] (submitted for 10, but bumped to 11), this scale-out story will be a lot more coherent in my opinion; so we could focus on other high impact stories for 10 and leave scale-out for 11. Michael [1] https://commitfest.postgresql.org/14/951/ [2] https://www.postgresql.org/message-id/CAEepm=1iiEzCVLD=RoBgtZSyEY1CR-Et7fRc9prCZ9MuTz3pWg@mail.gmail.com -- Michael Banck Projektleiter / Senior Berater Tel.: +49 2166 9901-171 Fax: +49 2166 9901-100 Email: michael.banck@credativ.de credativ GmbH, HRB Mönchengladbach 12080 USt-ID-Nummer: DE204566209 Trompeterallee 108, 41189 Mönchengladbach Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer
Re: [pgsql-advocacy] Assembling "top features" list for betaannouncement
From
"Joshua D. Drake"
Date:
On 04/08/2017 07:09 PM, Robert Haas wrote: > On Sat, Apr 8, 2017 at 3:48 PM, Josh Berkus <josh@berkus.org> wrote: >> From a PR standpoint, 5 is the absolute max, and 3 is better. Really, >> the ideal situation is a set of features which can be closely tied >> together around a simple theme. > > I kinda don't buy this, but I suppose if I had to pick three, I'd pick > partitioning, logical replication, and improved parallel query. If I > could add a fourth I'd add SCRAM, particularly because that really > needs testing and driver support. 3-5 is the sweet spot for a PR. I think SCRAM being listed is a very good thing. JD -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone appreciates your honesty, until you are honest with them.
On Sun, Apr 9, 2017 at 10:34 AM, Joshua D. Drake <jd@commandprompt.com> wrote: > On 04/08/2017 07:09 PM, Robert Haas wrote: >> On Sat, Apr 8, 2017 at 3:48 PM, Josh Berkus <josh@berkus.org> wrote: >>> From a PR standpoint, 5 is the absolute max, and 3 is better. Really, >>> the ideal situation is a set of features which can be closely tied >>> together around a simple theme. >> >> I kinda don't buy this, but I suppose if I had to pick three, I'd pick >> partitioning, logical replication, and improved parallel query. If I >> could add a fourth I'd add SCRAM, particularly because that really >> needs testing and driver support. > > 3-5 is the sweet spot for a PR. I think SCRAM being listed is a very good > thing. Well, the problem is that everybody's going to have their own ideas on what is most important. A release announcement that said partitioning, logical replication, and improved parallel query could sound dreadfully boring to someone who cares about SCRAM, better query planning when RLS is in use, and the default configuration allowing replication. I don't suppose there's a whole lot we can do about that - we can't list everything - but we're blessed (or cursed) with needing to promote a release that really has a ton of great stuff in it, and we don't want people to tune out because the stuff they care about isn't listed. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: [pgsql-advocacy] Assembling "top features" list for betaannouncement
From
"Joshua D. Drake"
Date:
On 04/09/2017 02:56 PM, Robert Haas wrote: >> 3-5 is the sweet spot for a PR. I think SCRAM being listed is a very good >> thing. > > Well, the problem is that everybody's going to have their own ideas on > what is most important. That is definitely true but I think that is where leadership comes in. What do our primary users think is important? Better Partitioning and Logical replication are a given. Parallel query (AWESOME!) and SCRAM are probably at a head based on the audience. It is true that some will care about better query planning with RLS but it is a *very* small subset of our user base. > - we can't list everything - but we're blessed (or cursed) with > needing to promote a release that really has a ton of great stuff in > it, and we don't want people to tune out because the stuff they care > about isn't listed. True but consider this: The people who care about better query planning with RLS are already watching the lists for better query planning with RLS. From an Adoption and "What is going to hit the presses" perspective, things like Replication and easy(er) Partitioning are going to hit more hearts and minds. From an adoption perspective the question we should ask is: "What are the people running RDS, Compose Postgres and Heroku Postgres going to care about?. The top three there are, "Better Partitioning, Logical Replication (Logical replication allows you to replicate OUT of RDS) and Parallel query with a very close combat of SCRAM." Parallel Query, Scram and better query planning with RLS are very much enterprise features (with Parallel query being the most cross culture). YMMV, JD -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone appreciates your honesty, until you are honest with them. Unless otherwise stated, opinions are my own.
JD, Robert, * Joshua D. Drake (jd@commandprompt.com) wrote: > On 04/09/2017 02:56 PM, Robert Haas wrote: > >>3-5 is the sweet spot for a PR. I think SCRAM being listed is a very good > >>thing. > > > >Well, the problem is that everybody's going to have their own ideas on > >what is most important. > > That is definitely true but I think that is where leadership comes > in. What do our primary users think is important? I don't actually know that it's our existing users that we want to be targeting with this. Further, I'm not sure I follow what you mean by "primary users." Did you intend "majority of?" > Better Partitioning and Logical replication are a given. Parallel > query (AWESOME!) and SCRAM are probably at a head based on the > audience. I prefer the term used elsewhere that I saw- "Native" partitioning, instead of "Better." In line with the idea that we really want to target people who don't currently use PG to reconsider why they don't and if they should, things like "Native partitioning" and "logical replication" are big, new, features which may make them consider PG more seriously than they did before. > It is true that some will care about better query planning with RLS > but it is a *very* small subset of our user base. I agree that a minority of existing users are interested in RLS. In addition- we already have it and made a splash with it being released as part of 9.5. Improvements to it are certainly note-worthy, but talk of improved planning with RLS strikes me as less likely to make non-PG users consider switching to PG. To some extent, the same goes for improved parallel query, though the situation is a bit different there since it wasn't enabled initially either. > >- we can't list everything - but we're blessed (or cursed) with > >needing to promote a release that really has a ton of great stuff in > >it, and we don't want people to tune out because the stuff they care > >about isn't listed. > > True but consider this: The people who care about better query > planning with RLS are already watching the lists for better query > planning with RLS. I'm not sure that's really accurate. I'm sure some are, but not all of them. That said, I agree that it's still a minority of our overall userbase. > From an Adoption and "What is going to hit the presses" perspective, > things like Replication and easy(er) Partitioning are going to hit > more hearts and minds. > > From an adoption perspective the question we should ask is: "What > are the people running RDS, Compose Postgres and Heroku Postgres > going to care about?. The top three there are, "Better Partitioning, > Logical Replication (Logical replication allows you to replicate OUT > of RDS) and Parallel query with a very close combat of SCRAM." I don't think we should focus only on what users of hosted platforms are interested in. For one thing, those users are already using PG and will be upgrading based on what the hosted platform provides, or forces on them. > Parallel Query, Scram and better query planning with RLS are very > much enterprise features (with Parallel query being the most cross > culture). I agree, and we should certainly be considering those users. Put another way, I'm worried that the focus here is on the number of PG installs (hosted platforms have tons) without consideration for the other end of the spectrum (large, locally hosted, systems). Both are important to us. I don't have a great answer to which is *more* so (one could start to consider "brand awareness" vs. "contribution to the community" and other such metrics, but we'd then have to determine what our goals for this are, exactly, and this thread doesn't strike me as the right place to try and hash *that* out..). Thanks! Stephen
Attachment
On 04/09/2017 02:56 PM, Robert Haas wrote: > On Sun, Apr 9, 2017 at 10:34 AM, Joshua D. Drake <jd@commandprompt.com> wrote: >> On 04/08/2017 07:09 PM, Robert Haas wrote: >>> On Sat, Apr 8, 2017 at 3:48 PM, Josh Berkus <josh@berkus.org> wrote: >>>> From a PR standpoint, 5 is the absolute max, and 3 is better. Really, >>>> the ideal situation is a set of features which can be closely tied >>>> together around a simple theme. >>> >>> I kinda don't buy this, but I suppose if I had to pick three, I'd pick >>> partitioning, logical replication, and improved parallel query. If I >>> could add a fourth I'd add SCRAM, particularly because that really >>> needs testing and driver support. >> >> 3-5 is the sweet spot for a PR. I think SCRAM being listed is a very good >> thing. > > Well, the problem is that everybody's going to have their own ideas on > what is most important. A release announcement that said > partitioning, logical replication, and improved parallel query could > sound dreadfully boring to someone who cares about SCRAM, better query > planning when RLS is in use, and the default configuration allowing > replication. I don't suppose there's a whole lot we can do about that > - we can't list everything - but we're blessed (or cursed) with > needing to promote a release that really has a ton of great stuff in > it, and we don't want people to tune out because the stuff they care > about isn't listed. > No quesiton. That's why we divide the beta announcement into two sections: one with the features we'd like press/reddit/etc. to pick up, and one with all of the features which we want users to test. This is different from the final release PR, which will include only the banner features. To clarify: Banner Features --------------- 2-5 Features which meet all of: - will excite the press/twittersphere - potentially ensnare new users - can explain to non-DB geeks in 1-2 sentences Ideally, the above fits around a "theme", but we'll see. Additional Features ------------------- 3-12 Features which meet any of: - major features not included in above - things we really want user testing on - big API changes - other stuff we think might grab attention in specific communities -- Josh Berkus Containers & Databases Oh My!
Re: [pgsql-advocacy] Assembling "top features" list for betaannouncement
From
Peter Eisentraut
Date:
On 4/8/17 04:50, Michael Banck wrote: > "Client-Side Connection-Failover" I don't think the libpq feature can be considered "failover". It just picks from a list of end points on the initial connection attempt. It does not discover connection failures and reconnect. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 10 Apr 2017, at 16:42, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote: > On 4/8/17 04:50, Michael Banck wrote: >> "Client-Side Connection-Failover" > > I don't think the libpq feature can be considered "failover". It just > picks from a list of end points on the initial connection attempt. It > does not discover connection failures and reconnect. Yeah, after reading Roberts blog post, I'd also agree that "failover" is a bad/misleading (in the "automatic" sense) description for it. + Justin -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi
On 04/10/2017 09:04 AM, Justin Clift wrote: > On 10 Apr 2017, at 16:42, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote: >> On 4/8/17 04:50, Michael Banck wrote: >>> "Client-Side Connection-Failover" >> >> I don't think the libpq feature can be considered "failover". It just >> picks from a list of end points on the initial connection attempt. It >> does not discover connection failures and reconnect. > > Yeah, after reading Roberts blog post, I'd also agree that "failover" > is a bad/misleading (in the "automatic" sense) description for it. Alternate phrasing? I do want to get this in the "extended list" of features, just because we want driver authors to add support for it. -- Josh Berkus Containers & Databases Oh My!
Re: [pgsql-advocacy] Assembling "top features" list for betaannouncement
From
Peter Eisentraut
Date:
On 4/10/17 14:10, Josh Berkus wrote: > On 04/10/2017 09:04 AM, Justin Clift wrote: >> On 10 Apr 2017, at 16:42, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote: >>> On 4/8/17 04:50, Michael Banck wrote: >>>> "Client-Side Connection-Failover" >>> >>> I don't think the libpq feature can be considered "failover". It just >>> picks from a list of end points on the initial connection attempt. It >>> does not discover connection failures and reconnect. >> >> Yeah, after reading Roberts blog post, I'd also agree that "failover" >> is a bad/misleading (in the "automatic" sense) description for it. > > Alternate phrasing? > > I do want to get this in the "extended list" of features, just because > we want driver authors to add support for it. I'm not sure what you mean by this. If the driver is based on libpq, they get it for free. If the driver is not based on libpq, they have to implement it themselves, but don't need to wait for libpq before doing so. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: [pgsql-advocacy] Assembling "top features" list for beta announcement
From
Magnus Hagander
Date:
On Mon, Apr 10, 2017 at 9:17 PM, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote:
On 4/10/17 14:10, Josh Berkus wrote:
> On 04/10/2017 09:04 AM, Justin Clift wrote:
>> On 10 Apr 2017, at 16:42, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote:
>>> On 4/8/17 04:50, Michael Banck wrote:
>>>> "Client-Side Connection-Failover"
>>>
>>> I don't think the libpq feature can be considered "failover". It just
>>> picks from a list of end points on the initial connection attempt. It
>>> does not discover connection failures and reconnect.
>>
>> Yeah, after reading Roberts blog post, I'd also agree that "failover"
>> is a bad/misleading (in the "automatic" sense) description for it.
>
> Alternate phrasing?
>
> I do want to get this in the "extended list" of features, just because
> we want driver authors to add support for it.
I'm not sure what you mean by this. If the driver is based on libpq,
they get it for free. If the driver is not based on libpq, they have to
implement it themselves, but don't need to wait for libpq before doing so.
And libpq already copied the idea from jdbc, did it not? So at least one non-libpq driver already has it.
Re: [pgsql-advocacy] Assembling "top features" list for betaannouncement
From
"Joshua D. Drake"
Date:
On 04/10/2017 12:17 PM, Peter Eisentraut wrote: > On 4/10/17 14:10, Josh Berkus wrote: >> On 04/10/2017 09:04 AM, Justin Clift wrote: >>> On 10 Apr 2017, at 16:42, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote: > I'm not sure what you mean by this. If the driver is based on libpq, > they get it for free. If the driver is not based on libpq, they have to > implement it themselves, but don't need to wait for libpq before doing so. It also seems like this is something we could put out separate from the PR. A "driver authors" notice or something like that. JD > -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone appreciates your honesty, until you are honest with them. Unless otherwise stated, opinions are my own.
Am Montag, den 10.04.2017, 11:42 -0400 schrieb Peter Eisentraut: > On 4/8/17 04:50, Michael Banck wrote: > > "Client-Side Connection-Failover" > > I don't think the libpq feature can be considered "failover". Hrm. > It just picks from a list of end points on the initial connection > attempt. Right. > It does not discover connection failures and reconnect. Does any other 3rd party tooling for Postgres do that? I assume the client will get the connection failure and, upon reconnection, will get a good new connection, essentially failing over to the next host. Oracle has "Transparent Application Failover", but that's a pretty high bar I am not sure anybody is expecting us to jump over just now. I phrased it "connection-based client-side failover" in another mail, would emphasizing that it is at connection granularity be enough to address your concerns? The pgJDBC called the (AFAICT) identical feature "connection failover" in their changelogs for 9.3-1100 and 9.4-1200: https://jdbc.postgresql.org/documentation/changelog.html#version_9.3-1100 https://jdbc.postgresql.org/documentation/changelog.html#version_9.4-1200 Michael -- Michael Banck Projektleiter / Senior Berater Tel.: +49 2166 9901-171 Fax: +49 2166 9901-100 Email: michael.banck@credativ.de credativ GmbH, HRB Mönchengladbach 12080 USt-ID-Nummer: DE204566209 Trompeterallee 108, 41189 Mönchengladbach Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer
On 04/10/2017 12:25 PM, Michael Banck wrote: > Am Montag, den 10.04.2017, 11:42 -0400 schrieb Peter Eisentraut: >> On 4/8/17 04:50, Michael Banck wrote: >>> "Client-Side Connection-Failover" >> >> I don't think the libpq feature can be considered "failover". > > Hrm. It's still valuable, though, especially in combination with replication which offers rapid failover. It means that instead of needing to update all of your clients with a new list of servers immediately, you can wait and assume that their reconnection logic will handle things until an admin can push out an update. Particularly, with MM replication (like BDR or MMSync), it means you don't have to worry about failover on the clients at all. So it's not a major feature on its own, but it's a good part of a scale-out story. -- Josh Berkus Containers & Databases Oh My!
On, some reflection, I think WARM should make the extended features list. It's been in the press, after all. -- Josh Berkus Containers & Databases Oh My!
On Wed, Apr 12, 2017 at 12:29 PM, Josh Berkus <josh@berkus.org> wrote: > On, some reflection, I think WARM should make the extended features > list. It's been in the press, after all. Uh ... that's not committed? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 04/12/2017 09:41 AM, Robert Haas wrote: > On Wed, Apr 12, 2017 at 12:29 PM, Josh Berkus <josh@berkus.org> wrote: >> On, some reflection, I think WARM should make the extended features >> list. It's been in the press, after all. > > Uh ... that's not committed? > Oh, well, *details*! (waves hands around) -- Josh Berkus Containers & Databases Oh My!
All, So apparently the press has already done our work for us. This is the first time in years that the press has paid enough attention to mailing lists and blogs to publish something without a PR being sent. It's nice. BTW, if anyone can get me these two reporters' contact info, it would be great. We need more friendly reporters. http://www.infoworld.com/article/3187803/database/postgresql-100-appeals-to-the-sql-faithful.html https://www.infoq.com/news/2017/04/postgresql-10-features -- Josh Berkus Containers & Databases Oh My!
On 2017-04-18 06:18, Josh Berkus wrote: > All, > > So apparently the press has already done our work for us. > > This is the first time in years that the press has paid enough > attention > to mailing lists and blogs to publish something without a PR being > sent. > It's nice. > > BTW, if anyone can get me these two reporters' contact info, it would > be > great. We need more friendly reporters. > > http://www.infoworld.com/article/3187803/database/postgresql-100-appeals-to-the-sql-faithful.html Serdar Yegulalp's email is on his Infoworld profile page: serdar_yegulalp AT infoworld DOT com > https://www.infoq.com/news/2017/04/postgresql-10-features Not seeing an email address on the InfoQ website, but it might be available only to registered users (which I'm not doing just for this :>). + Justin
Josh Berkus wrote: > https://www.infoq.com/news/2017/04/postgresql-10-features "PostgreSQL continues to progress towards its tenth major release" ... Evidently a version numbering scheme change is not without downsides ;-) -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services